It depends by your definition of polarity, it doesn't have polarity resulting from unequal distribution of electrons in bonds.
However, there will be extreme and I mean extreme instantaneous dipole-induced dipoles as there are a great number of electrons in the compound. I suspect the forces between benzene are far greater than the H bonds in water, so benzene would be pretty insoluble