Xena, I think the point the chap was making was about the university, not the city.
I'm from Leeds, study at Warwick and come to back to Leeds Uni to do research in the vacation period. I'm reluctant to generalise, and i know this is completely subjective, but whenever i'm up Woodhouse Lane, i do feel Leeds Uni has a higher percentage of what others would term 'rahs' or Sloane Rangers, etc.
I think it probably has more to do with the city's geographical location than anything else, and the fact that London and the south east have the highest number of university applicants in the country, so there's bound to be some imbalance.
It must be off-putting if your background is rather different, and yes, it's irritating when they start slagging off Leeds (even though they hail from some nondescript place in the south east), but Britain is a fairly unequal society, and if they can afford to go to the top public schools, they are more likely to dominate the top universities.