I think it has a lot to do with insecurity and I believe the majority of people who go on about ex-polys are people who have no real world experience, nor any experience at university.
The educational system in the UK is flawed. A hell of a lot of weight is put on A-Level results in determining your future academic pursuits. All this at an age where you may not truly comprehend the ramifications of not getting the very best grades. Also taking into account the huge step up in difficulty from GCSE level to A level in certain subjects (particularly the sciences, in my experience).
Selection of A-levels can be a huge factor as well. I went from 5A*s, 5As and a B (not great GCSEs, but they were the best in my year) to ACD in my A2s. I didn't change my work ethic or anything, the material was just a lot more difficult compared to GCSE and I think a lot of people tend to pick A-levels in the subjects they enjoyed at GCSE (for me this was Chemistry and Biology). But I got my A in Law, and so am now doing a Law degree at an ex-poly.
Do I wish I got better grades and went to a 'better' more prestigious university? Sure, but I know that's my inner gloating nature thinking for me.
When you're actually in the work place earning real world experience though, your degree won't matter.
Long and rambling post, apologies!