Three points:
1. When I was interviewed at Oxford in December, there were apparently 30 candidates for 6 places. Presumably all were plausible candidates.
2. Entry to Oxbridge not only needs an element of luck, but is a privilege, not a right for those who get the requisite grades. That said, the factors considered are a lot broader than 20 years ago, when, for example, you didn't do a personal statement. I don't remember UCCA (UCAS' predecessor) using references, either, and certainly not the 3 I had to submit for Oxford.
3. When I did my first degree at Cambridge, I never met anyone who I thought was too dim to be there. I met plenty of people who seemed far brainier than me (and I was top of my year at school for all 3 of my subjects); there were many undergraduates who had obviously had a better secondary education than me; and quite a few who, although higher academic achievers than I was, had absolutely no personality or savoir-vivre and would probably never do well in life by most normal people's standards.