I agree with OP to an extent. I know it sounds cliched to ask everyone to 'check their privilege' but how many of us would really be at university (or at good universities for that matter) if we hadn't grown up in nurturing, financially comfortable environments where we were encouraged to do well in class? That's not to say everyone is a complete product of their circumstance, there are plenty of exceptions (who have probably already pointed themselves out here), but it does go at least some way in determining outcomes in life.
Is it fair or right? I'd say it's hard to morally defend without some sort of an appeal to prescribed social order that has been on the way out for hundreds of years now. The argument that 'that's just how life is, you can't do it perfectly, so why bother' is a fallacious one that would be laughed at if it were applied to, for instance, modern medicine. Even parties on the right pretty much universally subscribe to 'equality of opportunity' as a thing to aspire to. That being said, exactly how you do it is what is really up for debate.