Hiya,
Oxford offer certain scholarships or bursaries to students from low-income family. When I was there, I was given a £10K grant (i.e. didn't need to ever be paid back) over the three years of my course, due to my family income (or lack thereof). That was quite some time ago, so things have changed but I believe the provisions are even better now (I finished my Oxford degree 11 years ago, so my info is out of date).
Check out this link for current financial provisions from the uni:
https://www.ox.ac.uk/admissions/undergraduate/student-life/help-with-the-cost Also be aware that certain colleges have extra support, in the form of college-based grants/bursaries/awards
If Oxford offers your course and you like the course content/structure/teaching method (i.e. tutorials), I'd encourage you to think about applying. You'd definitely be able to survive: money should never be a barrier to applying to or studying at Oxford
You will get the odd super-rich person who doesn't quite understand how "the other side" live but they are few and far between - and indeed, you'll get those sorts of people at any of the top unis in the UK
For the most part, talk of money/financial background doesn't come into things at all: people don't tend to ask questions about that kinda stuff, because very few people care how rich or poor you are.
Can I ask what you mean by peer pressure in this instance, just so I don't go off on unnecessary tangents about that?