I did maths at Oxford from 2010-2014 - feel free to PM me if you want specific advice.
My best tip is don't go there with any preconceptions. You will meet all kinds of people, with lots of different sides to them, and they will be just as nervous as you. Nobody likes being stereotyped. You will meet people who pretend to be something they are not (the 'Oxford' myth does generate some odd behaviour), but they usually grow out of it. Best thing is to just be yourself and be social, and you'll definitely meet people you get on with. In my year there were all kinds of different people.
It isn't the same university experience you might get at a 'party uni', but there are plenty of people willing to have fun (both of the clubbing kind and the normal friendship group kind), lots of clubs and societies, and the collegiate system means you automatically get to know your year, if you make the effort. Plus you will get to work with some of the most intelligent people in the world. I would definitely go if you have the chance.