I'm an Oxford mathmo too, or rather I was as I've just finished.
Would definitely recommend the course. No it isn't easy as others have testified, but it is do-able, or at least until you hit 4th year, where I did struggle a bit. You'll be surprised at first at how different the pure maths is from the pure you do at school and I found that really difficult at first. But when I got used to it I realised that pure (specifically algebra) was actually my favourite bit.
After the first four terms, in which everything is compulsory to give you a good grounding in most areas, you can pick options and in 3rd/4th year you have a huge range to choose from. That for me is the best bit about the course and I was very happy to ditch pretty much all applied maths! Unfortunately all of the teaching from 3rd year on is faculty-based, so you have intercollegiate classes rather than tutorials in college. That I didn't find especially effective, but I guess you can't win them all! And my college tutor was fantastic at giving extra help where it was needed (which in one course it desperately was!!)
Exams every year does suck a bit, but it was nice going into 3rd and 4th year exams knowing I already had a fair bit of my degree in the bag so it was worth it in the end. Also the option to choose between 3 and 4 years also a good thing - I really enjoyed having a 4th year in Oxford and will now have an MMath for my efforts.
Most of the way through each year I was probably doing less work than most of my friends, but I wasn't always the most conscientious with the problem sheets. But you can get by by making sure you go to pretty much all your lectures and classes, being efficient in getting the notes for what you miss, doing lots of practice papers and revising hard at the end of the year. I managed to cram a huge amount of extra-curricular stuff into my 4 years and still come out with a first. And I'm by no means a genius. From what I've heard of the Cambridge course it might not be possible to do the same there.
As for college, I'd personally go for Worcester over Merton. It's a beautiful college and everyone I know there loves it. Their stats tutor (Daniel Lunn) is possibly the best lecturer I have ever had, and other people I know who have interviewed there have had complimentary things to say about their other maths tutors. If you're into sport then it's certainly a good choice as their sports teams are strong and their playing fields on site. Merton is also lovely to look at but there is a lot of academic pressure there. Whether that's a good thing is for you to decide. Do also consider Queen's - two fantastic maths fellows who'll go out of their way to help, great college atmosphere, fairly laid-back academic attitude within the mathmos (not to say we don't work, we just all do plenty of other things and have a balanced approach to it). But hey, I'm biased :-)
If you have any more questions then feel free to PM me.