Hmm, this is tricky..
Thing is, before everyone goes to uni, they choose based on the course. Once you're there, the most important thing and the reason I (and everyone I know) gets up every morning is for the social side - not necessarily partying but just friends, laughs, eating together. If I was constantly sat in the library not understanding any of my modules and ploughing through problem sheets, knowing everyone around me had an IQ of about 200, I'd be miserable. Obvz that's exaggerating but you get the point. Struggling with 2/5 modules is bad enough! I know people on TSR constantly complain about how uni is "meh" but personally, it's been the best year of my life so far and I can't see how life after graduation could equal it. Living on your own, having to pay loads of bills and being the office junior while NOT going out twice a week with your now-gone mates? Uni years is the one of the best years of your life, not just a means to an end as some put it, so you should be happy.
The job prospects are obviously better at Cambridge but it's not like you'll be flipping burgers with a Notts degree, with enough work you can get to where you want to be. The point is, you seem to have convinced yourself you'll hate Cambridge - how do you know? It is an amazing opportunity you'd be turning down, and if you like maths, wouldn't you want to push yourself rather than settle for a course where you might be bored for the first year. There are people at EVERY uni who want to go and party - on our course I constantly see the same faces when out clubbing, and I also see the contingent who head to the library after every lecture. It'll be the same at any uni.
Basically, I'm not really on any side, you should just think about your decision carefully. You don't want to have a tough time and be unhappy at Cambridge, but you don't want to be in Nottingham finding the course easy and regretting turning down arguably the #1 uni for maths!