Make the decision you want to, if you know Cambridge isn't right for you. I'm not just adding an opinion for the sake of it either, I am currently a secong year at Clare college Cambridge doing natsci. I applied for physics because I liked it and enjoyed it at the time I applied and was very good at it. I seriously considered turning down my offer and heading to Durham or Bath so there would be less pressure. I took the place in the end but have really not being enjoying my time, I don't want to drop out but I feel I would have had a much better time at a different uni. However, its NOT the social life that made me feel this, there is loads of stuff to do and I feel Ive made great friends there.
BUT: I know loads of people who are loving the coursess and getting firsts and still have great social lives too.