Durham v Cambridge for physics

I'm struggling to choose between Durham and Cambridge. Everyone tells me that I'd be brave (and somewhat stupid) to decline an offer from Cambridge, considering its practically the best uni in the world and I'd have no trouble finding a job afterwards. But I've been to Cambridge a few times and to a 3 day long summer school and, tbh, I just wasn't feeling it. I don't know what it was, but I just didn't feel at home. Whereas, when I went to the Durham open day, I completely fell in love with it! The town, the people, the professors - as soon as I walked in, I just had a gut feeling that this place is for me.
Uni is gonna take three to four years of my life, and I want to enjoy it. I just feel I'd enjoy it more at Durham. But then again, studying at Cambridge is rare opportunity that many would die for...
Any advice!