I remember choosing my college partly because it had the *most* applicants for my subject (I assumed that meant it was the 'best' for my subject...I didn't know that so much of the teaching was faculty based, and that colleges often share supervisors).
In answer to the original question: back then, I'd have seriously considered turning it down. I get on quite well with guys, and went to an all-girls school so maybe would have wanted to be in a mixed environment. With the knowledge I have now, I would probably accept, as I loved the course I did at Cambridge and the city. On the other hand, it was mainly my college that made my experience so good, rather than anything about the university or city itself, so who knows.