The university did a huge study last year, which found no correlation between college choice and students overall happiness with their university experience. However college choice does matter a lot, in that college's differ tremendously. How do you reconcile these? Two ways, firstly, people apply to the one they'll probably fit in well at, so if you want to slack, you probably won't apply to Merton, and if you want old buildings, you probably wouldn't apply to Catz. Secondly people feel college pride, and have only experienced life at their college, so won't necessarily know how they'd find it at other colleges.
To show the disparity, this term I've had 3-4 tutes/classes a week, with work to do for each. Whereas my friends at Jesus have 2 tutes/classes a week. If you want to do well in exams and are prepared to work, my college would be preferable, whereas if you want to do society stuff and are happy with a lower mark, Jesus would be better.
Similarly, each college has different facilities, room rents/quality, scholarships/grants, political leanings and atmosphere. So while a staunch conservative might love it at Oriel or Christ Church, I've met a few socialists that wish they were at Balliol or Wadham.
So the advice "don't choose on reputation but on whether you would really want to be there" is bang on, but that doesn't mean college choice isn't important. As my brother advised me when deciding between Oxford and Cambridge: "in some cases the difference between two Cambridge colleges is greater than the difference between Oxford and Cambridge".