just remember people: The college does not maketh the man - there are people at all college at both oxford and cambridge who are completely different to what the stereotype of their college is. If you choose not to conform to the stereotype then fine, but i found that being around people who do match the stereotype does have an influence on you. For example, being at my college has made me work harder than i've ever worked before because i'm surrounded by people who work loads. Whereas a friend at another college works so much less even though he does the same subject, because people seem to work less overall at his college.
But as I say, you certainly do not end up matching the stereotype exactly....if at all!