I think you are right....might be better to opt for a mid-popular one.
Though to be honest, the "less popular" colleges are generally as nice if not nicer than the popular ones!
It's only usually a case of distance from centre (which isn't anything more than a nice brisk walk/bike ride) that makes some colleges less desirable.
In reality life at St Anne's, Hilda's, LMH, St Hugh's wont be signficiantly different from life at New College, Magdalen etc. You get cloisters at the grand old establishments, but it depends how much cloisters matter to you. x