Yer, you should definately do what you think is right for you for the next 4 years of your life. It's probably too late now but I found looking at the course content helps a lot in a decision, 'reputation' is certainely overrated and a future career is a tenuous reason to go to university X. But with physics I think the course at Imperial was genuinely better than other places, and thus why Imperial will probably be my firm.