Physicist or mathematician?

As a general rule, it's easier to go into physics from a background in maths than it is to go into maths from a background in physics. At my uni, for example, you can switch from Maths to Physics after your first, second or third year provided you've done enough theoretical physics and are willing to put in the reading to catch up; however, the reverse is normally not true (except some physicists who transfer into fourth year Maths, but that's only if they're going to do the theoretical physics courses rather than pure or applied maths courses). I originally applied for Computer Science, where the situation is exactly the same, and upon realising this I switched into Maths whilst I still could.

Also the maths you use in undergraduate physics isn't particularly conceptually complicated compared to pure maths. So if you're interested in abstract concepts and generalisations and that sort of thing, you won't get a lot of that in physics. [Don't get me wrong, things like Fourier transforms are much more abstract than anyone would have thought of 300 years ago, but compared to the stuff that's happening in pure maths today it's pretty concrete.]

Obviously I'm biased, but the idea is this: if you really like physics and you're not that bothered about seeing where mathematical insight can take you, then do physics; if you're geniunely in doubt, you're better off either finding a joint course or just doing maths (with a slant on theoretical physics if possible).