It could be a sign that you need a rethink, but it's hard to say. Physics is very different at degree level to A level, it's much more involved (and difficult of course) and much more maths based, which you will also get more of later on in the degree doing theoretical stuff.
Having said that though, everyone doing a physics degree will like some modules and be less keen on others, not many people like electromagnetism on my course, not many people find it interesting because it's more functional than exciting, but you might like it if you enjoy maths because it's just a maze of differential calculus (I quite enjoy it for that reason).
A lot of people dislike labs (I'm doing astrophysics and don't really enjoy them), but on theoretical physics you'll probably have considerably less labs after first year (where everyone has to do some basic labs) and then move on to mathsy/computer programming type stuff.
It's hard to say whether you'll like it or not, although I have to say, I'm surprised you've managed to get an offer from Imperial with little to no enthusiasm for the subject (although I don't know their selection process to be fair). I certainly enjoy it far more than I did at A level, and I never really took a massive interest in the current goings on in physics until I started and became more interested in it. It might be worth trying to get hold of some material so you know the kind of stuff you'll be learning. And also if you like the applied maths modules at A level then they're relatively similar to some of the more basic areas that you do (like mechanics).
Is there anything else that you think you might want to do instead? If you're realised you have a passion for chemistry or maths then try and change for that, but I don't think you'll know whether/how much you'd like physics until you get there, and you know better than anyone else. If there are any questions you have on doing physics at uni then I can help though, so fire away if you want.