I would say that if maths is your thing then physics won't be too bad. Higher physics is heavily based on calculations from equations - and you get a data book, so you don't even have to remember them. There is some explaining to do and that, in my experience, was the hardest bit. But the hardest things to explain e.g. semiconductors etc. tend not to come up too often anyway. I would say try it and if you really don't like it you can always switch to something you're more suited to.