In my opinion, if you have excellent teachers in both subjects, physics is easier. The questions are much more basic and as long as you understand it you won't have a hard time.
However, from experience, there are much more excellent maths teachers than physics. If you end up with a not so good physics teacher, you'll find it extremely difficult and will be spending a lot of time reading your textbook just to catch up on your useless lessons.
I'm kind of trying to say the physics content is easier to teach yourself than the maths content. But with maths you generally won't have to resort to that.
This is just my opinion, I don't know if other students have had similar experiences to me and can't say that you will too.