Like my contemporaries, I'm inclined to say 'do maths'. Higher Physics doesn't rely on mathematical ability at all, it's really just plug 'n' chug with the formulas to get the answer you want - so don't feel like you'll lose out on that front.
Mathematics has a wider range of applicability and I'd say would improve your application more than Physics would, but for me; I don't find one harder than the other. So it's going to come down to which you would feel benefits you more, besides, surely you can do one in S5 and the other in S6?