That sounds like a difficult decision then! I really enjoy both too, and the two AHs go really well together. If you're going for medicine, then I suppose it doesn't really matter, but do you receive a conditional offer based on your AHs? In that case, maths is maybe easier to get an A in without as much work, although AH Physics is really interesting. However, the hour long trip might be a bit of a hassle.
Presumably, you'll be taking AH Biology and AH Chemistry? Both of those subjects will require you to complete a project throughout the year, which is a lot of work. If you take AH Physics, you'll need to do three projects altogether, which could be very stressful. Since AH Maths is 100% exam-based, you'd have more time to focus on your AH Biology/Chemistry projects, and potentially achieve better grades for a conditional offer. Also, AH Biology is apparently much more challenging than Higher, with a lot of content and memorisation, and a lot more investigative work. AH Chemistry and Physics are really just a natural continuation from Higher, but AH Chemistry is a bit of a step-up, particularly in organic chemistry, and physics is more mathematical.