Is it better to risk failing my required subjects or not do my favourite subject?

Basically I am currently considering medicine, so obviously I need biology and chemistry and at least AAA. However my favourite subject is physics and I want to do that but colleges are saying if I do physics I have to do maths as well. I don't know how good I would be at four a-levels, especially since I've been told one a-level is more work than all of your GCSEs combined. I know I can get very high grades in both physics and maths, but it might cause me to slip on biology and chemistry. Then if I do 3 a-levels and don't do physics I feel like it might depress me because I love physics and I want to study it more (especially now since we're not doing any more new learning and just doing paper 2 revision). What do you guys think?