FM isn't compulsory even for the prestigious unis. They only prefer it if you're taking a Maths related course and in some cases, courses like Economics at very competitive unis like LSE. It all depends on what course you're doing and which uni you're aiming for.
Also, you should be careful about the workload especially if you're taking an EPQ. A Levels are not comparable to GCSEs. This is coming from someone who had classes from 7:00 am to 5:00 pm while juggling 18 exams, and an extra two for uni entry in my home country. A Levels are surprisingly much more draining. If you do choose to take FM and find yourself exhausted, drop it and continue with your main three. As long as you do well in any entry exams the competitive unis may have, you're fine with three A Levels especially if you have the EPQ.