That might depend a little on what you define as "top end". I believe Oxford, Cambridge, and Imperial all require FM; Warwick might but I think they take those without FM who get a higher result on STEP as part of their offer. I'm not sure about Bristol, and I don't believe UCL requires it. I can't think of any off the top of my head otherwise that might require it rather than just prefer or give lower offers without it (Durham maybe?). You'd need to look at individual entry criteria for courses. Imperial at the least doesn't seem to view negatively applicants who didn't take FM initially but take a gap year to do FM and apply in that same year, so that might be an option as well if you did want to broaden your options (you'd also have your results by then and would know whether that's a realistic route anyway).
I think the difficulty is less in that you haven't covered as much specific content as others, so much as you haven't had an experience of spending a lot more time on a broader range of mathematics. You'll be going from 33% of your studies being maths, to 100%, and that 100% is mostly very different in nature to A-level Maths (also FM, although FM is sometimes a bit closer to it). This can be a lot to take in, imo. Those who have taken FM will usually have gone from 66% or 50% of their studies being maths to that 100%, and are more likely to have gotten used to spending a lot more time working on and thinking about maths. Also as above some of the FM content is more similar to the style of degree level maths, and even if it's not required beforehand it's helpful to have had more of an experience with the topics already (you'll also probably have had more opportunities for individual attention from your teacher in school with them).
For universities that don't require FM, you shouldn't nominally be at a disadvantage as they won't expect you to have covered that material before, and they'll teach it from scratch. However you need to be prepared to put in a lot of time (in general) keeping on top of things once you actually start the degree, as you have far fewer contact hours to learn the material in compared to A-level (something you might've spent a month on in A-level you'll cover in 3-4 lectures, maybe, at uni) and so you'll consequently need to spend a lot more time independently learning the material.
As such for those universities it might be useful to spend the summer before starting going over some basic material on complex numbers and matrices (probably the major areas of FM that you'll be covering in first year). It might also be helpful to look at an introductory analysis text, such as Spivak's Calculus or any number of "gap-bridging" texts on the subject (there are many to choose among) to get an idea of how maths is done at university level (as compared to A-level). If you find that style of mathematics isn't to your taste then you may prefer something more applied along the lines of physics or engineering (where the maths is much more similar in style to A-level, and you develop more in that vein), but if you do like that approach then you will probably find degree level maths quite suitable.