No. I'm decent at maths, but I'm hardly naturally gifted. I got an A* at GCSE, but I worked quite hard to do so, more revision than any of my other GCSE's. I now do Maths and FM, and the AS levels are really quite simple.
The A2 modules are harder, as I found that last year, I would simply get things, whereas in year 13 I'm taking a noticeable amount more time to get my head around some concepts in FP2 and FP3.
What you haven't taken into account OP, is that you find that your ability to learn improves, not just your knowledge which means eventually you'll be able to cope with FM quite well.