The other option would be the OU, which does not require A levels. However, the learning curve is very steep because they need to get you caught up and up to speed. If you find that you struggle with maths, you would almost certainly need to take far longer to complete the degree, so as to give yourself time to fully grasp the material. Either way, you would need to be very self-disciplined and willing to put in a lot of hard work.
As a mathematician myself, I am biased, but I can say that I have always found my maths studies highly rewarding, even when I was tired, fed-up and/or stuck. The best reason to study maths is because you love it and can't get enough of it. That is what keeps you motivated.