You should decide what sort of career you want beyond the everyday tasks you'll have to do. Personally I'd say IB is vastly different from actuarial in terms of culture and working hours. IB would be more fast-paced, very long hours, higher pressure and fairly stressful, and that's why the pay is astronomical. Actuarial has fairly regular working hours and is relatively low-stress, depending on your sector/company. I work for a consultancy and usually work 7 hours a day, though I imagine as I get higher up the ladder the hours will get longer. I hear that hours in an insurance company are pretty nice.
Pay-wise, actuaries earn extremely well, especially for the hours they work, but anyone working in IB would think they were low. However, in comparison to most other jobs, actuaries earn a very high salary.
Personally I'd choose a maths degree for the flexibility. Actuarial science is well-respected in many areas and is more applicable to the real world as it applies statistics to economics, but maths is more traditional so I feel like it could be a better choice (particularly as more people understand what maths is!). You can get up to 8 exemptions from an Actuarial Science degree, but you won't necessarily. I work with someone with 7 exemptions from the LSE one. Another colleague did a one-year Master's at Cass and got 6 exemptions, so maybe that's something to think about if you later decide you want to do actuarial.
Finally, you can't qualify to be an actuary in less than three years regardless of how long it takes you to pass the exams. So if you got a job as an actuary you could just work really hard and do it all in three years (like my manager). I'm aiming to take four years having had no exemptions. That being said, you do actually progress in the job before qualification - exams are just a hoop to jump through in some ways.