It depends what you want to apply for at uni.
Generally, universities only tend to care about your HL subjects and how well you do in those and your overall score. Of course in practice it usually means for the most competitive universities you need to do well in all of your subjects, SL and HL, to get the required overall score.
Example: say you wanted to do a Bsc in physics at Birmingham. They ask for:
'7,6,6 at Higher Level, including Mathematics and Physics, with a minimum of 32 points overall. 7 must be in Mathematics or Physics.'
In my experience that kind of offer is pretty typical of competitive unis, like I said, they tend to ask for specific scores in specific HL subjects plus an overall minimum score. They don't tend to specify which SLs they want you to take or what specifically you need to get in each of them. Some unis might, but I haven't seen any that do.
Check entry requirements for the course(s) you think you might want to do and you should get a better idea of which unis/courses specify SL requirements.
Regarding SL maths, as you can see from the above example a lot of unis ask for HL maths for maths or science degrees. Again, you will need to entry requirements yourself because they differ between unis and courses. All unis put their entry requirements on their websites so you can see what they ask for for IB.