Taker of too many A-levels here.
As ecolier already said, 4 subjects can end up being worse for you academically, resulting in being negative in such a situation. However, if you are unsure as to which combination to take, I would personally recommend starting with all 4 for at least 2-6 weeks, after which you can drop either psychology or mathematics, as it seems you aren't sure which of the two you would prefer. You could even keep both if you decide you want to.
It may also be worth while to find someone who takes both Maths and Psychology and ask them about the subjects compared to each other, and have a look at the specifications (i.e. what will be taught) to get a better idea of which you may prefer or perform better at.
TSR is relatively anti 4+ a-levels unless the 4th is Further Mathematics, which is fair when the additional could bring you down and the "negatives" (university and employer wise) outweigh the positives, for most people. You know yourself best and whether you are most people.
I have never taken 3 A-levels so I can't help with how much 4 A-levels worth of work is compared to 3, but 5 A-levels worth compared to 4 is barely a difference in my personal experience, largely due to my 5th being Further Mathematics and sciences having naturally a lot of crossover points. Biology and Chemistry have an okay/decent amount of cross-over, and a lot of Chemistry is Mathematic ability and Algebraic manipulation, so being good at this is helpful (less so Maths at A-level, as you are only expected to know GCSE level maths). If you took Additional Maths at GCSE, that + Physics pretty much covers all of first year A-level Mathematics (dependent on specifications). I don't know anything about Psychology so can't help there.
I would not recommend taking 4 A-levels to completion if you can not score at least A*A*A*A, personally. Likewise for 5, would not recommend unless at least A*A*A*A*A. I'm a believer in no evidence is better than negative evidence. This includes A*A*AA, because this might suggest that dropping the fourth could've gotten you A*A*A*. If you fancy a subject's content, in terms of finding it interesting, remember that you can always self-teach it to yourself by looking at the specification online and using online resources, it's not a good idea to sit an exam for a subject that you think you'll do poorly on, even if you enjoy it. But that doesn't mean you can't learn the subject still.