I really like it - read around the subject and everything.
I think it's just because the stuff they teach in schools tends to be really boring and/or irrelevant to a lot of people! Who is ever going to use matrices in their day to day life? Or use the poisson probability thing?
I also think that there's a general notion that maths is difficult, so people don't even try to understand if they can't do things first time.
I also think that girls in particular 'hate' it. I can't quote where it was from, but there was a study done where men were more likely to say they enjoyed and were good at maths. In my old (grammar) school we had fairly equal numbers of boys and girls, but in my new college-y met one, saying I do maths has had other girls' mouths dropping and them wanting to know why I'd do it, and they seem confused when I say I enjoy it and am relatively good at it. The class I'm in has 25 boys and 3 girls.
That was a bit off-topic, I guess.
What I don't understand is how if you couldn't read or write, you be ashamed and would never tell people that (if you could avoid it) but being bad at maths and not being able to do basic percentages (the number of people who can't work out, even roughly, a 15% tip...), adding and multiplying etc almost seems to be 'cool' and is just widely accepted. It's shameful.