Because more often than not, the man that just "complimented" you will then approach you, invade your personal space, ask questions about you - and if you ignore him, he gets all offended and angry because, why would you ignore a bloke that just gave you a compliment? Stupid b****.
It's quite intimidating actually... Especially if there's a few of them.
It's also embarrassing. People tend to do it for attractive women, but also sarcastically. If you're somebody with low self esteem, it can be quite insulting. And, you know, then the whole road starts staring at you.
So no, the problem isn't really the whistling, at least not for me, it's the fact that it's often followed by an aggressive confrontation or is just plain embarrassing.
It's also bloody weird. Imagine if the next woman you saw in the middle of a busy space hollered "OI MATE UR FIT" and her and her friends start laughing at you. It'd probably make you uncomfortable...