If they say appearance doesn't matter at all, they are probably lying, as obese guys are excluded. But if they say they are not very visual, they are not lying - men are more visual than women.
Physical attraction is important, but that is not just about 'classic' good looks. If a man is a 5 on the scale looks wise, but is funny, charismatic, confident, rich or has high social status, a woman can be 'fooled' into thinking he is better looking than he is. All the attractive girls I know in relationships are with men of this kind.
However this preference changes with age, most teenage girls are more visual than older women. Firstly because it takes some time to realize that you cannot base a relationship on looks, secondly because when women reach commitment-age, they value stability and success more than looks.
I am not very visual when it comes to men, but that does not mean I don't care about physical attractiveness - I care about it a LOT. It's just that a man's personality will always shine through. I like a certain 'style' and look though, and I need him to be in decent shape (that means not fat or anorexic, but I don't care about a six pack AT ALL). Several guys I have dated have been "a lot less attractive than me", according to friends. But of course they've been attractive to me.
Growing up, the 'hot' guys at school were confident guys from wealthy families, with high social status. They were never the best looking ones.