I'd say it varies from guy to guy. To some it will matter, to some it won't. Some will like it, some won't. From my personal perspective, it sound like you're a little overweight to the point that it doesn't matter. I think many guys will agree with me in saying I like girls with "a bit of meat" on them, but I don't think that's something many girls like to hear!
As for should you do something about it... You can take comfort when people say to you "you are fine as you are, just find guys that will accept you for who you are". You are fine as you are, and the only guys worth it are the guys who like you as you are.
But if you changed, what would be different? Your weight would be lower, your body-image may improve boosting your confidence, meaning you may be more confident around guys which may further boost your confidence. I'm all for self-acceptance, but if you can improve yourself in which ever way and it's what you want to do, do it. That's if you want said change, that is.
Maybe think of it this way. Would you rather a slightly chubby, shy and unconfident guy, or the same guy, but somewhat athletic, more open and more confident in himself?