It definitely is a lot harder when you're younger and have less of an independent outlook. I'd say you should make a choice depending on just how visible the hairs are - as in are they noticeable by people standing 1-2 metres away?
Anyway yeah I'm relatively up for just being comfortable and doing what you want. I think as you get older, you get more confident and accepting of yourself. I must admit, I do shave (not obsessively but relatively regularly) but I leave my arms like gorillas lool. They do look quite manly and dark but dunno, life's too short.
Obviously being smooth and hairless can make your skin seem softer and more attractive (I hate to say more feminine as clearly shaving is not natural) but any man that INSISTS on impeccable grooming at all times is not worth your time, seriously. It's quite an immature attitude to be honest. However, at the same time, if hair removal is not a particular hassle for you (time/money/pain wise) then I don't think it's a great idea to march about purposefully not conforming and having bushy armpits and legs!