Doesn't bother me. I know plenty of guys that don't go to the gym, and they are still pretty good looking. As long as they are in shape and still relatively attractive, I don't see how going to the gym would make a difference. I'm not particularly into huge muscles anyway, don't get me wrong, I like tone and some muscle, but I've seen lots of guys who go to the gym that have just taken it too far.