Just imagine, if you meet a pretty, slim girl with big boobs, who tells you her biggest hobby in life is eating salads for lunch (for health, no-dressing, +- botox), doing planks crunches whatever it is they do to get a big butt, because it will help her attract more male attention, as men are attracted to big butts and boobs, and her natural-looking makeup and pushup bra is very integral to her identity, she feels it makes her more of a woman, and she spends a hour each day for the above routines, and they make her a more confident person. And, the planks makes her more healthy too, of course. She is an active, workout type of girl, although she doesn't have time to master most other sports, but she makes up for it by tanning phototherapy. ...You might still have sex with her I guess. But the more burning question this girl likely desperately wants to know is, do you find her attractive?
I guess we do have some negative connotations to gym goers, because the gym is mostly a solitary place where people train for result - be become the ultimate beefcake (which is for health of course. +- steroids.) Aside from some men intrinsically preferring to life weights repeatedly (which can be very hard to understand for non-gym goers, as we assume it would be more rewarding to go running outdoors, or doing rockclimbing hiking skiing or other sports for strength and endurance, which offers a whole array of other benefits from socializing to cardio to exploring new places to learning new skills), we mostly have the association that these men may solely want to be buff, because they think it feels masculine, or they want to attract girls. Which gives rise to the damning associations you mentioned.
We think overcompensation, are you not confident in your masculinity so you want a boost?
If looking buff is equated to strength and makes you feel more of a man, what do you think of women? Of gender stereotypes?
Superficial, because you think appearance matters so much? (It doesn't, unless you reach obesity). Women find chefs, men sweaty and dirty kneeling at the car engine they are fixing, an athlete performing with intense concentration, or a volunteer roughhousing with children or building rescue doghouses extremely attractive. Manly. Masculine. This is because these moments showcase the perfect wedding of their strength and skill, all put to use in tasks that have potentially non-self-serving ends, which sends the instinctive message of desirable male qualities: providing, nurturing, non-aggressive, intrinsically driven, confidence and not-selfish.
Pure testosterone is associated with violence and aggression, but muscled arms doing pro-social, dedicated behaviors subconsciously reassures us with safety and protection instead. The gym effectively distills out the precise qualities women are looking for, because the muscles apparently was never the end point, rather, just the inevitable side effects of serving and providing and nurturing others. Throwing the ends out the windows, extreme bulk (that you would only see in the gym, not in the dude too busy building rescue doghouses) sends negative messages of self-serving, vanity, superficiality. While it makes you more aware of your manly strength, women are more wary of the potential damage said manly strength would do. Lean muscles are the exquisite dinnerware hinting at the even more exquisite main courses. Don't forget to serve the main courses. Or serve a dish on a stack of 20 plates. (Even if you are really proud of your 20 exquisite solid gold plates. You will look unhealthy.)
Even if we assume beefcake is truly attractive to women, that begs the question, you want to buff up to... attract more girls? Any girl? Preferably fertile? Will you think all girls share your vision of ideal male physique? (Some girls do like it but we generally don't; we like lean strong bodies, not buffy testosterone overload.) Since most females you ask will tell you that's not really what we prefer, why do you think it is? You haven't asked? Or asked but dismissed their answers? Or just assumed it because, because.. I honestly don't know.
These are quite unfair assumptions for a guy who genuinely wants to work out for fitness and to impress in one go. I mean, you might just prefer to look buff, just like you prefer to have curly hair or preferring apples to oranges, or be sexually orientated to women instead of men, and you shouldn't be judged by that, because really there is nothing bad about caring about your appearance.
But prejudice exists, and it's definitely something a gym goer should keep in mind and be aware of at least.
If you are glancing up from your sweat-drenched fringes and half finished rescue doghouse / car engine / cycling / just any other hobby you intrinsically enjoy, and mention you love working out in the gym, your gym will gain your the additional labels of persistence, hardworking and healthy. If you tout gym and bulking up as your Life's Greatest Goal, Your Identity, as evidenced by your (transient) bulging six pack, it can backfire. Life's unfair.
Anyway, next time if you really want a roll with the imaginary big butt pretty girl who intrinsically adores low-calorie salad, maybe avoid stuff like, 'I like to be muscular; I feel more manly that way. Females prefer beefcakes. Hitting the gym is really my biggest hobby in life. It's all about living healthily, and taking care of your body, motivating yourself to just work hard and stay fit. As for my taste in women, appearances matter tbh, but it's the heart that counts.' If they are considerate and mature adults at least they won't follow up with why-not-other-outdoor-healthy-sports or would-any-generic-female-do-for-your-grand-ultimate-life-purpose.
Allure and attraction is about association, suggestion, hints and teases. Superficially we find men more sensual in an unbuttoned shirt than in no shirt. For bonus the man who's confident and sensitive enough to don an apron and mittens and quietly lays a simple meal on our sickbed is much more of a man than the beefcake reveling in his manly strength and his stereotypical manly hobbies. The man who's driven and focused on his newfound rockclimbing skills or his newest piano piece is more assertive and dominant in planning his own life than the man who's preoccupied with winning female(s) approval.
Masculinity is shown, not flaunted.
That's just from my point of view, my instinctual take on gym-obsessed men that I have to consciously curb, because the people I know who goes to the gym for such reasons are all friendly, good people, albeit more simple and direct and tend to be collateral damage whenever politics and social dynamics are involved (social games sometimes viciously invoked by the guys who are socially suave enough to pick up the social cues, from the ladies' lack of enthusiasm regarding beefcakes to how to make your unsuspecting rival look bad in his/her business presentation).
Yes, my views are prejudiced and unfair, I know. But well, sexual attraction always is. (I mean, would you still do the salad girl?).