I think it's a phenomenon that has developed out of social media and the internet.
Most guys who train properly at the gym are good guys because any kind of activity like that which trains discipline and sacrifice builds character. You tend to find that they aren't guys who are out drinking all the time, they are usually humble and there's a good community spirit with guys helping each other out with routines and advice and stuff.
However there are some guys who work out who are loud, obnoxious and vain and the problem is these have a high profile on the internet they will be all over social media with topless selfies in the mirror showing off their generic tattoos, they go on bodybuilding.com and try to sound cool and fit in with all the brah lingo. They are also very active on dating sites and unfortunately part of the negative stereotype with women is that most women who have used dating sites have received messages from these kind of guy with their topless selfies, usually written in some kind of illiterate form like "hey baby, u ok lol? u wanna chat? wat u up2 lol". So girls go round saying guys that go to the gym are idiots etc.
The other guys who are serious trainers and decent guys aren't really high profile on the internet as they don't do that sort of stuff as much they just go about their normal lives and jobs without needing to put topless selfies everywhere. Probably most of them have wives or girlfriends anyway so they aren't on the dating sites either.