You're implying people who go to the gym don't play outdoor/other sports as well. For me I got to the gym because I desire to stay in the university rugby team, its a means to an end tbh.
That's great for you, but has little to do with people who don't visit the gym.
The girl I like (well, everyone likes) is currently going out with a "rugby player" which at our school is code for gym goer/general beast. I don't blame her for that (of course, he could be uber funny too and have a generally great personality) but it does back up the view that I assume you're getting at which is the hottest girls only/only seem to go for the "buffest" guys. I'm also not appreciative of the fact that my life is sounding increasingly like the song "Teenage Dirtbag".
Like most things, I assume it depends on the girl. However, from a guy's point of view, I'd rather not go out with a girl who thinks the only type of man who can "protect her" (from what exactly, I don't know. I live in the South West of England, I don't think a lion is going to emerge and force me to fend for me and my girl's lives) has the body of Hercules. That's a bit narrow-minded IMO.
"Oh, you don't have the body of Mila Kunis, go away."