I used to go to the gym when I lived at home, did weights, got myself looking more toned. But now I can't afford it/too busy, but I still play football once a week and play tennis once a week. My body has pretty much stayed the same, still got six pack and look toned, so I have no need for the gym any more.
I don't think you can just ask "what do you think of people who don't go to the gym", cos it depends if they're fat, weak, unhealthy looking, etc or if they look fine and you wouldn't know whether they went to the gym or not. Outside exercise is better than the gym anyway, it's more healthy I think!