I'll be brutally honest. The answer to the question of whether men would be attracted to you without makeup on entirely depends on what you look like in the first place. But if you feel comfortable not wearing makeup, then do it. No one's stopping you. You don't have to go around making sure men will be attracted to you. There will always someone who'll find you attractive no matter what.
Personally, i'm getting sick of girls wearing a lot of makeup. Sometimes they don't know they're beautiful without makeup. It's more reassuring to know that someone naturally looks like that if you are interested in them, rather than worrying about what they will look like the next day.