I'm pretty sure it's fear of sexual assault. If it were just fear of not being able to fight off an attacker, then smaller and weaker men wouldn't go out at night either. The problem is that women draw the wrong kind of attention from men just by being women, through no fault of their own. It also depends on where they live... there are some places where women wouldn't be as afraid to walk the streets at night as others. Even though I don't have a problem with homosexuality in principle, knowing that a guy swings that way makes me not want to change clothes in front of him, or risk finding myself in a situation where I'd have to be alone with him. I imagine that women have to deal with feeling that way all the time around straight men, worrying about sending the wrong signals.
But the thing is, I'm a lot less likely to be sexually assaulted at night by a gay miscreant, than a woman is to be sexually assaulted at night by a straight one. And even if I am, I'm not likely to get pregnant. I might feel really violated and need therapy, but that's about it.
Now, I don't know how I'd feel if I were actually strong enough to fight people off. I'm a 5'6" computer nerd with a sedentary lifestyle... I'm not exactly going to be a better fighter than a lot of women. A mugger will probably just take my money. I can always limit the amount of cash I carry and report any credit cards stolen before they're used. But despite not being very strong, just knowing that I'm not the sort of target most perverts are looking for helps me feel considerably safer at night than the average woman probably would.