The Israeli Army actually tested this. They got their most trained (in unarmed combat) female soldiers to go against ordinary civilian men. They found that, at best, the women could hold off the men.
Basically, size DOES make a huge difference, except in fiction.