I can't give you numbers on when a guy is too large or small, I don't even know the numbers for the guys I've slept with, that's not something you discuss.
Size matters to the extent that, obviously, a guy can be too small. But it's rare. I think about 0,6% of the population is born with a 'micropenis' which does nothing for a woman during sex.
I have never experienced a guy's penis being too small for me to enjoy sex, but I have experienced one being too big (sometimes it's just painful, in this scenario it was simply too big altogether). One ex-boyfriend was slightly on the smaller size, and yes, I remember when I started dating someone new I did like the feeling of a slightly bigger one. But sex was just as enjoyable with my ex. Please remember that 90% of women need repeated clitoral stimulation to orgasm, so your finger and oral skills are of 10x the importance of your penis size.
Vaginas don't vary too much in size, only slightly, and they can 'adapt' to penises of different sizes. Of course after childbirth it will be larger.
An average sized penis can hit the back of your cervix in the right position. There are basically no places a large penis can 'reach' which an average size can't, on a normal woman. If your legs are bent far enough back, an average sized penis can also hurt you by going too deep. There seems to be a bit of ego and illusion attached to this - I know some guys feel 'proud' when a girl complains it hurts slightly. If a girl feels really tight or she complains it hurts, she either hasn't had sex for a while, or - most likely - she's not aroused enough.
Of course some girls like the visual appeal of a large penis. But altogether, enormous penises are overrated as they make both penetrative and oral sex more difficult. My best friend's boyfriend had a very large penis, she mentioned it because it was a problem for them and it took a long time for them to enjoy penetrative sex. I think an average size would actually be of advantage to a man - you can enjoy rough sex with her without hurting her and she has an easier time giving oral sex - she can even deep throat you, which is impossibly for very large penises.