It depends on the girl tbh. I don't think a guy can be 'too small' really and I'm sorry that girls have told you you're too small as I know that it won't do much for your confidence. Personally, I don't care what size a guy is. I've been with guys from around 3 inches to 7 inches and all have been a nice experience. If you're in a relationship and with a girl who cares about you then I don't think she'd have an issue if you are a bit smaller than average. It doesn't define you as a person and there are all sorts of ways to please a woman - sometimes smaller guys are even better!