No not at all. My boyfriend didn't start back until he was 23 after a rough adolescence that kept him from going earlier. Granted, he's also working a full-time job and he owns his own house, so he's not really getting the full "uni experience,"but he's still going and getting his degree etc. and doesn't regret it at all (he's just about finished undergrad).
Plus, I did a year at an American uni before I came here (I'm 20) and although I'm significantly older than most people, it's fine. It's just weird that they're all my little sister's age lol.