Firstly, what do you hope to do after university? What course are you taking now? What do you hope to get out of education?
As far as anecdotes, I took my BA in history when I was 20-24 and MA at 25. I don't think age featured at all in the experience. It might, perhaps, if you were looking for a social experience. Yet I went to learn only, and got what I wanted from it.
Education is a lifelong pursuit, so you shouldn't be apprehensive about being a few years older. Some university experiences, however, like youth itself, are only for the young.