All mature students worry about their age and not fitting in, then when you get there you find it doesn't matter so much. The mature students tend to hang around with the other mature students anyway - and mature is a wide range. I've found the 25 year olds are more likely to hang around with the 50 year olds than the 20 year olds.
One potential difficulty can be how much support you are getting at home. I'm lucky that my wife is very supportive and as long as she can see that I'm working hard on uni stuff, she accepts that I'm not able to do as much at home at weekends and in the evening, as well as accepting that I'm not earning for 3 years. Other people don't get that support at home and partners can be bitter, jealous, angry, worried about money, or just plain selfish and lazy.