Well i think the old adage "you're only as old as you feel" is very true. And if you feel at 24 that acting your age means you wont really mix with some people who are only 5/6 years younger, then i'm not really sure what can be said to change this viewpoint. Your post sounds like you are almost trying too hard to fit in with the "youngsters" which means not acting like your normal self - perhaps a better approach is to not try so hard, act as you would normally do and just see what happens.
Its interesting your viewpoint on a 24 year old in your college class when you yourself were above the "normal" age of someone at college. I myself was also 24 when i decided to go back into education, and i also went to my local college with all the youngsters fresh out of school. But i don't recall running into anyone who had an issue with my age, or anyone saying "god what are you wearing?" because in reality there isn't really that much of an age gap and its surprising how much in common i had with people i met, and yes i can honestly say i made some good friends, do i care that they are 6/7 years younger then me? no, and i would suspect they don't care either. Anybody who may have had a problem with my age, is not the kind of person i would want to associate with anyway.
Most mature students get hung up on the age thing at first, and i include myself in this, but its best to just look past it, and get on with what you set out to do. I think you should try to be less concerned with what other people may or may not be thinking about you, and do your own thing, and have fun, if someone has an issue with that, then to hell with them.