That is a bit odd, if you're not going to have sex... he may just be using you. If he isn't, then you shouldn't care what other people think and actually have a relationship if you want. But just that, may make things weird.
I don't see any problem with the age gap, in fact I prefer significantly older men myself. However, judging by your post, maybe you are too immature to deal with this sort of thing and should perhaps move on?