hey, don't worry! I've been to parties and I've been the only one not drinking. (I can't cos I'm on loads of medication). People don't usually notice. They're normally too drunk to notice and if they do notice, they'll probably have forgotten by the next day. And if they havent, they may just be jealous because a) you can remember what they did when they cant and b) they have to get drunk to have fun whereas you can have fun without drinking.
Go to the party. You'll only regret it if you don't go. Plus, everyone is only 18 once and if this guy wants you to be celebrating it with him, you should be pleased.