the casual thing is doing my head in... Watch
It seems like you only meet up at parties. Maybe ask him if he'd like to meet you at a coffee shop or a bar or something, but don't tell him you want to chat about something specific as he might be evasive or think of an excuse not to go. If he makes excuses not to meet up anyway (as in on more than one occasion, if you suggest alternative dates etc.) then chances are he may well not be interested anyway.
Still, it sounds like he fancies you, which is always a good start