You don't have to avoid, just a straight up 'I don't drink' will do it. I don't really like drinking so I just said I didn't, they went 'Fair enough' and left it at that. Some occasionally asked why and just told them I don't like it. No one had a problem with it.
You can still join in without drinking, Ring of Fire for example is very funny to watch when you're sober. Beer pong, I found someone else usually volunteers to drink for you so you don't have to. That's pretty much the only games my flatmates ever played last year.