I'm not a drinker or clubber at all and although it took me a while to settle I haven't had a great deal of trouble making friends once you find those who don't want to get peed up every night of the week. There were some who thought it a bit odd that I didn't drink.
In terms of activities away from drinking then Durham's very big on sport, so there is always that
Durham's nightlife leaves a lot to be desired anyway, so most go to Newcastle.
As has been mentioned you have good student theatre, plus there's the Gala in Durham and theates in Newcastle (Theatre Royal specifically) and Sunderland (Empire). If that's your thing.
THe university has good musuems, and of course you have the cathedral. They are always worth a visit every now and then (I'd rather have a night out at a museum than pub).
Buy, yeah. Sport is THE thing but there's lots else.
Of course you've got bar crawls but I don't think you'll be stigmatised. We are all adults and respect others opinions and choices in life. If you do get the odd idiot well bugger 'em. I for one wouldn't judge you negatively.