I went to one of the Durham open days this year and although I didn't get to go to all the colleges I think the architecture and the atmosphere are probably the main differences.
The older the college the grander it is it seems. E.g, university college is in the castle and it's basically a Harry Potter set. Colleges like St Cuthbert's and Hatfield are very Victorian; not as imposing as university but still quite grand. The hill colleges (like van mildert and trevelyan) are more modern and so not grand at all; van mildert was built in the 70s I think so it has quite an institutional vibe
(I.e. The dining hall looked a lot like my secondary school's :P)
The atmosphere varies as well so the more modern colleges I found more relaxed and easy going whilst the posher ones were a bit too posh for me basically
but obviously some people like living in a Harry Potter set which is why they go for university or w/e and that's totally fine :P
The modern colleges have nicer rooms imo, the older ones are either really small or really cold or both. In some colleges you have to move out of your room every holiday because people stay in them as a sort of hotel (this is for the grander colleges that people stay in for the experience, not the hill colleges I don't think)
the grander colleges are much more traditional i.e. at university I think they have Formals every week or something like that and you have to wear your robes whereas in trevelyan they're only every couple of weeks and often they're fancy dress rather than full on robes
Hope that was some help?? I have some pictures of a couple of the colleges if you wanted to look, although I think some college websites have virtual tours