Yeh I know what you mean, I think sometimes it is that you have spent too much time with this person, so that even subconsciously their little personality traits annoy the hell out of you.
Theres also those people you just dislike from the moment you met them. They have never done anything bad to you, said anything bad to you, in fact they've always been quite nice, you just hate them. I figure they secretly hate you for some reason, and you are picking up on their hate energy!