Sorry, I am just ranting.
Perhaps, I do not mean to come across as childish though. Trust me, if you knew him and I wrote all the things I do not like about him, you would see that I am not the problem.
You're focusing on all the negative stuff. Come up with a list of positive things about him - that have nothing to do with you, and then compare it with the negative things that have everything to do with you. You need to be a bit more objective here. What's he like with others, how does he treat his wife, are they happy etc.
I can't knock him for doing things for my family, things that I can't do, as I do not have the same skill set. As you said, perhaps I should consider all his good qualities, but when you dislike someone, you can't see them as being good. Maybe I do have the problem, but I know he will drive a wedge between my family and me.