I guess he kinda qualifies as sexy-ugly. He is attractive in Sherlock because of his character but he's not my physical preference. I know a few people who are into him in a big way, though.
I think he looks like two attractive people got together and had a child, and he is the child. Sort of good looking with the high cheekbone thing going on and the eyes, but definitely unique.
But as someone else said, I love him as Sherlock. I would.
I think he only looks good in few pictures. I find his face a really odd shape and he has pretty small eyes which isnt my thing - still who am I to judge.