Yes, I would say it's dishonest to both you and him; we know when someone, who we love and lust after, isn't physically attracted to us and it plagues the back of our minds constantly.
Coming from personal experience; I dated a guy who wasn't sexually attracted to me. After a year of dating I noticed he complimented other women for their looks but hadn't once complimented me. I asked him why this was and he bluntly said "because I don't find you physically attractive", I asked him why he was dating me and he said "because I like your personality". I was crushed by this; someone who I loved and lusted for didn't feel the same way yet I stayed with him for another year before breaking up due to lack of affection.
He didn't like being touched by me (I loved his bum and would squeeze it whenever I could yet he always made a comment about it and how I shouldn't "touch what I couldn't afford"), very rarely held my hand (he would do it for a while before breaking off) or kissed me and we only had sex 4 times in our 2 years together.
There is no point in being in a relationship where you are not fully attracted to the person; not only do you hurt the person you are dating, you are not being true to yourself either and you should do yourself and him a favour and break up. Let him find a woman who is sexually attracted to him and his personality and you find another man who you can love fully too.
Do they actually matter?