I don't really see the point of asking this on a forum.
It's entirely up to you. At the end of the day, it's no secret that short guys are unattractive, so if you aren't physically attracted to him, then there's no point in dating him.
On the other hand, if you are still attracted to him and like him, then why not just continue dating him?
It's completely a personal decision for you.