If the guy isn't interested enough to be with you then fair enough. Why couldn't you have a decent but slightly secretive relationship anyway?
How, by just sleeping with you, is he stopping his ex from ever finding out? Surely she wouldn't be happy about him sleeping with other women, if what he says is true.
This just seems like another case of: he really just can't be bothered making the effort to go out with you and probably doesn't think you're worth it. But he wouldn't mind a little bit of nookie on the side, so he finds an excuse, albeit a rather elaborate one: but still doesn't really explain why having a bit of nookie on the side is ok but not a proper relationship. Any person with half a brain would see the inconsistency there.
Either he wants a relationship with you or not. Don't settle for half measures just because the guy isn't interested enough. You'll end up emotionally attached and badly hurt.