If something has happened with the other person, they'd probably be very uncomfortable when you try the same with them. I.e. if they kissed, and you kissed, there'd be some tension perhaps. When they are initiating the "affair" then they'd often try to find time to communicate, meet up, etc.
It's an incredibly difficult situation to bring up, as you run the risk of ruining the relationship whether its true or not - trust will always be a major factor, and perhaps only act upon it if you have really solid ground to be thinking that way. You could perhaps be honest and explain why you feel X is cheating, say the distance in your relationship, the secrecy, the defensiveness - gently ask if X is getting feelings for someone else.