The classic triangle. Whatever you do, something might get wrong. X will sooner or later find another woman, if you don't take care of him. Your partner may leave you, if you start an affair with X. Living with two men, can be very complicated. If you go behind the back of your partner, he might be very upset, if he finds out.
You are the only one that can decide what to do, as you are the one that has to take the consequenses of your choices. What would you prefere would be your partners choice, if he had your problem?