This is very difficult and I understand how you feel. It is obvious that your husband is happy with his colleague and perhaps doesn't want to spend all his time with you. Nevertheless, he wants you and loves you. Of course, the more upset you become, the less time he will want to spend with you.

Try to create your own exciting life (with work, colleagues, interests and friends) which you can devote yourself to during the time your husband doesn't want to devote himself to you. You will probably not be able to change him completely but will have to accept him as hi is if you want to retain him.

You can also suggest to him that he takes you to a good restaurant, without his colleague. Then show him that you appreciate this.

To become bitter and sour will not strengthen your position with regard to his colleague.