He knew it was wrong. He was taking advantage of his position and your lack of safety. The fact he backed off could mean that he saw a confrontation with your husband as blemished.