Perhaps you could give the guy the opportunity to respond to your concerns, then decide based on his answers. Bottom line though is put yourself first. This isn't about HIM, its about YOU. No T should put his needs above yours.

I will remember you
Will you remember me?
Don't let your life pass you by
Weep not for the memories
Sarah McLachlan