First of all, you never really LOVED him if you went outside the marriage. When you truly love someone, you refuse to hurt them (adultry HURTS). And you did NOT leave due to a lousy sex life; you aren't being honest with yourself. Every marriage has its problems and a sex life is probably at the bottom of the list for women. I guarantee that if you are honest with yourself, you'll come up with another reason you left him (the problems over his daughter may have caused great stress in the marriage).
Why would he be your "romeo" when he's calling other women? Do you not respect yourself more than to be in a relationship where you don't come first in his life? How can he be your "romeo" when he keep secrets? He apparently doesn't even love, repect or trust you enough to discuss his medical problems with you.
I'm sure you are not the only reason to blame for walking out; it usually takes two to tango and the truth is somewhere in the middle. If you are now having to check on him, you obviously don't trust him (and probably vice versa).
The relationship you describe is not one of love, respect, maturity or even caring. Ask youself why you would even WANT to be in such a relationship. I suspect you are in love with the idea of LOVE (and getting this man back), but I am confident this is a loveless relationship. Love has little do with sex. It has to do with trust, honesty, sharing, caring, respect and ALL THOSE OTHER THINGS you and this man don't have together.
If you are lonely, find an activity that makes you happy. Once you are happy with yourself and truly love yourself, only then will you be ready and willing for a healthy relationship (and it won't be with this man)! Best of luck to you on your journey.