Many of the people we run into every day "swing" they just don't advertise what goes on behind THEIR closed doors. They are self-confident and self-aware, passing no judgement on others and they don't push their lifestyle on others. Swinging is just an added dimension of a person.
Since when does having dinner out with a group of friends, laughing at each others jokes, celebrating an anniversary - lower self-esteem?! That group having a great time next to you in the restaurant (without any sexual inuendos) could very well be swingers! (And no they're not all going home together afterwards.)
Drug abuse, prostitution, excessive drinking - these activities WILL lower self-esteem, and unfortunately can be found in the family next door, your minister's family, your coworker's daughter.
There is a problem if one spouse wants the "threesome" experience and pushes it on their spouse. You have to be on common ground in a relationship regarding sex, when you start requesting something that doesn't feel natural and true to your partner, then it becomes deviant. The husband and wife who's older children were being involved - completely out of line. Sounds like there are other issues in this marriage, he'd use the kids to side with him in a divorce as well. That's a character problem.
Lastly, Mr. Truelove is correct - you don't explain this to your kids, and if my kids know I'm having sex down the hall with my husband, I'm sure they're trying to forget that! I'm 48 years old, and my parents NEVER had sex! LOL
And no, my husband and I don't swing (as least not yet)...