Let me be as direct as possible (even at the risk of offending you) because you're about to make a mistake that will never bring you the happiness and security you are seeking. It would be wrong for me to do otherwise.
My comment is this: God did not bring you together (no matter what you feel), nor is it His will for you to break your marriage vows and turn your backs on your families. God never tells us to do something that is contrary to His Word, and His Word is very clear: Breaking your marriage vows would be wrong. Jesus said concerning marriage, "Therefore what God has joined together, let man not separate" (Matthew 19:6).
You see, when you were married you made a solemn vow a vow not just before those who were present at your wedding, but before God. You vowed that you would be faithful to your spouse as long as you both would live, and God takes that vow very seriously. This is one reason why God said, "I hate divorce" (Malachi 2:16).
Right now, you're facing a major crossroad a decision about which way you will go. My prayer is that you both will turn from the path you are now on and recommit yourselves to your marriages. More than that, I pray you will commit your lives to Christ and His will. No, I know it won't be easy, but in the long run it will be best both for you and your families.