No matter what they think of you, you are doing the right thing in God's eyes, and I commend you for it. When you were married, you and your wife made a vow before God to be faithful to each other "in sickness and in health ... as long as we both shall live" (or words to that effect). God takes that vow very seriously, and so should you.
After all, what if your situation were reversed -- that is, what if you were the one who had suffered a health crisis and become disabled, and she was the healthy one? How would you feel if you learned somehow that she had decided to break her marriage vows by becoming involved with another man? I suspect it would be very painful for you -- and it should be. Jesus said that once a man and a woman take their marriage vows, "they are no longer two, but one. Therefore what God has joined together, let man not separate" (Matthew 19:6).
Don't let your friends sway you, and don't be guided by their limited views. Be guided instead by God and His will -- for His will is perfect, and it is always best for us. Perhaps your example will cause your friends to examine their own lives more closely, and help them face their own need of Christ.
Your path isn't easy -- but "My grace is sufficient for you, for my power is made perfect in weakness" (2 Corinthians 12:9).