Yes, I can understand why you feel this way–and so can God. After all, Jesus knew what it was to have others look down on Him; the Bible says, "He was despised, and we esteemed him not" (Isaiah 53:3).
But which should be more important to you: worrying what others might think about you–or helping your children learn to follow Jesus? I think you know the answer. As they are growing up, your children will face all kinds of pressures and temptations–and they will need God's help to see them through. Helping them realize that God loves them and has a plan for their lives is the most important thing you can do as a parent.
And in any case, people who truly love Christ won't look down on you. They know God loves us no matter how poor or rich we are, and they know they are called to love others in the same way. Jesus said, "My command is this: Love each other as I have loved you" (John 15:12). Ask God to lead you to a church where people truly want to follow Jesus and will welcome you.
The most important thing I can tell you, however, is that God loves you, and He wants to help you as you face the challenge of being a single mother. It's not easy to be a single parent–but you don't need to face it alone. Ask Christ to come into your life today, and then ask Him to help you become the best parent you can possibly be. May God bless you.