Yes, Jesus did say, "Do not give dogs what is sacred; do not throw your pearls to pigs. If you do, they may trample them under their feet" (Matthew 7:6).
But Jesus wasn't telling us to avoid people who are in spiritual need, or to refuse to offer the Gospel to them. We are to turn away from them only after they have rejected us and show no willingness to listen to God's Word. Later, Jesus sent His followers on a mission to tell others about Him, and told them that "If anyone will not welcome you or listen to your words, shake the dust off your feet when you leave that home or town" (Matthew 10:14).
However, don't use this as an excuse to ignore these neighbors or refuse to pray for them. They need Christ -- and the only way they will hear about His love is through someone like you. The Bible says, "Remember this: Whoever turns a sinner from the error of his way will save him from death and cover over a multitude of sins" (James 5:20).
Think of the people Jesus dealt with during His ministry. Some were society's rejects; others were powerful and sophisticated. But all needed the Savior, and they all were the objects of His love. Not everyone responded -- nor will they with us. But that didn't keep Jesus from trying -- nor must it us.