Yes, this should definitely concern you. If you are a sincere Christian, you're only setting yourself up for trouble if you marry someone who not only doesn't share your faith but rejects it.
For one thing, you will face problems on a practical level if you marry this man. When you marry, you will be making your vows not just to your husband, but to God. To you, marriage is a sacred responsibility given you by God. And when you marry, you'll be seeking to honor God in your marriage. But none of this will be true for your husband, no matter how deep your love is right now.
Or consider the question of how you will raise your children. Will he support you when you want to take them to church? Will he encourage them to give their lives to Christ? How will he answer them when they ask why he refuses to go to church, even if you do? Face these problems honestly, because they won't go away.
I know it will be hard for you to face, but if you are a Christian, God's will is for you to marry a Christian. The Bible is clear: "Do not be yoked together with unbelievers. For what do righteousness and wickedness have in common" (2 Corinthians 6:14). Instead, trust God to lead you to the man of His choice, who will encourage and support you in your walk with Christ.