No, you aren't making a mistake -- not if you've prayed about your decision and sensed God was leading you there. The Bible says, "In all your ways submit to him, and he will make your paths straight" (Proverbs 3:6).
After all, God doesn't want us to be isolated from people who don't believe the way we do, or even ignore Him or hate Him. If we did that, we'd never make any impact for Christ or win others to Him. Remember Jesus' words: "My prayer is not that you take them out of the world but that you protect them from the evil one" (John 17:15).
See yourself, therefore, as someone whom God is sending to your campus -- not to be negative or obnoxious, but as a winsome and wise witness for Christ. The Bible calls us to be "blameless and pure, 'children of God without fault in a warped and crooked generation.' Then you will shine ... like stars in the sky as you hold firmly to the word of life" (Philippians 2:15-16).
Make your commitment to live for Christ firm and unshakable -- not just for college, but for the rest of your life. Then ask God to lead you to a Christian organization on your campus where you'll find fellowship and encouragement from other believers. Above all, no matter how busy you get, take time every day to be alone with God through prayer and Bible reading. "How can a young person stay on the path of purity? By living according to your word" (Psalm 119:9).