Some of your coworkers might draw this conclusion about you - but not everyone. Some, in fact, may secretly admire you for your willingness to go against the crowd and stick to your convictions. They wish they had the same courage.
Admittedly, it's not always easy as Christians to know how to react to social situations like this. You're right; we don't want to do anything that might make people think we look down on them - because we don't (or at least we shouldn't). We too are sinners, and we can't take credit for any of the blessings God has given us; they all have come from His grace. As the Apostle Paul said, "By the grace of God I am what I am" (1 Corinthians 15:10).
At the same time, however, we are called to be like Christ - both in His love and in His purity. The Bible says, "Abstain from all appearance of evil" (1 Thessalonians 5:22, KJV). Remember: No matter where you are or what you're doing, you are called to live for Christ.
Pray for those you work with, that they may face the emptiness of their lives and see their need of Christ. And pray too for yourself, that God will use you to point others to the peace and joy He alone can give.