Frankly, the reason you and I feel guilty when we do certain things is because we are guilty. We feel as if we've done wrong because we have done wrong.
But why do we feel this way? Although I realize you don't believe it right now, the reason we feel guilty when we do wrong is because God has put within each of us an inner sense of right and wrong what we call a conscience. Our conscience is like an alarm bell, warning us when we do wrong and alerting us to its dangers. An animal has no conscience, but we do, because we were created in the image of God. As I've traveled the world, I've discovered that these feelings of guilt are universal.
The problem is that we can ignore our conscience, or even deny that it exists. But when we do, over time our conscience becomes dull and insensitive, and we begin to lose our sense of right and wrong. The Bible warns us not to be like those "who suppress the truth by their wickedness" (Romans 1:18).
I pray this won't happen to you. Instead, I pray that you will stop and ask yourself why you refuse to believe in God. Is it because you have honestly investigated the evidence? Or is there another reason? Many atheists, I find, reject God because they want to run their own lives and don't want God to interfere. But God loves you, and life's greatest joy comes from knowing Him. Open your heart and life to Jesus Christ today.