One reason you have these guilty feelings is because God has planted within each one of us a conscience—a sense of right and wrong—that is impossible to erase. The Bible says that when people feel guilty when they do wrong, they "show that the requirements of (God's) law are written on their hearts, their consciences also bearing witness" (Romans 2:15).
But I believe there is another reason why you have these feelings of guilt, and it is this: God wants to draw you to Himself. Right now, you may not believe in Him—but that doesn't keep Him from existing! And down inside, He is giving you a hunger—a spiritual hunger—that only He can satisfy. The person who is in the most danger spiritually is the one whose heart has become hardened toward God. Don't let that happen to you.
The most important thing I can tell you, however, is that God loves you, and He wants you not only to believe in Him but to come to know Him personally as your friend. And this is possible because Jesus Christ came into the world to take away our guilt and bring us back to God.
I urge you to get a Bible and discover for yourself who Jesus is and what He has done for you. Don't be blinded by your past, but ask God to show you His truth—and He will. Life's greatest joy comes from knowing Christ and following Him every day.