Yes, you can be assured that your baby is now in heaven, safely beyond the sickness and pain of this world. Your grief is real, and it won't go away overnight - but take comfort in the fact that this little child is now in God's presence forever.
Whenever I'm asked a question like this, I think immediately of King David in the Old Testament. God described him in the Bible as "a man after my own heart" (Acts 13:22), and he ruled over Israel with great wisdom and courage. But as the years passed, David drifted away from God, and on one occasion he committed adultery with another man's wife, named Bathsheba. She became pregnant, and in time she gave birth to a son.
But almost immediately the child became very sick, and seven days later he was dead. During those seven days, David pleaded with God, asking Him to spare the child's life - but to no avail. David grieved for the child - but at the same time, he was confident that after his own death they would be reunited: "Can I bring him back again? I will go to him, but he will not return to me" (2 Samuel 12:23).
Thank God for His love - both for you and for this child. Thank God most of all for the hope we have of heaven because of Christ's death and resurrection for us. Is your hope in Him?