It's difficult to forgive someone who has deeply hurt us but if we don't, the hurt will simply continue and no healing will take place. By refusing to forgive someone, we allow anger and bitterness to keep their grip on us, and they will poison our souls.
But forgiveness doesn't mean we pretend the deed never took place which is what this person seems to be suggesting you do. When God forgives us, He does so freely and fully (and the reason is because Christ took upon Himself the punishment we deserve). But listen: We may still have to face the consequences of our actions. God forgave King David for his act of adultery with Bathsheba but the child born of that illicit union still died, and David's reputation was irreparably damaged.
In the same way, if this individual has broken the law, then he must face the consequences for his actions. He also should be discouraged from doing this in the future but if he is simply let off the hook, that may not happen. Instead, he may commit this crime again and again until he is finally brought to justice.
My main concern, however, is with your family, and especially your children. Do all you can to put this incident behind you, and to build your life together on Christ and His love. The Bible says, "Above all, love each other deeply, because love covers over a multitude of sins" (1 Peter 4:8).