I realize that what your father did was very hurtful to you and your brother, and those hurts are still with you. Just as we may carry a scar from a childhood accident on our body the rest of our lives, so we often carry emotional scars from a childhood trauma on our minds and hearts for years afterward.
But listen: If you refuse to meet your father now, won't you always wonder why he wanted to see you? For example, what if he has finally realized how selfish and hurtful he was, and he wants to admit it to you and seek your forgiveness before he dies? Wouldn't it heal at least some of your hurt to have him tell you he is sorry? This may not be his reason, of course—but wouldn't it be better to find out? I think so.
This, however, raises a serious question: If he asked for your forgiveness, could you give it? It's hard to forgive someone who has hurt us deeply—and yet an unforgiving spirit is like a poison to our souls, and we need to face it and seek God's help to overcome it. The Bible says, "A man's wisdom gives him patience; it is to his glory to overlook an offense" (Proverbs 19:11).
This is why I urge you to turn to Jesus Christ and ask Him to come into your life. He can give you a new ability to love others, and he also can help you forgive the past—just as He has forgiven yours. Open your heart to Christ today, and then urge your father to do so as well.