I don't know if your pastor is spending too much time on this topic; if you think he is, perhaps you should express your concern to him or to a member of your church board.
But I do know this: How we handle our money is of great concern to God, and it's just as spiritual an issue as anything else. Why is this? The reason is because how we handle our money indicates what's really important to us. If our goal is to put ourselves first and provide just for our comfort and security, then that's how we'll spend our money.
But if God is at the center of our lives, we'll want to use our money for His glory. Instead of spending it all on ourselves, we'll want to use it to bless others and extend God's kingdom. This is undoubtedly your pastor's concern -- and it should be yours, as well. Jesus said, "No one can serve two masters.... You cannot serve both God and Money" (Matthew 6:24).
Who is more important to you? Is it yourself -- or is it Jesus Christ? Remember, Christ gave His life for you; why do anything less for Him? Ask God to help you use the resources He's given you for things that will count for eternity. The Bible says, "Each man should give what he has decided in his heart to give, not reluctantly or under compulsion, for God loves a cheerful giver" (2 Corinthians 9:7).