If you feel strongly about this, I suggest you talk with him -- or perhaps with someone who serves on your church's budget committee -- about your concern. Your pastor may not realize how some people are reacting, and he needs to know.
However, if you do this I hope you'll be open to listening to his side, because he (or the budget committee) may see things you don't see about your church and its needs (and also its opportunities). Remember: God's work requires financial support; even Jesus' little band of disciples needed people to support them financially so they could continue their ministry (see Luke 8:2-3).
I can't help but wonder, however, if your real problem may be something deeper -- and that is the place money has in your life. Money can be our servant, but it also can be our master -- and when it is, we aren't seeing it from God's point of view. God gave our money to us; we can't take credit for either the abilities or the opportunities He gave us to earn it. The Bible puts it this way: "Everything comes from you, and we have given you only what comes from your hand" (1 Chronicles 29:14).
Make sure of your commitment to Jesus Christ. Is He the center and foundation of your life -- including your money? Put Him and His will for your life first, and then ask Him to help you use your money for His glory.