We can't always live the way we'd like to; we all have responsibilities we need to fulfill -- even when we wish we were doing something else. One of the marks of a mature person is a willingness to accept responsibility.
A lazy person, however, ignores responsibilities, and puts their own comfort and desires first. They seek the easy way in life instead of the right way, even if the easy way causes hardship down the road. In the long run the lazy person hurts not only himself but everyone who depends on him. The Bible warns, "One who is slack in his work is brother to one who destroys" (Proverbs 18:9).
Is your wife right about you? Only you can answer this -- but I pray you will, because if it's true, you need to take steps to correct it. God doesn't want you to be lazy or irresponsible! Ask Him to help you be more disciplined, and to set aside time for the things you need to do. Make lists if you tend to let things slide; you'll find great satisfaction in crossing off each task when it's finished.
Is it wrong to enjoy ourselves and have times of relaxation? No, not at all; God wants us to enjoy His good gifts. But don't neglect God's greatest gift -- the gift of His Son, Jesus Christ. Commit your life to Him, and then ask Him to help you become the husband He wants you to be.