When you live paycheck to paycheck, everyone (including you) likes to blame your financial woes on the tiniest expenses. However, you’ll have a harder time than you think ruining your retirement plans by buying a morning coffee.

As personal finance author Helaine Olen explains in Pound Foolish: Exposing the Dark Side of the Personal Finance Industry, the idea that you can ruin your budget with small indulgences is a tempting one because it’s something we can control. The idea, as popularized by personal finance advisor David Bach, is that if you simply skip the coffee every morning and invest that money instead, you’ll be a millionaire by the time you’re ready to retire.

However, as Olen points out, the math on this concept doesn’t add up. For example, check out this Latte Factor Calculator. Assuming a $5 expenditure every single day, a generous 8% annual return for investing that money, and a 30 year wait, the potential interest only comes out to around $171k on top of $54k in coffee savings. While that’s not chump change, you also can’t retire on it.

The broader lesson should be that reducing expenses overall is better for your retirement. Sure, you could skip your beloved coffee to save maybe $1,500 a year and put that towards a portion of your retirement, or you could opt to live somewhere that costs $200/month less in rent for the same effect. Or you could cut your cable bill in half by dropping all those useless channels. Or you could campaign for a raise from your boss and put the excess you earn towards your savings. Best of all, you can do all of these things to build an awesome retirement plan. There’s nothing wrong with saving the money you would normally spend on your coffee, but it’s not the only part of your personal finance plan. If you have the rest of your ducks in a row, you don’t have to guilt yourself over the little things.

Buying Coffee Every Day Isn’t Why You’re in Debt | Slate