If you max out retirement accounts like I do you need to be aware that you won't have access to that money until you're 59.5 years old, unless you want to pay a penalty. That's a problem if you want to retire 20 years before that. This money is great to build, but it's almost like Social Security, you can't really factor it in yet.
As a result you need to think in terms of cash flow. What are your expenses and what does your revenue need to be? And as a result where will my revenue come from and how stable will it be?
To use myself as an example. I generally spend on average about $3,000 a month on everything. When the mortgage is paid off it'll drop to a little over $2,100 a month. As a result I would need to find a way to make $3,000 off of my investments each month in order to sustain my lifestyle. Also, I would need to make sure that those investments increase enough each year to keep up with future inflation. (Hint: the less you spend the less inflation will impact you.) Another way to think about this is that you need anywhere from 25 to 33 times your yearly expenses in investable cash generating assets.
That's a big range in numbers, so think about it like this. If you've got it all in stocks yielding 3% you're going to need 33 times your expenses. If you've got it in real estate yielding 6% you're going to need far less. If you've got it all in CD's yielding 1%, you're fucked!
So me for instance, I would need some where in the range of $900,000 to $1,180,000. In all the calculations from all the different scenarios I've run I keep coming up with a very conservative figure of $1,100,000 net worth figure. As a result that's what I'm shooting for. Interestingly I already have the cash flow now to sustain my lifestyle, but it's really close and I don't want to risk something major in the event of a serious emergency.
Paying off the mortgage would make a huge dint into what I need each month. It would drop what I needed from $1,180,000 to around $832,000, which comes to about 4-5 years worth of additional savings. It would be a drop in the bucket to pay it off in 25 years when the IRA funds become eligible without penalty, but by then it'll be paid off. See the types of problems that you need to think about when you plan to retire extremely early?