If you have gotten a mortgage lately, the odds are good that it is backed by taxpayers. Indeed, the government supports nearly nine out of 10 loans made today. Few think this is a good situation, but addressing it will be more difficult than simply getting the government out of the way, as many assume.

The federal government stepped in to insure the entirety of the mortgage market during the financial crisis of 2008-09. The government nationalized Fannie Mae and Freddie Mac, and greatly expanded lending through the Federal Housing Administration, the Department of Veterans Affairs and the Department of Agriculture. Lending made without any government protection had all but stopped, as millions of loans made during the housing boom went sour and those holding the risk on these loans fled the market. If the government hadn't stepped in, the U.S. housing market would have collapsed, taking the financial system and economy with it.

Since the government's aggressive intervention, the housing market has begun a gradual return to health. Home sales, housing construction and house prices are all rising. As housing revives, though, the government should begin to wind down its role in the market. Taxpayers are being put on the hook for billions of dollars of loans backed by Fannie Mae, Freddie Mac and the FHA each year. While this extraordinary measure was essential in the throes of the crisis, it is less and less necessary with the market returning to health, as private lenders and investors are increasingly willing and able to take on the risk.

To date, discussions of how best to reduce the government's dominance of the market have focused almost exclusively on two levers: reducing the maximum size loan that Fannie, Freddie and other agencies will support and increasing the fees charged for this support. Policymakers and others have taken this approach because they believe that private investors cannot compete with the government in the current environment. If correct, then enticing these investors into the market would only require backing the government out of it, either by limiting the loans the government can support or making those loans much more expensive.

But investors are not shying away only because they can't compete with the government. They face numerous other barriers that will keep them out even if the government recedes. First among them is a deep suspicion of the parties they depend on when investing in mortgage-backed securities. They depend on credit-rating agencies to assess the creditworthiness of their investments and upon lenders to underwrite and service the loans that stand behind these investments. Both of these players failed to perform their roles competently during the crisis, and investors will remain deeply wary of returning to the space until their confidence is restored.

The second barrier to the return of private capital is the significant uncertainty that policymakers continue to generate in the mortgage securities market. One source of uncertainty is the rule requiring financial institutions that pool and securitize mortgages to retain a portion of the credit risk on those loans after selling the securities to investors. Part of the financial regulatory reform enacted under Dodd-Frank, the measure is intended to reduce risk in the system by forcing securitizers to retain some skin in the game.

How this rule is implemented will have a significant impact on the terms under which investors return to the market, yet four years after the passage of regulatory reform the rule remains unfinished. This delay has frozen the private mortgage-backed securities market in place as investors await the rules of the road that will impact how and where it makes sense to invest.

Another source of policy uncertainty is the prospect that some local governments may use eminent domain to seize the mortgages of distressed homeowners. Understandably frustrated with the ineffectiveness of lender and government efforts to address the struggles of their residents, these communities are threatening to seize the mortgages of underwater homeowners to reduce the principal balances on their loans.

Whether this would actually serve the interests of these communities – and there is reason to believe that on balance it would not – the effect on investors is chilling. Faced with the prospect that local governments might actually terminate their investments unilaterally, many investors are thinking twice before stepping back into the mortgage market.

So there remains a thicket of barriers to clear away before investors are prepared to return in earnest to the mortgage market. In the absence of these investors, the only lending that will be done beyond the border of government support will be provided by a few large banks. But they will make loans mostly to the wealthy, who are promising clients for other bank products, and those whose credit is so pristine that they pose almost no risk. If the government pulls back from the mortgage market before these issues are resolved, then they will not pull in private capital as intended but simply shrink the market, perhaps dramatically. This could threaten the housing recovery and the broader economy.

The bottom line is that policymakers should focus less on the government's retreat from the mortgage market and more on creating the kind of investment environment needed to attract significant private capital. Only then will we be able to strike a healthier balance between private capital and taxpayer risk.

Jim Parrott is a senior fellow at the Urban Institute and former senior advisor on housing policy with President Obama's National Economic Council. Mark Zandi is chief economist for Moody's Analytics