If consumer confidence is the top reason why the housing market has been so slow to recover, then banks' reluctance to lend ranks a close second.

Speaking Friday at a finance conference hosted by the Consumer Federation of America, Dave Stevens, the president of the Mortgage Bankers Association, said the combination of low home prices and rock-bottom interest rates has made home ownership more affordable now than at any time in history. Yet home sales remain sluggish because many lenders are so spooked by proposed new regulations aimed at protecting consumers that they are lending to only the most qualified borrowers or staying on the sidelines entirely.

"What they are saying is that they would just as soon get out of the mortgage business than make a bad loan" that could get them in hot water with regulators, Stevens told the attendees gathered in downtown Washington for the two-day conference.

Stevens and others at the conference acknowledged that new regulations are necessary to help prevent another housing boom and bust, but they remain deeply concerned about what the final rules will look like. The Consumer Financial Protection Bureau is drafting its rule on the so-called "qualified mortgage," and lenders remain worried that it will set a debt-to-income ratio that could shut out large swaths of low- and moderate-income consumers. Similarly, they are concerned that another proposed rule could set minimum down-payment guidelines, which would give lenders little flexibility in determining a borrower's creditworthiness.

"We're all waiting on pins and needles as to what [the rules] will say," Joe Rogers, an executive vice president at Wells Fargo Home Mortgage, said during a panel discussion at the conference.

To be sure, consumers' general lack of confidence in the economic recovery is still a big reason why home sales - while on the rise - remain sluggish, said Frank Nothaft, the chief economist at Freddie Mac. Though the recession has technically been over for three years, consumer confidence remains weak because unemployment is still hovering around 8%.

"Unemployed people don't buy homes - at least no longer," Nothaft said.

There are signs that the housing market is improving. Housing starts are up 26% this year over last and prices are up 8%, though Nothaft pointed out that those figures are up from "Depression levels."

The number of vacant homes is also at its lowest level in a decade, to around 400,000, which suggests that the market is poised for new construction.

Still, the general feeling at the conference was anxiety about the pace of recovery, the onslaught of new rules and the long-term role of the federal government in housing finance.

Wells Fargo's Rogers said he is worried that the CFPB is trying to "get to the perfect" on the qualified-mortgage rule as it aims to satisfy members of Congress who wrote the rule and that as a result, banks will have little leeway when making credit decisions.

As the nation's largest mortgage lender, Wells Fargo will undoubtedly feel the sting of any regulations, but Rogers said he is more concerned about the community banks that he fears are becoming "too small to play."

Small banks won a bit of a reprieve this month when regulators said they would table - for now - proposed Basel III rules that would require them to hold more capital against mortgage assets. But the added costs of other regulations could still deter them from mortgage lending.

"For many of them, the juice is just not worth the squeeze," Rogers said.

Rogers shared the podium with Sarah Wartell, the president of the Urban Institute, and Janis Bowler, director of wealth-building policy at the National Council of La Raze, a Latino advocacy group, and their discussion focused largely on the future role of Fannie, Freddie Mac and the Federal Housing Administration.

In the short term, Bowler said, she believes the housing market would recover more quickly if Fannie and Freddie were allowed to offer principal reductions on underwater mortgages. Other lenders are reducing principal on loans in their own portfolios, but Bowler argued that it's doing little to reduce the inventory of underwater mortgages because Fannie and Freddie still back the majority of outstanding home loans.

Longer term, Wartell suggested policymakers could wean the housing market off of government support by lowering the conforming loan limit.

As it stands, Fannie, Freddie and the FHA back roughly 95% of all mortgages, a rate most economists and policymakers believe is unsustainable. Wartell said that she believes the government should have a hand in mortgages for lower- and middle-income borrowers, but that loans to higher-income borrowers should be left to the private sector.

"I don't think the government should [be backing] my mortgage," she said.

Subscribe Now

Access to authoritative analysis and perspective and our data-driven report series.

14-Day Free Trial

No credit card required. Complete access to articles, breaking news and industry data.