Nonbank mortgage lenders are chafing at a report from a government watchdog claiming they are less regulated than large banks and pose increased risks to Fannie Mae and Freddie Mac.

On its surface the report states one of the more obvious lessons of the financial crisis: Small lenders are less well capitalized than banks. Since independent mortgage lenders rely on warehouse lines of credit, their business can abruptly stop during a period of financial stress.

The report released last week by the Federal Housing Finance Agency's Office of Inspector General delved into the recent rise — and risks — in direct sales of home loans to Fannie and Freddie by independent mortgage lenders.

Lenders are concerned that the FHFA, which oversees Fannie and Freddie, will latch onto the report and use it as a reason to raise minimum net worth requirements for smaller lenders that do business with the government-sponsored enterprises.

"We are jumping through hoops every day to comply with regulations," said Mike McHugh, the president and CEO of Continental Home Loans, in Melville, N.Y. "The inspector general made it out like we have no regulations and made no mention of state regulators or the Consumer Financial Protection Bureau. I'm angry because they clearly didn't do their homework."

Kristine Belisle, a spokeswoman for the FHFA's Office of Inspector General, said the criticism from independent lenders "fails to acknowledge the breadth of information in the report."

"We stand by what we have reported," Belisle said, adding that the report mentioned in two different footnotes that independent mortgage lenders are regulated by states and the CFPB.

"Part of our job as IG is to identify risks facing the GSEs, assess FHFA's oversight of identified risks, when necessary provide suggestions for mitigating those risks, and finally, report on our findings to Congress and the American taxpayers," she said. "We reported on potential risks, which FHFA had already identified."

The inspector general's report examined the rise in sales to the GSEs by independent mortgage companies at the same time that large banks retreated because of massive repurchase requests for subpar loans.

The report highlights an astonishing statistic: since the financial crisis, Fannie and Freddie have recovered nearly $100 billion from mortgage lenders for repurchase claims on soured home loans.

Yet, the inspector general's report essentially laments the ebbing participation of deep-pocketed banks.

The 34-page report describes how the mortgage market has shifted as large banks like Bank of America stopped purchasing mortgages originated by nonbank lenders because of forced buybacks on bad loans. (Some lenders like Citigroup have recently returned to correspondent lending.)

Without the large bank aggregators to theoretically perform an additional layer of review, the GSEs now have to track potentially hundreds of smaller lenders' financial conditions and compliance with guidelines.

"Some smaller lenders and nonbank mortgage companies have limited financial capacity," the report stated. "In some instances, they are subject to less comprehensive federal oversight than larger financial institutions. Consequently, some may pose a heightened risk of financial loss to the enterprises."

Glen Corso, the executive director of the Community Mortgage Lenders of America, a trade group, said the report ignored the fact that the concentration of risk from a handful of large lenders "didn't work out too well pre-2008."

He, McHugh and other lenders claim the report fails to accurately describe the regulation of nonbank mortgage companies.

Nonbank lenders are regulated by states and their loan officers go through a rigorous licensing process and must register with the Nationwide Mortgage Licensing System. By contrast, bank loan officers employed by federally regulated depositories do not have to get state licenses — long a bone of contention between the two competing sectors of the mortgage market.

The disparity has created a skewed mortgage licensing system. An individual can be denied a state license to originate loans for a nonbank lender, but could get hired by a bank in the same state and originate loans without a license.

Rob Zimmer, a lobbyist for the Community Mortgage Lenders of America, said the FHFA and the inspector general should first determine whether direct sellers to the GSEs have higher loss rates.

"Before the inspector general makes these assertions, they need to dig into the data and find out if there would be higher losses in the event of a downturn," Zimmer said. "There has to be some way to analytically evaluate the risk."

Fannie and Freddie have put some limits on the amount of loans they will buy from independent mortgage lenders.

Last year, Fannie gave lenders a three-page description of delivery limits in the event that an individual lender's selling and servicing volume rises above its financial capacity, said Callie Dosberg, a Fannie spokeswoman. The limit in the volume of loans that can be sold to Fannie depends on the financial strength of each lender including net worth, asset quality and liquidity.

For years mortgage banks essentially enjoyed low barriers to entry since they could sell loans to the GSEs with a net worth of just $250,000. In 2010, Fannie and Freddie raised minimum net worth requirements to $2.5 million. The FHFA has in the past considered raising net worth requirements to $5 million, which McHugh says would cause some lenders to go out of business.

Since the downturn, the GSEs adopted major quality control initiatives. They now require multiple reviews of loans they purchase and securitize. Credit scores on loans sold to the GSEs now average 720 or higher. Independent lenders claim the current batch of home loans delivered to Fannie and Freddie are among the highest quality in years, in not in history.

There is also a further backstory to the rise in direct sales by independent mortgage lenders.

Before the financial crisis, Fannie and Freddie gave volume discounts on so-called guarantee fees to large banks. Small lenders were not eligible for the discounts, which skewed the market and risk toward bigger lenders. Independent lenders typically sold their loans to large bank aggregators, who cut them in on some of the discounts before then selling those loans on to Fannie and Freddie.

By 2011, with the GSEs in conservatorship, the FHFA's former acting director Ed DeMarco wanted to bring more private capital back into the market. He decided to gradually increase guarantee fees and eliminate the volume discounts to big banks. That, combined with bank buybacks for bad loans, resulted in a shift in the mix of home loans being sold to the GSEs.

Last year, 47% of Fannie's mortgage purchases came from nonbank mortgage companies, up from 33% in 2011, according to the inspector general's report. Freddie purchased 20% of its loans last year from nonbank mortgage companies, up from 8% in 2011.

From 2011 to 2013, the smallest mortgage lenders — those below the top 50 — increased their market share with Fannie Mae to 22% from just 8%. At Freddie Mac, midtier sellers saw the biggest gain during that same three year period, expanding their market share to 43% from 24%, the report found.

"Once Fannie and Freddie were in conservatorship, they followed a policy of a level playing field as far as pricing," said Corso. "The result has been that the percentage of mortgages from midsized and smaller lenders has risen quite a bit because now they're getting fair pricing."

Subscribe Now

Access to authoritative analysis and perspective and our data-driven report series.

14-Day Free Trial

No credit card required. Complete access to articles, breaking news and industry data.