The country's biggest banks are again making handsome profits from buying and servicing home mortgages. The smaller institutions that produce the loans have not been so fortunate.

That imbalance, which many believe is likely to endure, has irked a wide range of loan originators, some of whom argue that the big banks are not only underpaying them for mortgages but also effectively driving up mortgage rates for consumers. The big banks service around 70% of the nation's home mortgages, and their margins on mortgage purchases, around 25 basis points during the boom years, are routinely four times that now.

The big mortgage aggregators "know their customers hate what they're doing," said Ken Richey, partner of Richey May, a national accounting firm for mortgage companies. "But they're going to milk it as long as they can. … There's concern as to whether there's a movement, if you will, for the big to squish the small."

Even loan sellers for whom day-to-day pricing is not the main concern are uneasy with the extent of market concentration.

Some, like the financial services company SWBC and the military lender USAA, are already moving toward keeping their own mortgage servicing, returning to a business that small and midsize companies largely abandoned years ago. Speculating about which other companies might make a similar move led to animated conversation at the Mortgage Bankers Association's recent conference in San Diego. But the levels of capital and expertise required to build and hedge a servicing portfolio are likely beyond the reach of most.

There's always been something attractive about keeping mortgage servicing rights — particularly for banks, which can use the monthly contact with customers as a chance to deepen relationships and cross-sell products. But by early in the last decade, most institutions had switched to selling their servicing. Aggressive competition among aggregators guaranteed that the sellers would receive a good price and could focus on other things.

The prices the aggregators offered plunged during the crisis. Industry failures and consolidation have left a handful of giant banks as the principal buyers of servicing rights on new mortgages, heavily limiting price competition. Though prices have recovered somewhat, along with the capital markets and broader economy, they are still considered poor. Few expect the market to return to what it was pre-bust, but there is a conspicuous gap between what big banks are paying for mortgages with servicing rights intact and what investors like the government-sponsored enterprises Fannie Mae and Freddie Mac will ultimately pay for the loans.

While hedging a servicing portfolio is tricky, "the economics of the mortgage servicing business is really quite terrific," Wells Fargo & Co.'s chief financial officer, Howard Atkins, told investors in February. "We love this business at Wells Fargo."

There's a reason for that. According to Scott Stern, president of Lenders One, a cooperative representing both independent brokers and correspondents for the major banks, there is little pressure for the surviving big banks to compete for loans.

"This is simply supply and demand," Stern said. "Mortgage bankers aren't the good guy. Correspondents aren't the bad guy. There's a high supply of loans, and you don't have a lot of correspondents."

Matthew Pineda, the president of Castle & Cooke Mortgage in Salt Lake City, said that currently aggregators will buy a plain-vanilla, 5.25% mortgage at a 25-basis-point discount to its face value. Given where securities backed by similar mortgages are trading, he said, it would be reasonable for that loan to fetch a 25-basis-point premium. But the aggregators have no incentive to undercut each other, he said.

"They've got the ability to dictate to us the price they're willing to pay," Pineda said. "They know what the competition's buying it for."

Castle & Cooke has enough capital to entertain the idea of retaining servicing, Pineda said. Doing so would mean selling mortgages directly to Fannie and Freddie, then hiring a subservicer to take care of the logistics of processing thousands upon thousands of monthly payments. In addition to bringing in a stream of income, retaining the servicing and selling directly to the GSEs would allow Castle & Cooke to avoid the hassles of an additional layer of review imposed by aggregators.

Some big names are already in the process of making at least a partial switch to subservicing. Competitors and observers say USAA recently hired Dovenmuehle Mortgage Inc. to oversee a portfolio composed of a portion of its new production, and SWBC is working out the details of a contract with Cenlar, the largest of the independent subservicers, to handle a portion of the loans it writes. USAA declined to discuss the specifics of its servicing arrangements.

SWBC's move has been in the works since last year, when it hired Kerry Dannenberg from Centex Corp.'s CTX Mortgage to become its executive vice president of capital markets. SWBC Mortgage, which originated $1.4 billion of loans last year, expects to start building its servicing portfolio this summer.

SWBC isn't as concerned about current prices — they've improved since last year, Dannenberg said — as it is about relying on a market that has become so consolidated. In its first year, SWBC expects to retain "less than 50%" of its servicing, Dannenberg said. But it's important to the company's future to have more options.

"We felt that it was undesirable for our future to depend on decisions made by those four mega-aggregators," Dannenberg said.

Adding to that discomfort, he said, was the fact that some SWBC clients are credit unions, which are not keen on funneling their client relationships to big-bank competitors.

Concern about protecting borrower-lender relationships is common among financial institutions looking to retain servicing rights, said David Miller, the head of business acquisition for the subservicer Cenlar. "Why would they sell to these guys, turn over their customers?" he said.

More institutions may be asking themselves similar questions. Ken Richey will host a panel on servicing retention at an upcoming Richey May conference. And David Stephens, CFO and chief operating officer of United Capital Markets, a Denver firm that advises small and midsize institutions on hedging of servicing rights, said such institutions are starting to realize that pricing isn't going to improve if they can't find new outlets for their production.

New entrants to the aggregation business might force the big banks to compete more on price — lowering their margins. If that doesn't happen soon, Stephens said, smaller and midsize players will seek other options, such as retaining servicing rights.

"There was a sense in the fall that [origination] volumes would be dropping and it was time to start talking about getting paid more fairly," Stephens said. "There doesn't seem to have been a reaction."

Subscribe Now

Access to authoritative analysis and perspective and our data-driven report series.

14-Day Free Trial

No credit card required. Complete access to articles, breaking news and industry data.