Wells Fargo (WFC) and Citigroup (NYSE:C) are laying off hundreds of employees at their mortgage units, one of the most tangible signs yet of the falloff in refinancings and the painful shift underway in the mortgage market.

Wells Fargo, which originates one out of three mortgages in the U.S., is laying off 350 employees nationwide because higher interest rates have led to a dramatic drop in home loan refinances. Citi shuttered a Danville, Ill., refinancing office this week and laid off 120 workers, according to a memo sent to employees.

Mark Danahy, a Citi managing director and head of origination services, said in the memo that "lower demand during the past few months" had caused refinance applications to plummet.

"While business decisions such as these are always difficult, they are a necessary part of our strategy of remaining a competitive and strong player in the mortgage industry," he wrote.

More layoffs by banks with heavy exposure to mortgages are expected in the third and fourth quarters, experts say. The four top banks in particular hired thousands of employees in recent years to handle a flood of refinance applications as interest rates fell to record lows.

News of the cuts added to the anxious mood surrounding mortgages this earnings season. JPMorgan Chase (JPM) officials warned on July 12 that mortgage revenues could drop by 30% to 40% if interest rates remain elevated.

Fifth Third promised to cut mortgage-related expenses, and Bank of America (BAC) has said it will cut more than 400 servicing employees in Dallas.

Jane Fraser, CEO of CitiMortgage in St. Louis, told employees in an email this month that "general conditions in the mortgage market aren't as robust as they were six months ago," and as a result Citi "will need to optimize our structure."

A Wells Fargo spokesman echoed those comments Friday. "The demand for mortgage refinancing is slowing and after evaluating the current market and our business needs, we are reducing some mortgage production positions," the spokesman, Tom Goyda, said. 

To be sure, banks will realize benefits from rising rates, such as fatter margins. And there are ways to manage interest-rate risk. But the shock is immediate and the adjustments take time, as officials at SunTrust Banks (STI) explained in trying to reassure investors about its choppy mortgage results last quarter.

"Purchased volume … continues to gain momentum, though obviously not nearly enough to offset the expected refi decline," William Rogers, the Atlanta bank's CEO, told analysts Friday.

Ironically, the housing recovery from the mortgage meltdown that began in 2007 is actually hurting the industry now — but only after banks have reaped massive profits from refinances in the past few years.

Last year, banks scrambled to hire more employees to help process refinance applications. Layoffs now are simply part of the mortgage industry's typical boom-and-bust cycle in response to changes in rates.

As recently as December rates averaged 3.35%. But they spiked to 4.51% in mid-July, the highest since July 2011, before pulling back to 4.37% on Thursday, according to Freddie Mac.

Keith May, a partner at Richey, May & Co., an accounting and consulting firm, says that as refinance volumes shrink, banks and mortgage bankers will have to return to normal staffing levels.

"We're just now starting to see layoffs," May says. "It's a cyclical industry with these types of cycles that are boom or bust. When refinance volume disappears it creates a short-term problem that everyone has to work through to get back to normal levels. Refinances made up 70% of total volume last year so production volumes are going to be off significantly from 2012, and companies will be forced to start laying off staff," May says.

Frank Nothaft, the chief economist at Freddie Mac, expects average rates on 30-year mortgages to rise to the 4.6% to 4.7% range by yearend. Higher interest rates have sparked fears that the nascent housing recovery will be "choked off," he says. Home prices in some cities have risen so dramatically that "affordability is already out of reach for a typical family," he says.

"While rising interest rates will reduce housing demand, rates would have to increase considerably more before the reduction in demand for home purchases would be substantial across the country," he says.

But higher rates have significantly crimped demand for refinances, which are expected to drop 22% this year to $972 billion, from $1.3 trillion last year, according to the Mortgage Bankers Association. The trade group expects refinances will drop another 60% to $388 billion next year.

While there is a bigger appetite for home purchases now as home prices recover, industry experts say it will not be enough to make up for the past volume of refinances. Home purchases are expected to jump 19% this year to $600 billion, from $503 billion last year, the MBA says. The outlook is for home purchase to rise 17% to $703 billion in 2014.

The large banks have been criticized for charging relatively high interest rates to borrowers that qualified for the government's Home Affordable Refinance Program. The program was designed to help underwater borrowers whose loans are guaranteed by Fannie Mae and Freddie Mac take advantage of low interest rates. Moreover refinances often require very little additional work or cost.

In recent months, the top banks have closed default and loss mitigation centers and laid off thousands of employees that handled special servicing for distressed borrowers and the process of loan modifications.

JPMorgan Chase (JPM) recently closed a default operations center in Albion, N.Y., with more than 400 employees and has said it expects to eliminate up to 15,000 mortgage-related jobs by the end of 2014.

Subscribe Now

Access to authoritative analysis and perspective and our data-driven report series.

14-Day Free Trial

No credit card required. Complete access to articles, breaking news and industry data.