Just a few months ago, mortgage bankers were worried about a potential spike in interest rates as the Federal Reserve prepared to remove some of its stimulus from the system. Rates have, in fact, done the opposite. Yet that has some market participants just as concerned.

"The immediate cause of rates falling has been a much weaker economic picture than most economists were predicting in March," said Walt Schmidt, senior vice president and manager of structured product strategies at First Horizon National Corp.'s FTN Financial Capital Markets. "You have such a poor employment picture right now that you need another facet of the economy to take off."

Since the beginning of April, rates on 30-year fixed mortgages have dropped more than 60 basis points to a record low of 4.58% last week, according to Freddie Mac's weekly Primary Mortgage Market Survey.

On the surface, low mortgage rates would seem to be a good thing, encouraging consumers to take out a new loan or to refinance an existing one. But no such surge of buying or refinancing interest has taken place.

"Pipelines are up, but not as much as anybody would think at these levels," said Tom Millon, the chief executive of Capital Markets Cooperative, a Ponte Vedra Beach, Fla., company that provides secondary marketing services to banks.

"It's like a vicious cycle that keeps driving rates lower and lower. If people don't have incomes or values to refinance, that means the economy is in bad shape, so rates go down. And then rates go down because people don't refinance."

And with rates falling fast, there is the chance for mortgage loans in the pipeline to fall through.

"The 5.25% loan that has not yet closed is in significant danger of falling out," Millon said. "When things move that far that fast in that short amount of time, a borrower is like, 'I can get 4.25% across the street.' "

Meanwhile, investors of bonds backed by mortgages are on edge.

"Interest rate risk might be the next thing that people have to pay closer attention to," said John Jay, senior analyst at Aite Group. "Mortgages, they perform the best when there is not that much rate movement. It's when there are big rate shocks, that's when there's a problem."

At the same time, Schmidt said, "investors are continuing to look for yield and it's not there."

It's hard to tell for sure where rates will head from here, but with a sluggish outlook on the economy and increased volatility in the stock market, investors will likely continue to flock to the safety of government bonds, which will in turn put pressure on mortgage rates. (The 10-year Treasury yield is closely tied to mortgage rates.)

"My personal feeling is the stock market [is] in for a bearish market," Schmidt said. "If that's the case, I think mortgage rates could stay pretty low."

Subscribe Now

Access to authoritative analysis and perspective and our data-driven report series.

14-Day Free Trial

No credit card required. Complete access to articles, breaking news and industry data.