Financial institutions have to rethink business as usual for their investment banking units to survive.

Investment banks are facing declining returns on equity thanks to stricter regulations and higher capital and liquidity requirements, according to a new report from The Boston Consulting Group.

This could spell trouble for many of the nation's largest commercial banks, which provide investment banking services. These banks have struggled to increase profitability during a persistently low interest rate environment, with some using expense cuts and lower credit costs to prop up earnings. Others have relied on revenue from investment banking to pad results.

"There are a number of regulations we don't even want to think about," including the Brown-Vitter legislation, which calls for raising capital standards for the largest banks, Philippe Morel, a BCG senior partner and coauthor of the report, said during a presentation about the report on Tuesday. "Those are massive-destruction weapons" in terms of cutting into banks' capital markets and investment banking activity.

Future regulatory developments are likely to create more pressure on capital and liquidity and lower profitability, according to the report, called "Survival of the Fittest: Global Capital Markets 2013." Already many banks have achieved Basel III-mandated capital ratios and have reached liquidity coverage ratios above minimum targets.

But capital requirements could be even stricter than envisioned under Basel III, the report warned. Supervisors are questioning how banks measure risk-weighted assets for market and credit risks. Differences in modeling can create varying risk-weighted asset measurements on the same portfolio. This may cause regulators to become more prescriptive for modeling standards and use more conservative, standardized measures, the report said.

Pending rules on over-the-counter derivatives under the Dodd-Frank Act are also likely to affect returns at investment banks. Margin requirements on uncleared OTC derivatives will significantly increase liquidity costs.

Investment banks previously posted after-tax return on equity levels of 15% to 20%. However, at the end of 2012, the industry average ranged from 10% to 13%. The average is likely to decline another 3 percentage points from regulatory changes, according to the report. Banks will have to cut costs, increase revenue or do both to reinvigorate returns.

Faced with these challenges, some banks have decided to unload investment banking units and focus efforts elsewhere.

KeyCorp (KEY) announced in February it would sell Victory Capital Management and its broker-dealer affiliate. The deal initially raised questions because the unit generates fee income for the Cleveland company at a time when interest rates remain low.

But Victory didn't fit with KeyCorp's strategy on building strong relationships with consumers, Chairman and Chief Executive Beth Mooney said during KeyCorp's first-quarter earnings call.

Regions Financial (RF) sold Morgan Keegan and related affiliates last year after facing legal issues stemming from its management of funds hit hard by the subprime meltdown.

Yet there is still money to be made in investment banking. Citigroup (NYSE:C) reported a 6% increase in revenue for the first quarter from a year earlier, largely from its bond trading and investment banking businesses. And the investment bank units of diversified U.S. banks have an advantage over their specialist competitors, Morel said after the presentation about the report. "[Diversified U.S. banks] have access to capital and diversification of funding" that some noncommercial banks don't, Morel said.

But if Brown-Vitter or similar legislation is enacted, "this rule could actually destroy that advantage."

Companies looking to stay in investment banking need to evaluate their operations, the BCG report said. Banks need to review their products and exit lines that do not add value for their clients.

There are several business models that investment banks can follow. For example, an institution can become a "relationship expert," which means building deep and long-term ties to clients. Evidence shows a correlation between client satisfaction and share of wallet, so these firms need to act as one-stop shops to provide all of the products their clients may need.

Companies may also focus on being "advisory specialists" by providing advice to their clients' top management for mergers and acquisitions and capital structuring. These firms need deep product and sector expertise, especially in M&A, and relationship depth. Costs will mostly be driven by talent as opposed to technology needs.

Those that act and make the right decisions can ensure a healthy 12% ROE with some of the top players achieving even better results, according to the report.

"Naturally the road will not always be smooth. But those players that act now will be far better positioned than those that fail even to attempt the journey," the report stated.

Subscribe Now

Access to authoritative analysis and perspective and our data-driven report series.

14-Day Free Trial

No credit card required. Complete access to articles, breaking news and industry data.