Federal stress tests are forcing banks to continually upgrade their technology in order to meet the shifting demands of banking regulators.

For instance, while banks are used to tracking loan losses by portfolio, many are now being asked to track the performance of individual loans over time.

And while the numbers they generate for the stress-test forms may be acceptable, regulators are increasingly asking them to show, through reams of documents, how they got to those numbers.

The upshot is that banks of all sizes must continue to invest in technology that will help them better manage data, document processes and stress-test models, according to bankers and consultants who attended a stress-testing conference in New York hosted by American Banker. And these will not always be one-time investments. Observers say banks will need to keep making improvements to their infrastructure as regulators and bankers themselves find gaps in their processes.

"The industry will spend millions to manage this process," said Daniel H. Bley, executive vice president and chief risk officer at $20 billion-asset Webster Bank in Waterbury, Conn.

Mark Levonian, managing director and global head for enterprise economics and risk analysis at Washington, D.C., consulting firm Promontory, said that banks need to do significant work on data and infrastructure — and that's not necessarily a bad thing.

"It's rarely a waste to do that — good data feeds good decision making of all kinds," Levonian said at the conference.

One challenge for banks is that they, like many other types of companies, often keep data in silos. Financial information is in one place, loan-loss data is in a different system, and compliance data is stored elsewhere. Stress tests require synchronized data from all three sources.

"To me, everything starts with data management," said Ed Schreiber, an executive vice president and the chief risk officer at Zions Bancorp. in Salt Lake City. "The industry as a whole doesn't manage its data and it's never had to. Everybody has disparate systems that don't talk to each other. Nobody's paid attention to data governance."

Schreiber advises banks to set up a centralized warehouse for stress-test data and spend the money to get it right.

"As you're growing, if you don't take time to get your systems in order, that will be a mistake further down the road," he said. Stress-test-related data includes loan performance information; current capital levels; loan-loss provisions; planned capital distributions, such as dividend payments, stock repurchases, or planned acquisitions; and local economic conditions.

Those that don't address this area will pay a high cost, he said. For one thing, where banks fail to fill in fields on the regulators' stress-test forms, regulators will provide their own values. For instance, if the bank doesn't enter a 70% loan-to-value ratio for a borrower, the regulators will enter their own number, say a 100% loan-to-value ratio.

"Your losses will be a lot higher. It will cost you money and make you hold more capital," Schreiber said.

One reason stress-testing data gathering is hard for banks is that the data requirements for stress testing are different from the traditional requirements of calculating loan losses, said Shaheen Dil, managing director at Protiviti. "Stress tests require things banks didn't think they needed."

For example, banks tend to track loan losses on an aggregate basis, and once a loan goes on to a watch list, it moves into a different system. If a client starts out as a small business and later grows into a much bigger company, that loan is moved into another portfolio. "Banks don't have a good way of tracking that," she said

Regulators are increasingly indicating it's not enough to look at portfolios of loans; banks need to predict the risk of individual loans, too.

The specific stress-testing models banks use to evaluate their loans are another area in need of improvement. Some banks create their own models, others use those provided by third parties such as Trepp.

There's a lack of general lack of model accuracy in the industry, Dil said, and a lack of appropriate model governance in many banks. "A number of banks passed [stress tests] based on numbers, but failed because of their process and governance not being good enough," she said.

Banks that rely on vendors' stress-testing models need to validate them, several at the conference said. It's also a best practice, many observers say, to triangulate estimates with different modeling approaches and methodologies, in order to challenge the models' results.

Documentation of the stress testing process is another issue that's come out of the early rounds of stress tests.

"If you don't have documentation, from the point of view of the [Federal Reserve, the Office of the Comptroller of the Currency and Federal Deposit Insurance Corp., it doesn't exist," said Zions' Schreiber. (Zions, as a holding company with eight banks of varying sizes, is regulated by all three agencies.) "You can talk till the day is long, but it doesn't exist."