When Farmington Savings Bank (FSB) staged a mock disaster affecting a branch, back office operations, as well as accounting and lending, the post-mortem on its disaster recovery plan revealed weaknesses the plans' creators couldn't have anticipated.
"If more than 10 people were on the phone, then the satellite link suffered," says Jeffrey McGinnis, FSB's vp of information systems. "Also, the slow bandwidth affected printing of documents stored on our disaster recovery servers, as well as email synchronization with these servers, particularly if 10-12 people were trying to synchronize emails simultaneously."
To conduct the business continuity test, FSB, a $1 billion mutual savings bank with 13 branches in central Connecticut, had its recovery provider, Agility Recovery Solutions, supply a trailer with a generator, 20 PCs, two servers, a fax, printer and a 512Kbps satellite voice-and-data link. "We simulated processing transactions via satellite to the mainframe and servers at our disaster recovery site," McGinnis says.
The shortcomings identified during testing led FSB to upgrade the Agility-supplied satellite bandwidth to 1.5 Mbps. The bank also ordered an additional server in the trailer to act as domain controller and print server, speeding up users' network authentication and document printing.
FSB's not alone in its hands-on approach to planning for calamities, an increasing number of community banks are using the ever-more in-depth disaster simulation approach to continuity testing to making sure plans are up to snuff. And finding shortcomings in network disaster recovery plans is fairly common. Adam Quilty, Agility's Testing Manager, tells of an unnamed bank in North Carolina whose BCP staff came into Agility's office to do a test. "They didn't have a power lead for their portable backup drive, so we had to buy a lead," he says. "Also, they didn't know the password for the backup drive, nor did anyone back at the bank. So they backed up everything again, and did a new test, with a new password."
Another common issue is that clients don't tell their outsourced core processor that they are going to test a recovery link, Quilty says. "This needs a software adjustment at the processor's end so the client can hook up to them via our facility," he says. "If they didn't alert the processor, there will be a delay while the adjustment gets done."
Rodney Tyler, a supervisor at Dayton, Ohio-based Rentsys Recovery Services's mobile recovery center, says many banks also find their back-up data is not as current as it should be, typically because the bank has updated its applications and data and failed to alert the recovery service provider. "We download an image of their apps and data onto the PC we put in the trailer. So, if they've been updating at their end and don't tell us, the mirrored image we have for their PCs is out-of-date," Tyler says.
But the tests sometimes reveal that banks are spending more than necessary to handle a disaster. Dennie Clark, president of Marietta, Georgia-based BCP consultancy CyberVault Services, says the threat of failing an audit or regulatory recriminations leads some small banks to 'over-recover.' "When they test, they may find that they don't need all their staff working in the trailer from day one," he says. "They might find they only need six or seven people for the first few days."
While the FDIC and FFIEC don't mandate disaster recovery tests, Doug Johnson, vp, risk management policy at the American Bankers Association, says regular testing involving an actual simulation of recovery procedures is part of the FFIEC's Business Continuity Planning (BCP) IT Examination Handbook. Given the high stakes, the guidance provided by this document on best practices is followed very closely by banks, technology vendors and bank examiners. "Inadequate disaster recover testing may subject a bank to a harsher risk rating by the FDIC," says FDIC spokesperson David Barr.
The demand for testing is flooding the market with vendors, which is making technology accessible to small banks. "Agility and Rentsys have got their prices down to such a low level that, if a small bank doesn't do a test, it's not on pricing grounds," Clark says.
Charlotte, N.C.-based Agility charges a one-time fee of $500 for a mobile test, plus the firm's expenses. Under an annual disaster recovery contract, Agility's clients pay a base coverage fee of $275 per month plus a testing fee of $95 per month for a single bank location.
Other firms offering mobile testing include Wayne, Pennsylvania-based SunGard Data Systems' Mobile Data Center, and Atlanta, Georgia-based Jack Henry & Associates.
