Just hours after Hurricane Katrina hit the Gulf Coast and Lake Pontchartrain burst its levees in New Orleans, most of the affected credit unions were serving members from recovery hot-sites, according to the industry's core system vendors, who found that disaster plans were being tested in more ways than many had ever imagined.
Six vendors, representing more than 100 disaster-struck credit unions, told The Credit Union Journal that only about 10 of their client credit unions had not recovered core data as The Credit Union Journal was going to press.
"Certainly this is a significant individual and industry accomplishment considering the magnitude of the event," said Dick McConnell, director of Marketing at AFTECH.
The vendors interviewed said they were not aware of clients who have had electronic security breaches directly related to the disaster, although several credit unions were still waiting to enter branches that had been vandalized to see if any computers had been stolen.
And while systems may be up, of course, it doesn't mean the member has electricity or access to a computer or branch.
The three USERS clients hit by Katrina had their systems up one day after the disaster was reported, said John Schooler, USERS president.
And all backup systems were go for the 26 hard-hit CUs partnered with Symitar's business continuity partner, Centurion Disaster Recovery, according to Doug Barton, manager of Centurion. "Our planning and testing has proved to be validated as all of our backup systems for our clients are fully operational," Barton said.
At Jefferson Parish School Board Employees CU, members could use the Internet branch and homebanking only hours after the New Orleans levee broke, and ATM, shared branching and audio response services were in full swing in ASP mode from AFTECH's Disaster Recovery Site two days later, according to Jodi Boudreaux, vice president of Operations at JPS CU.
That despite the fact that the $110-million CU's Harvey, La. headquarters is shut down.
"You never anticipate a disaster to this extreme," Boudreaux said. "But as long as you continue with your back-ups, your planning and your training, it all falls back into place."
Credit unions with hosted systems had "fewer worries" than in-house clients, according to Hugh Butler, vice president of Mercury Product Management for Fidelity Integrated Financial Solutions. For example, the $5-million Ochsner Clinic CU's online audio-response system is housed at Mercury's Charlotte, N.C. data center and was continuously operational through the disaster, Butler said.
IntegraSys' in-house clients had to wait longer to resume operations than their hosted peers, according to Anne Stauch, senior vice president for IntegraSys CUBE.
"With the exception of keeping power up to their systems, our in-house clients were up and running once they could get into their building," Stauch said. "Most had restored some level of operation within one week."
There remain some serious technology blips for Gulf Coast CUs. For example, four of the 29 displaced credit unions using the Mercury system had not yet recovered data because there were no backups and no remote access, according to Butler. "Modest assets don't provide a budget for a true hot site," Butler explained.
In addition, some Mercury clients "are simply not in a hurry to get Mercury running because they are caught up in the disaster personally," he added.
Another challenge that looms large stems from the failure of the Gulf Coast's telecommunications networks, which caused the initial disruption to CUs' electronic services, according to vendors.
"This disaster underscored how much we rely on communication technologies, as well as their vulnerability," said Schooler. "Both cell phones and land lines proved unreliable. Interestingly, text messaging from a cell phone proved much more reliable."
For LaPlace-based Louisiana FCU, a Summit Information Systems client, the communications breach brought to a halt homebanking, audio response, and ACH processing with the Federal Reserve, even though the $85-million CU had already restored operations from its data center via a gas-powered generator.
Summit reestablished ACH processing for LFCU the day after the storm, whereas audio response and homebanking were restored three days after, said Christine Pearsall, vice president of marketing and sales support at Summit.
Vendors and fellow CUs are providing substitute connections for many of the powerless credit unions via dial-up, Virtual Private Network or remote site communications.
In the wake of the storms and flooding, vendors sang the praises of their disaster planning and testing services.
"IntegraSys offers a business recovery service for our CUBE clients," said Stauch. "One of our clients that was affected by the storm told us their recent completion of the disaster recovery steps enabled them to be prepared to restore normal operations quickly."
Schooler attributes USERS credit unions' quick recovery to advance planning and the vendor's hot-sites. "USERS' Business Recovery team had established communication with each client that was potentially in the path of the hurricane well before it made landfall."
And the 26 hard-hit Symitar clients that were partnered with Centurion Disaster Recovery "came up easiest and quickest," of any of the Symitar clients, Barton said. Symitar did not provide figures for the number of impacted credit union systems that were not covered by Centurion's services.
"We did encounter other Symitar credit unions that were not adequately prepared," he added. "Although we were able to get them going, it took longer and was more costly to get them back to square one. They learned that they need an enterprise Business Continuity Plan, a professional DP vendor and that they need to test both before a disaster happens."
In particular, clients that hadn't tested tape backups found that some didn't work. "Clients that relied on Data Vaulting (tapeless backups) were better prepared," said Barton.
Furthermore, Symitar credit unions that backed-up data with hard-hit local companies were not able to recover their data, Butler added.