CIOs Learn Lessons About Importance Of Communications Redundancy
Katrina and Rita's heavy-handed message to credit union Chief Information Officers: Keep open the lines of communication.
Members across the Gulf Coast were shut out from Internet banking, audio-response and ATMs anywhere from hours to days as public communication networks were blown away or submerged after August and September's Gulf Coast disasters.
However, after a bit of technological patchwork, the same electronic services quickly became a beacon to members as brick-and-mortar drowned.
"Katrina exposed us to a whole new set of disaster issues," said Jeffrey Crutchfield, vice president of Information Technology at Members First CU of Florida.
Data communications lines for Members First's ATM, debit and Internet banking services ran through New Orleans and were shut down by Katrina, even though the CU did not suffer a direct hit, Crutchfield explained.
Whereas many CUs proved their mettle after Katrina and Rita by successfully using backup media to restore core system data after the season's hurricanes, they weren't prepared to use backup communications to sustain electronic services.
"Our customers would have had little outage time (after the storms) if they had had ISDN backup circuits," confirmed Brennan Baas, vice president of Client Services at Cavion Plus, providing websites and Internet banking platforms to 1,400 CUs nationwide.
The $139-million Members First has used ISDN backup since the 1990's, but took Katrina as an opportunity to add Internet VPN backup to its storm recovery and disaster plan, continued Crutchfield. Thus, after Katrina, members were allowed "full access" to Internet banking with "very little down time."
"In addition to having dedicated ISDN routes built into each online system, our plan now includes the use of Internet VPN's as a backup to ISDN, allowing for more depth in the redundancy," he said.
Listening To Mom
EECU has also listened to Mother Nature. A tornado hit Fort Worth, Texas in 2000, prompting EECU to "take disaster recovery very seriously," said Bill Burrows, CIO at the $576-million CU. Hence, "we have a fairly sophisticated network backup," Burrows continued.
During an outage, each branch automatically dials up on an ISDN Basic-Rate Interface to connect to the EECU backup server within seconds. And ISDN is also used to support Automated Teller Machines and electronic bill payment, whereas credit and debit card processing is backed by Internet VPN, he said.
In addition, EECU recently abandoned its former point-to-point star configuration in favor of an Asynchronous Transfer Mode (ATM) network to provide voice and additional data backup. The ATM backbone will automatically fail over to EECU's disaster recovery site in the event of an outage.
Paired with a redundant telephone network this fall, the ATM technology will "allow survival of phone capability and enhance data survivability," said Burrows.
Though the investment in redundant technologies can be dear, Burrows offered a tip: "In terms of cost, we have tried to be as frugal as possible. We bought our backup Symitar and web servers used for less than 50 cents on the dollar. My staff kids me about it, but I constantly scan eBay for bargains for our DR site."
Crutchfield and Burrows emphasized the fact that backup requirements could be different for each CU. "What we did would be inadequate, for example, if we were on the Gulf Coast," Burrows said. "We set up what we thought was best for EECU."
For more info on this story:
* EECU at www.eecu.org
* MembersFirst CU at www.membersfirstfl.org