Catastrophic System Failure Requires Trust, Cooperation And Late Nights
Rogue Federal Credit Union's Gene Pelham and the credit union's information services (IS) team are now recovering from a series of catastrophic computer system failures.
Pelham, the former president and CEO of Oregon Metro FCU prior to a merger, joined Rogue FCU as executive vice president on Nov. 1. His duties included oversight of IS, the tellers and the back office. In just his third week on the job, Pelham was put to stern test by a series of disasters.
On a Saturday evening, Rogue's Hewlett-Packard computer gave a warning. "We reset it, and it seemed to be working fine," Pelham recalled.
But by Monday the system had crashed. With no computers available, members were limited to $100 withdrawals, which tellers recorded on paper. Diagnostic software reported a failure of the small computer system interface, also known as the "scuzzy card." Hewlett-Packard dispatched a technician to the CU that afternoon and replaced the card.
"We had to stay until 7 p.m., hand-posting all of the transactions for Monday. We thought everything was fine," said Pelham.
At approximately 10 a.m. the next day however, as the tellers and phone center representatives logged on, the system drive crashed, bringing all electronic transactions to a standstill. Pelham said the scuzzy card malfunction masked the real problem-the system drive was ready to fail.
Raising The Limit
When Rogue's managers realized the computers would be down for an extended period of time, they raised the daily withdrawal limit to $500. Meanwhile, the repair process continued. Once the system drive was identified as the culprit, Pelham and the four-person IS team had to protect member data by systematically backing up all files.
"We had to reload the operating system, test it, then reload the data," he said. "We were there until two (a.m) Wednesday morning, then back in at 7 a.m. We had to restart the system with Tuesday's date on Wednesday because we couldn't process transactions for the current day."
"On Wednesday, we got the system up," Pelham continued. "We were cheering and jumping up and down-then we lost power because there was an accident down the street!"
Because Rogue Federal had backup generators, the power outage did not undo the hours of work the IS team had put in. Pelham said everyone who worked on operations at Rogue performed his or her job exceedingly well.
"As a new person, it was great to see the teamwork and dedication of the staff, and how committed everyone was to the members. Even when we were there until 2 a.m., no one complained, no one asked to go home. It was a considerable team effort, and the IS team kept coming through."
Pelham said he could not have planned a better emergency drill, because what happened to Rogue tested every system's emergency recovery capability. He also complimented other CUs in the area, which offered to help in any way they could.
"It was a good demonstration of the cooperative nature of credit unions."
With the system now up and running, Pelham said there was only one case where a member withdrew more money than he had in his account, and that member has made arrangements to pay back the funds.