For nearly four years, Sue Ozburn would check every night that the hardware housed in a tiny computer room inside Cashmere Valley Bank's headquarters had not been reduced to a smoking hulk.
The miniscule space, which had been retrofit inside the main branch and walled off from the rest of the bank a few years prior, contained the entirety of the bank's operations.
It was immediately apparent to Ozburn when she joined Cashmere as CIO in 2005 that the bank faced unacceptable levels of risk for major outages due to overheated equipment, unless drastic changes were made. The computer room was simply too small to be kept sufficiently cool, because it lacked space for heat to dissipate. Servers had already gone down after the room's cooling unit expired trying to reduce the hot temperatures the hardware produced.
"We never lost our core system in there, but we did lose some non-critical servers from time to time," Ozburn says. "Several times the cooling unit would go down and the room would become very warm. So the risk was very real. It bothered me enough that I'd actually come in every day of the week to check on things. If we had come to town for dinner I'd stop at the bank before I went home."
The downtime risks were compounded by another problem: The bank's business continuity plan made it unlikely Cashmere could restart its operations quickly enough to satisfy customers or regulators in the event of an outage: Cold backup sites that Cashmere had contracted to use, located in Minneapolis and Monett, Mo., and run by the Centurion unit of Cashmere's core banking provider, Jack Henry & Associates, were thousands of miles away from the bank's Cashmere, Wash., headquarters: too far to travel for a speedy recovery.
"In a real disaster recovery situation, getting a flight, flying to the location, reloading the data and opening up some sort of communications back to the bank to transmit it; the recovery-time objective could not be met," Ozburn says. "If you say in your business continuity plan that you're going to be able to recover in 12 to 24 hours but you're flying all the way to Monett, San Diego, Minneapolis; that isn't a practical plan in my opinion. I don't believe I could convince anybody from the FDIC that that's a practical plan."
So Ozburn looked closer to home and found IT-Lifeline, a Liberty Lake, Wash.-based disaster recovery solutions provider. The location of the vendor's recovery center - a 183-mile drive east from Cashmere - was "far enough away that regional disasters would never hurt us, and it's close enough that it's practical," Ozburn says. "We don't need a plane; we need less than a tank of gas."
The bank decided in September 2008 to sign up with IT-Lifeline. Last month, the vendor announced a formal partnership enabling Jack Henry and Centurion customers located in the Northwest to recover at IT-Lifeline's Liberty Lake facility.
For core banking recovery, IT-Lifeline houses an IBM iSeries AS/400 machine for Cashmere on the Level 2 portion of the Liberty Lake recovery center that's ready to be loaded with the bank's backup tapes if the need should arise. All underlying software and systems values are on standby at the warm site. Windows hardware covering the bank's non-core but critical operations like email stands ready in case of emergency too. A software agent sends data from non-core systems every night over the Internet from Cashmere to Liberty Lake, and then the information is vaulted to a disk at IT-Lifeline's facility.
A copy of the disk is then made and sent offsite, "so we have our own backup of Cashmere's backup," says Matthew Gerber, IT-Lifeline's CEO. The vendor uses Simpana 9 from Commvault and IBM's Tivoli Storage Manager to perform the data backup.
The bolstered backup gave Cashmere the confidence to build a new 600-square-foot data center in 2009 on the grounds of the bank's headquarters nestled in the eastern foothills of the Cascade Mountains in a windowless annex staffers call the "bunker." The space is state-of-the-art, only half-full of equipment, and well-cooled. "We would not have made the decision to build a true dark data center if we didn't have such a good continuity strategy with IT-Lifeline," Ozburn says. "Dark" means the data center is unstaffed, except briefly when tapes need changing or servers require upgrading.
Cashmere is currently working on an arrangement with Jack Henry and IT-Lifeline whereby the bank's core banking data would be vaulted daily onto disk, versus backed up to tape, Ozburn says.
CASEFILE
BANK: Cashmere Valley Bank
PROBLEM: Backup facilities were too far away from headquarters for timely disaster recovery.
SOLUTION: Finding a site closer to home.
