In May, a training session on a new CRM package being implemented at Grow Financial Federal Credit Union was hit with a performance hiccup that was quickly corrected. Essentially, a lot of people were running big queries against a database server that wasn't configured to handle such a computing load. The situation was atypical, because staff permitted to run such queries would never in actual practice run them simultaneously.
But because the Tampa, Fla.-based credit union had virtualized key functions of its infrastructure, James Stock, assistant vice president of network services at Grow, simply clicked his mouse button five times, which doubled the amount of RAM and quadrupled the number of processors dedicated to the system. Then he rebooted the machine. In five minutes, he had resolved the issue.
Only because the system was running on a virtual machine was such mouse-click provisioning possible. Ten years ago, if performance suddenly tanked on a Windows 2003 server, Stock would have had to shut down the machine, slide it out of the rack, pop the case off the hardware, physically install the extra memory sticks and processors, put everything back, and then reboot the system twice so it could recognize the hardware.
"Even if I was the best in the world at doing this, it still would have taken a solid 20 to 30 minutes," Stock said. "The system was put back to its original spec, but it goes to show how quickly something can be scaled in a virtual world."
Another time, back in November 2011, Stock's network engineer was testing a script to purge log files off a file transfer server. Unfortunately, he had made a mistake in the way the script ran and instead of deleting just log files, it deleted any files, including system files, such as the critical .dll and driver files from the operating system; any data files that had been transferred to the FTP server; and log files that were older than the timeframe he sought to purge. But because Grow installed in January 2011 storage arrays in its data center from Compellent (which Dell acquired in February 2011), the network administrator was able to restore the system in 15 minutes.
He did so using a "snapshot" taken the night before on the storage area network via a Dell Compellent feature called Data Instant Replay. The solution worked in concert with a virtual machine manager, a hypervisor Grow uses from VMware that sits on the physical hardware and controls application executions of the virtual machines, and is connected to the storage area network (SAN).
"He had done this early in the morning," Stock explained. "So basically all we had to do was tell the VMware system, 'Hey, don't look at this old file anymore; we got rid of it. Look at this snapshot that I took last night.' So you pick up right from the snapshot you took."
The snapshots are replicated to Grow's disaster recover facility in Jacksonville, Fla. But the original image stays local on the SAN. "So I can bring that snapshot up to bring up a server that has failed in probably 10 minutes, because it's right there waiting on the same local storage," Stock said. "The data would be a little old, but that allows me to recover to whatever that last point was fairly quickly. I could probably recover 50 or 60 servers in an eight-hour day if I had to.
"Even without Compellent, I could have simply restored the last backup server file and been back in business in about 30 minutes, because we're virtualized," Stock added. "In the days before virtualization, I would have had to reinstall the operating system, then reload the missing data from tape or disk backups, which would have taken several hours."
Compellent's automated data-tiering assigns processes to slower, larger capacity disks whenever possible, or to faster disks with less space if required, based on the throughput it expects the tasks to require. "Because you have twice as much cost and one-third of the storage space for one fast disk to one slow disk," Stock said, explaining the feature's value.
Under an original virtualization project begun nearly six years ago, Grow reduced its physical server count from 70 to approximately 14 servers. But as the network expanded, Grow added five more hosts for disaster recovery at the Jacksonville recovery site, so that only a year ago, the CU housed 19 Dell PowerEdge 2950s. Today Grow runs just seven physical servers, three of which are Dell PowerEdge R910s and four of which are R720s, rack-mounted servers that power approximately 165 virtualized servers. The Jacksonville facility now sports two R910s versus the five 2950s it used to house.
"I tremendously reduced my server footprint, but I'm accomplishing way more performance than those systems had before, on much fewer machines," Stock said.
CASEFILE
CREDIT UNION: Grow Financial Federal
PROBLEM: A need for quick, efficient data recovery.
SOLUTION: Virtualize; keep copies on the SAN; retire hardware.
