In late August, HSBC Bank USA ran into another H.S.B.C-the heat-seeking ballistic customer. For nearly a week, the North American arm of the London-based bank experienced a disastrous outage of its core processing system that failed to post transactions, held up direct deposits and locked HSBC Direct online customers out of their accounts.
The outage, which the bank blamed on disk failures in Web-site notices and emails to customers, created a whirlwind of bad press for the bank, as customers reported they were unable to access their accounts or tally balances. The nadir came when an estimated 8,000 employees for a Buffalo, NY area health system failing to receive their direct-deposit pay, forcing the bank to extend branch hours to work out arrangements, according to press reports.
Unfortunately for HSBC and its customers, dealing with failed systems and irate customers is becoming old hat for the venerable Hong Kong and Shanghai Banking Corporation.
The North American outage, which was concentrated in the upstate New York region, was the third major processing failure the global bank has suffered since 2005 (the other two occurring in the UK).
HSBC Bank USA restored its systems on Aug. 22, five days after the weekend outage occurred, promising to refund any fees or charges that customers incurred from the lack of availability to their funds. It remains to be seen if the customers' faith will be restored after such a fundamental flop.
"This just should not have happened," says TowerGroup senior analyst Virginia Garcia. Calls to HSBC to further explore the problems were not returned.
Analysts and sources familiar with HSBC speculate that the system failure likely had as much to do with a creaky, legacy batch system along with hardware malfunction, since any off-site, redundant backup system could have handled a disk crash.
One source told Bank Technology News that HSBC runs an in-house, "old as the hills" core processing system held over from the Marine Midland Bank operations HSBC acquired during the 1980s. "It's hard for disks to fail," the source told BTN. "When you consider that if it was 'kloogy' software, with lots of integration points, there could have been multiple points of failure. "It's not surprising, given the age of that software and that internal code would be holding everything together," the source says.
A core vendor executive who spoke with BTN was also scratching his head over a bank having a five-day outage. "I don't know how you pass your regulatory examinations if you don't have [disaster recovery] in place," the principal says.
Celent senior vp and banking analyst Bart Narter says delays in recovery at HSBC could also be attributable to the slow nature of batch recovery. "That's exactly what happened to Wells Fargo [in 2005]," says Narter, referring to the San Francisco institution's own two-day brush with a major systems failure affecting online access, ATMs and merchant processing across seven states.
"When you're in a batch system and you have a disk failure, what you've got to do is go back to yesterday, and get as much as you can from your last known state, update it with the logs and run the overnight batch again...that is a long, time-consuming process," says Narter.
Garcia says such a rare core meltdown will spur bankers, and examiners, to take another look into their disaster recovery and redundancy plans.
HSBC's mishap only underscores the bank's questionable tech performance with other incidents that have affected different regions and business lines in the past three years.
None of the incidents can be attributed to any single point of failure, experts say, given HSBC's variety of core and application systems running in given geographic regions from past acquisitions.
Back during the New Year's Day holiday period in 2005, HSBC UK online customers plus debit and credit cardholders were unable to transact during a busy holiday shopping day as ATMs, POS terminals and Web banking services went kaput, according to press reports.
Earlier this spring, British reports detailed how merchants and customers were howling when HSBC's e-payments systems reportedly went awry in two separate episodes.
That left some e-commerce companies screaming for make-good compensation.
HSBC also had an embarrassing public snafu by failing to renew a digital security certificate this past March-meaning UK business customers got a browser message warning them they were on an unsecured SSL page, according to Brit sci-tech paper The Register.
For good measure, HSBC later lost an unencrypted CD containing the names and life insurance policy details of more than 370,000 customers. (c) 2008 Bank Technology News and SourceMedia, Inc. All Rights Reserved. http://www.americanbanker.com/btn.html/ http://www.sourcemedia.com/
