There was a collective shudder in the industry after reports that some customer-facing Fiserv software had a flaw in it that could expose consumer data.
Fiserv is one of the largest core banking system providers in the U.S. — it has 37% market share, according to the data firm FedFis — and it has long prided itself on security and stability. Thousands of banks have stuck with traditional core banking software like Fiserv's for many years, rather than switch to new products, on the grounds that established vendors put reliability and safety above all else. That premise was a bit shaken this week.
The problem has been fixed, and Fiserv is investigating what happened and how to prevent such vulnerabilities in the future.
The flaw
Earlier this week
The security researcher Kristian Erik Hermansen had discovered the issue while logged into an account at a small bank that uses Fiserv’s platform. Hermansen noticed the email alerts he received when new transactions posted to his account were assigned event numbers, and he wondered whether the numbers were sequential, Krebs wrote. Hermansen did some tinkering in his browser, changing one of the digits in the event number.
"In an instant, he could then view and edit alerts previously set up by another bank customer, and could see that customer’s email address, phone number and full bank account number," Krebs wrote.
One worry was a cybercriminal could change contact information to reroute alerts. "This would allow any customer of the bank to spy on the daily transaction activity of other customers, and perhaps even target customers who signed up for high minimum balance alerts (e.g., ‘alert me when the available balance goes below $5,000’),” Krebs wrote.
Byron Vielehr, chief administrative officer at Fiserv, said some of the assertions in the blog are correct but others are not.
The first part is right, he said. The Fiserv software in question, a messaging platform called Event Manager that according to Fiserv’s website works with its Cleartouch, Precision, and Premier core systems, provides one-way alerts to users, such as low-balance and transaction alerts.
A flaw in Event Manager allowed Hermansen to view other customers’ old alerts. Vielehr pointed out that this was hard to do — it required having an account with a bank that uses Event Manager, setting set up alerts, receiving an alert, using an HTML editor to edit the HTML script on that page of Event Manager code, and submitting that to view another person’s old alerts.
What’s not true, Vielehr said, are claims in Krebs’ column and in some subsequent articles that customers could use the vulnerability to hijack new alerts about other people’s accounts.
“You couldn’t go in and start getting alerts on somebody else’s accounts,” he said. “It’s impossible to set up alerts on somebody else’s account.”
Vielehr acknowledged that the flaw shouldn’t have existed, should have been caught in penetration testing, and was “a miss.” The company is still looking into how it happened.
He also said that to the company’s knowledge, no one besides Hermansen and Krebs has taken advantage of the flaw to view information they shouldn’t.
“We’ve done a ton of forensics work, we have looked at all the log files, the activity files,” Vielehr said. “We have no record of any consumer information being breached. None of our bank clients have said they think there’s an issue. There have been no bank reports, no consumer reports, and we were not able to find any consumer data leaking out or any impact to consumers in our review of the log.”
Fiserv’s patch
Krebs notified Fiserv of the vulnerability before writing about it. According to Vielehr, Fiserv built a patch within 24 hours of the notification. Within 12 hours after that, the patch had been pushed out to all affected hosted clients and distributed to clients to who use the affected software on premise.
Vielehr declined to share the number of bank clients affected but said it was a single-digit percentage of Fiserv’s client base. More than 90% of those clients had implemented the patch on Wednesday, he said Thursday.
“As you can imagine, we’ve been working with our clients closely and have made sure they have the right people available to download the patch, and if it wasn’t getting downloaded we’re escalating at our clients,” Vielehr said. “We’ve had a full court press on with our clients to make sure they get this loaded and tested.”
After patching Event Manager, Fiserv also ran penetration tests to check not only for the viewing-other-peoples-alerts issue but also to check for any other vulnerabilities.
Fiserv is analyzing all its customer-facing software to make sure the issue doesn’t exist anywhere else.
“We have not identified any other platforms that have a sequential ID pattern,” he said.
Asked if Event Manager was properly tested before it was ever launched, Vielehr said he was finding that out.
“Obviously this flaw did not get caught, so we’re trying to understand that,” he said. “Our first priority was to fix it and make sure our clients had installed the patch. We’re going back through all the history and controls around this platform to understand when it was tested, the results of the test, and the tools that were used for the testing. We would have expected this issue to be found as part of our testing process.”
Event Manager is old software. Vielehr said he didn’t know how old, but said it has been around a long time. Fiserv has a new messaging platform called Notifi that some clients are moving to.
One commenter on Krebs’ blog said this is what happens when firms like Fiserv outsource their code development. However, Vielehr said, this platform was built by Fiserv employees.
Alex Hamerstone — goverance, risk management and compliance practice lead at TrustedSec, a security firm that works with banks — said this type of vulnerability is not uncommon.
“It’s something we would expect someone to test for, especially something used for this type of application,” he said. “It’s surprising to have a big name company that’s in so many financial institutions have something like this. The challenge is when you have a reputable company that so many banks use, who else would you use? But they did address this quickly.”
Jake Olcott, vice president at BitSight Technologies, a cybersecurity rating company, said the issue has not affected his company’s cybersecurity rating for Fiserv yet.
“Our initial impression of this is that it was an application-logic error, an error on the part of the developers, not a vulnerability that’s being exploited,” Olcott said.
Vielehr said Fiserv makes large investments in security.
“This is an example of where we didn’t get it right,” he said. “We’ll figure out exactly what we missed and how it got missed and make sure we don’t see this problem again.”
Fiserv has not yet put anything in place to prevent similar problems from happening in the future, he said, because he does not know exactly what went wrong.
“If I had a smoking gun, and found there was a test that didn’t get run on it or there was some funny pattern that didn’t get picked up by our penetration-testing tools, we’d make that change,” Vielehr said. “We’re still in the process of going back through the forensics work to understand: How did we miss that? Was it a process issue? Was it a testing-tool issue? Did we skip a step? I don’t have the answer yet. As soon as we get to the bottom of that, we’ll make whatever changes we need to make sure it never happens again.”
Editor at Large Penny Crosman welcomes feedback at
