In the fight against fraud and cyberattacks, banks are finding the rules, in a manner of speaking, no longer apply.

At one time, banks could set up software to look for known signs of fraud and security breaches. For instance, expense checks cut on a Sunday would trigger an alert, as would payments and invoices that don't add up to the contracted amount. Network traffic coming from malicious IP addresses could be blocked. If a user based in Washington made transactions in Tokyo, a red flag would be raised.

Today, sophisticated fraudsters know all the rules the banks set and how to get around them. In the new order, in which cybercriminals are several steps ahead of their bank victims, the banks have to closely watch behavior — of people and of software — to observe anomalies of all kinds, and then figure out what those oddities mean.

"We're moving fast away from signature-based systems and rules-based systems, because the bad guys have learned what these signatures and patterns look like and they're creating new ways to hide," said Ed Ferrara, a principal analyst at Forrester Research. "So what we need is a finer-grained view of the behavior, then we need the background baseline to decide if this behavior is outside the norm."

Many vendors are moving to behavioral science and Big Data to detect fraud. "All of these systems use statistics to give you some indication of how far out of the norm this behavior is," Ferrara said. "Is it one in five? That's probably OK. Is it one in 10 billion? It's probably not OK."

Software such as ThetaRay, BAE's NetReveal and IBM's Watson can ingest many kinds of data -- network traffic, mobile app traffic, core banking transactions — and look for suspicious behavior that could indicate cybersecurity breaches or fraud, using pattern definition, pattern matching, and anomaly detection.

The approach is like setting up a camera on a street corner and watching all the traffic go by, Ferrara said. "What you'll see is patterns," he said. "Local residents who drive their kids to school, a delivery truck that arrives Thursdays at 3:00. You'll develop a baseline of activity, and if you see something that's outside of that — let's assume an armored car comes through that's never been seen before on that corner — that would be triggered as an anomaly."

The software casts a wide net over a broad set of data elements, looking for indicators of compromise and attempted breaches.

Replacing Rules with Math

The startup ThetaRay was founded by two mathematicians, Amir Averbuch of Tel Aviv University and Ronald Coifman of Yale, who spent seven years developing algorithms that can quickly process huge amounts of data.

"They decided to solve a problem: can you look at data you've never seen before and find something that you don't even know you're looking for?" said Mark Gazit, ThetaRay's CEO. "After 15 years, they built algorithms and patterns and acquired technology from universities to protect critical infrastructure." The founders applied these algorithms to cybersecurity, designing a threat detection solution for the protection of critical infrastructure like power plants, dams, command and control systems, and telecommunication networks. An early client was General Electric.

The company tells its customers to give it all their data. "Don't bother thinking, is it relevant or not?" Gazit said. "Because if you think about what's relevant or not, you're already biased, and you might lose the most important information. We look at all the data, compare all the parameters to the threats. We used to think what we're doing is looking for a needle in a haystack. But actually, what we've found is we're looking for a needle in a needle stack, because they all look the same, but only one of them is dangerous."

The company says it is working with several major global banks that it doesn't have permission to name publicly. (Bankers generally are reluctant to discuss cutting-edge security measures, for fear of tipping their hand to criminals.) ThetaRay says its software can uncover fraud, money laundering, ATM hacks, zero-day attacks, and advanced persistent threats by simultaneously analyzing security and operational data, without requiring rules, heuristics or signatures.

"ThetaRay will say, 'OK, for the last six months, this is what normal looks like, now I'm seeing something new and different, is it normal?' " Ferrara said. "Then you have to go figure out if that's a new normal or an indicator of compromise."

IBM's Watson, which beat human contestants on "Jeopardy!" four years ago, provides a combination of search, natural language processing, business intelligence and artificial intelligence tools. It can comb millions of scanned documents, web pages and transactions to study examples of fraudulent and innocent behavior and figure out how to tell the difference. "How does the machine learn what is a good set of behaviors?" Ferrara said. "The machine goes off and figures out how to do it itself."

BAE Systems' NetReveal software also uses machine learning and profiles consumer behavior. "From a financial institution perspective, we look at how individuals interact with products and how they use those products and services within the network they belong in," said Dena Hamilton, vice president of the vendor's business solutions group. "We provide customers the ability to have alerts based on suspect transactions or payments, and see how those transactions and payments look within an overall environment within that consumer's network."

NetReveal looks at customer, account and payment activity in the context of past behavior. For example, a customer might send a $3,000 wire transfer or deposit that looks fine. But because NetReveal has been watching the customer's behavior for a while, it would recognize if the transaction is significantly larger or smaller than is normal for that person.

"When people create rules in a traditional fraud environment, most only see in one or two dimensions," Hamilton said. "In an analytical environment, we're able to provide multiple detection approaches based on the type of fraud we're trying to identify, using many dimensions and allowing advanced analytics to determine what is suspect."

Finding Mortgage Fraud

In one case, ThetaRay's software was used to find fraud in a batch of mortgages originated online, according to Gazit.

"When we looked at those loans, they look very normal," he said. Then the software looked at correlations between three fields: borrower age, transaction amount and type of loan. The type of the loan was mortgage, the amount was the average and the age range was 16-19.

"Each one of those fields by itself was OK. That's why the transaction passed through the fraud detection system," Gazit said. "But when you combine the fields together, you realize you gave mortgage loans to minors. Somebody discovered a way to bypass the age-checking algorithms or somebody broke into the system and changed it. We detected this in real time. Without us, the bank would never have detected it."