The biggest security threat banks face — malware that pervades their online and mobile banking users' devices — is being met by an increasingly sophisticated class of software called clientless malware detection.

Mobile malware increased more than 600% in the past year, according to a recent survey from Juniper Networks — the number of malicious mobile apps has grown to 276,000.

One common type of malware, the Zeus Trojan, has affected more than 3.6 million PCs in the U.S.

Malicious programs are becoming more specialized. One type of Trojan can intercept the text messages banks send their customers for purposes of dual-factor authentication and for SMS banking (this is sometimes called an SMS grabber).

Another variant targets Facebook users — It embeds itself in a link that appears in Facebook messages and fan pages. If clicked on, the link sends users to a fake bank website that captures the victim's Social Security number and other sensitive information that is later sold on the black market.

The idea that a bank can get its customers to download security software to protect their own mobile devices and computers is simply unrealistic, observers say.

"Consumers use a lot of tablets, iPads, mobile devices, TVs," says Eyal Gruner, who heads technical sales at Versafe. "You can't store something on the consumer's machine … You can't trust the end user. Our goal is to protect the financial institution from malicious activity in online banking." 

A newish crop of malware detection software looks for signs of phishing, URL redirects, SQL and HTML injections, logins from unknown machines, anomalous behavior and other signs of online and mobile banking foul play — all without requiring any software to be downloaded onto the customer's computer or mobile device.

"The bank can see if the session coming in is infected with malware, without having to have anything on the client's desktop," says Avivah Litan, vice president and distinguished analyst at Gartner. We've taken a closer look at some of these technologies.

ThreatMetrix's basic software is not bent on detecting malware but on examining attributes of a transaction for anomalies. "If someone is using a credit card from an IP address in New York, we can see if he's using a proxy and is really in Eastern Europe," says Andreas Baumhof, chief technology officer of the software company. ThreatMetrix provides real-time feedback and a risk assessment to the financial institution. The company has 1,500 customers around the world, he says. "All this information is shared across our customer base — if we see a device that's been used for fraudulent transactions at a number of merchants and a user of that device tries to log into a financial institution protected by ThreatMetrix, we issue a warning."

But forensic examination of transactions only detects certain types of crime, Baumhof acknowledges. If a fraudster completely hijacks an internet banking session to steal personal information or conducts automated wire transfers in the background, ThreatMetrix software can't see it.

In January, ThreatMetrix acquired TrustDefender, a malware detection company Baumhof founded. TrustDefender makes two kinds of malware detection software: one that requires nothing to be deployed to the end points, and another that requires a download and is typically used in corporate banking and within large enterprises.

The clientless version can inspect an online banking transaction for signs of tampering caused by Trojans and targeted attacks. For instance, the software will compare the code in a login page used in a transaction against the bank's actual login page.

"A lot of 'man in the browser' Trojans try to change the login page and add a field that asks you for your phone number," Baumhof says. Evidence of altered code on the page triggers an alert to the bank.

What if the hacker has not changed the login page? Some hackers inject code into a banking site that causes a window to pop up after login, and tells the customer there's been unauthorized access to his account and asks for a new piece of data such as a credit card number.

And some attacks are completely invisible to the end user. For instance, a cybercriminal can inject Javascript into a banking site that conducts automated wire transfers in the background. The user might try to transfer $200 to a colleague. The Trojan could intercept the transaction, change the $200 to $5,000 and change the recipient. "The trick is, you authorize this transaction, perhaps using a one-time password."

In such cases, when the software detects the suspicious behavior, it gives the bank a real-time alert. 

The financial institution then decides whether to block the user or to continue observing the behavior to understand the scheme, if there is one. "Often if there's a fraudulent transaction on an account, then the bank can go into the system and find out which device was used for it," Baumhof relates. "Then they look at all the other accounts the same device has accessed. If you know the first account was fraudulent, there's a high risk that all the other transactions were fraudulent as well."

A financial institution can set the software up such that if ThreatMetrix detects a high-risk transaction, it will send a second factor authentication token to the customer's mobile phone.

Most banks already have fraud analytics software that flags suspect transactions. "The trouble is they also flag a lot of good transactions," Baumhof says. Some financial institutions use this software to not just throw off cybercriminals, but to better understand their good customers.

In the case of malware that includes an SMS grabber that can detect and intercept text messages to a phone and steal online or mobile banking credentials or a one-time password, TrustDefender can tell that the ensuing transaction is coming from an infected computer, Baumhof says.

Where a customer is infected with a keylogger — malware that watches a user's keystrokes, ready to steal banking passwords or a Social Security number — Baumhof says the TrustDefender software can identify the keylogger malware. And when someone tries to use the stolen information, the software can see that someone is accessing the account from a different computer.

Versafe's technology uses encryption, website analysis and device ID to protect banking customers from malware on customers' devices.

The software encrypts all information at the application level, including online and mobile banking credentials, credit card data and one-time passwords. "If you're a customer of a big financial institution and you have malware on your computer, your credential will be encrypted on the fly in the app level," he says. That way, he says, a man-in-the-browser Trojan won't be able to see the password and credentials the user is typing in. Versafe's technology is embedded in the banking website or app, using Java or .net.

If a fraudster injects Javascript in the banking website that produces a malicious pop-up window, Versafe software can detect the modification to the web page and take down the fraudulent operation, Gruner says.

The software can also detect a number of Trojans, such as Spyeye, Carberp, and Zitmo on the end user's device, as well as zero-day Trojans (malware that has a brand-new, previously unseen signature and behavior pattern).

If someone has stolen a user's credentials through phishing, intercepting SMS messages, or other types of social engineering, the software's device identification feature will kick in. This will also help detect an automated transaction conducted by Javascript code.

Trusteer offers clientless malware detection software called Pinpoint that looks for signs of account takeover, the use of data stolen by malware, phishing and social engineering. (The company also offers software banking customers download on their PCs.)

Like Versafe, Pinpoint is web-based and embedded in a bank's web page or app. It can detect the presence of malware on an incoming user's device, according to Etay Maor, senior product marketing manager. "That means the user's credentials have already been stolen or maybe there's malware that will create the transaction," he says. "The device is high-risk." The software also looks for "malware footprints" in the browser itself.

Man-in-the-browser is the most prominent threat to banks, according to Maor. "We need to know how they operate, intimately — that way you can detect all these threats," he says. Trusteer has more than 100 million users covered with Pinpoint worldwide. "We have a huge team of security experts, they get alerts of suspicious behavior and create malware logic," he says.

To detect fraud that happens outside of the bank's website, such as the Facebook fraud mentioned earlier, Pinpoint looks for signs of account takeover, for instance someone logging in from a different device than usual or using a proxy server (an intermediary computer that sits between the user's computer and the Internet).

"Who uses proxies?" Maor asks. "The bad guys, to mask where they're coming from. As soon as there's a proxy, that's suspicious."

Oslo-based Norman Shark focuses on malware analysis, looking closely for signs of targeted attacks.

"Targeted attacks are where the malware author goes for the user of a specific system or users in a specific organization," says Torjus Gylstorff, vice president of global sales. "These attacks are hidden by nature, they are not noisy like traditional malware. These attacks are stealth. And the user's computer is compromised for quite a while without anybody recognizing it's compromised, which is a major source of concern for these organizations."

Norman Shark needs to have a malicious file or a URL to conduct its behavioral analysis. "We gather information about the behavior and traffic on the site, then we're able to establish if there's something specific going on," Gylstorff says.

Examples of the types of behavior the software looks for include files being dropped in system directories, could be changes to registries, files overwriting other files, files renaming themselves, and files attempting to connect to IP addresses. "We define and classify patterns and rate activity malicious or benign on a scale of one to 10 — something that's very malicious for one organization may be less for another."

The software analyzes internal or external traffic coming into a web server or core banking server and looks for signs of code injection.

Other solutions to the malware threat include behavioral biometrics, which analyze mouse movements or typing rhythms to catch an unauthorized user. Plain device ID can also help. "It isn't perfect, but if you're coming from a different device, the bank can challenge you," points out Litan.

She sees a lot of interest in fraud analytics among banks in general, to try to curb social engineering. "There's a lot of social engineering of customers' call center reps and trusted agents," she says. "The attack vector of social engineering is becoming more pervasive and problematic."