Already a member? Current customers are kindly asked to reset their passwords. Simply select LOGIN, then RESET PASSWORD.

BNZ Technology Reads People's Faces for Honest Feedback

Aiming to draw attention to its brand — and potentially its branches — Bank of New Zealand (BNZ) has launched a website called EmotionScan that comes with an app that reads web camera-equipped consumers' emotional reactions to varied financial scenarios, such as buying a last-minute plane ticket to attend a wedding. Participants will receive a report on their "reading" after completing the scan. The idea behind the campaign is to make personal finance more top of mind, and to prompt people to meet with a financial advisor in one of its 180 branches or over the phone for a so-called MoneyReview.

"In bank marketing, it's very hard to get noticed," says Rob Cooke, head of marketing at BNZ. "People care about money, but they don't care about banks."

The roughly four-minute EmotionScan works like this: consumers who agree to be filmed through a webcam listen to financial scenarios while the software captures their muscle movements to decode their micro-expressions. "Reactions have characteristic signs that you can see on a face," says Tim Llewellynn, chief executive and co-founder of Switzerland-based nViso , BNZ's technology partner for EmotionScan.

A raised eyebrow coupled with an open mouth — depending on the person — may suggest surprise, while a wrinkled nose could convey disgust. The degree to which mannerisms are displayed naturally depends on the individual — a machine can learn the patterns, even subtle ones.

"The face is the most powerful way to communicate emotions," Llewellynn says. "You can teach a computer to understand what a face is. …The computer learns through examples."

Beyond serving as a marketing campaign, the tool is designed to help a person understand how an emotion could impact the financial decisions he makes. "EmotionScan is meant to get people talking about money," Cooke says.

More than 6,500 people have completed an EmotionScan reading since its launch in late September. The bank collects the data in aggregate.

BNZ partnered with nViso, a Swiss emotion recognition software company, and psychologist Dr. Stuart Carr to deliver the tool.

Typically, businesses use nViso for internal customer research projects, such as whether people like a commercial. "It's the first time we are doing something like this," says Llewellynn. "Essentially what BNZ has done is open the doors for the next step. …By capturing the emotional reaction of the consumer, you can better understand them and propose alternative actions."

Such technology could help banks determine areas where people have financial issues, such as budgeting, for example, as well as where consumers show signs of suffering during a digital experience — data that could benefit product and marketing developments. (nViso makes available an API to integrate into a website or mobile app.)

To date, nViso's algorithms are drawing from a database of more than one million subjects from around the world. "The software needs to be refined to track subtle movement. …things you can barely see with the visible eye," Llewellynn says. BNZ has made a number of efforts to draw attention to EmotionScan, including a behind-the-scenes YouTube video, prompting testers to share their experiences on social media networks and making available a lighter version of EmotionScan at a bus shelter that included a webcam to take a picture of a person and a printer for participants to take home their results.EmotionScan is part of the bank's overall "be good with money" marketing campaign.

Historically, banks would — and do — collect customer feedback from focus groups, surveys and other methodology to influence marketing campaigns, digital designs and more.

The advantage facial imaging software could have over written surveys is more honest data. When a company asks a person about how he feels, he could fake his answer. Perhaps the person is less attuned to his emotional state, reluctant to share a feeling that contrasts with the majority of a test group, or simply uncomfortable discussing money.

Already in in-person studies, financial institutions are scouting out for non-verbal clues to better inform their research.

"As we observe, sometimes it's not what is being said but what is not being said that leads us to think about improvements and ways to create new functionality," said Secil Watson, head of wholesale internet services for Wells Fargo, in an interview with BTN.

To be sure, it's early days for enterprises using human-machine interaction facial technology. Even so, nViso's deal with BNZ offers an example of the broader biometric trend of new tools that get inside people's heads by measuring their sweat, breathing rates or brain activity, says Llewellynn.

nViso competes directly with Affectiva in Waltham, Mass., and more generally with traditional survey techniques. Even so, Llewellynn views facial imaging technology as something to supplement other forms of research — including neuromarketing, which measures brain activity (and a research method banks have deployed).

Many challenges lie ahead, including understanding context.

"To decode emotions is one thing, to interpret them is another," Llewellynn says. "You can't magically tell what a person is thinking. Emotions need to be interpreted."

For reprint and licensing requests for this article, click here.