• Key insight: Research presented at a Federal Reserve Bank of New York conference shows that AI chatbots can make users more confident in wrong answers and suboptimal decisions, posing unique challenges to the bank culture and governance.
• Expert quote: "There's a whole bunch of risks by being extremely enthusiastic without a healthy check and challenge of what we're implementing." — Jose Placido, CEO, BNP Paribas USA
• Forward Look: Subject-matter experts say risks can be mitigated by inserting more frictions into large language models and educating users on how to best use AI features..

Artificial intelligence makes users more confident in their decisions, recent research shows, but not necessarily more accurate.

More troubling still, the decisions being made may be coming from the AI itself, rather than the human users, said Steven Shaw, a researcher from the Wharton School of Business.

"We coined the term 'cognitive surrender,'" Shaw said, "which is essentially outsourcing thinking itself to AI."

Shaw discussed his findings Tuesday during the Federal Reserve Bank of New York's annual culture conference. His study was one of two discussed at length during the event, which explored the potential detriments of large language models on both individual users and organizations. 

The other study, spearheaded by Alexandra Chesterfield, a behavioral economist at the London School of Economics, highlighted the dangers of so-called "sycophantic" tendencies in AI — programming that wires chatbots to prioritize favorable responses over critical ones. 

"These systems tell us what we want to hear rather than maybe what we need to hear or what we should hear," Chesterfield said during the conference. "That matters because these systems are not just giving us information, they are increasingly shaping how we think, how we decide and how we act."

The studies were not focused on bankers or the financial-services industry, but rather human behavior more broadly. Still, the results were seen as highly relevant to culture and governance in the financial sector, given the impact decisions can have on firms, customers and the broader financial system.

Jose Placido, CEO of BNP Paribas USA, who also spoke during the conference, said the findings did not deter his belief in the benefits of AI technology, but rather highlighted the importance of getting internal oversight over the technology right.

"We need to be excited about AI and know its strengths and at least be aware of [its weaknesses] — it's cognitive surrender today, it's bias tomorrow," Placido said. "There's a whole bunch of risks by being extremely enthusiastic without a healthy check and challenge of what we're implementing."

Cognitive surrender

Shaw, who worked alongside Wharton marketing professor Gideon Nave, said his study builds on the concept of "thinking fast and slow" — a long-used economic paradigm that compares intuitive judgements against deliberative reasoning — by adding an AI twist.

Read more:

• As fintechs nab bank charters, what's happening to BaaS?
• Overheated data center knocked 23 credit unions offline
• Chris Britt still says banking is broken, but Chime can fix it
• Two thefts decades apart, and a victim's quest for answers

The experiment presented participants with a set of logic-based questions. The people were divided into three groups — one that could use a functional AI chatbot, another that had access to a chatbot that intentionally gave false or misleading information and a control group with no AI access. 

The researchers found that participants defaulted to the AI response 80% of the time, regardless of whether its reasoning was sound or not. 

"If we look at the human accuracy when … participants had access to AI, it increased quite dramatically when AI was giving that good, correct information, and it decreased quite dramatically when participants were given incorrect information," Shaw said. "So, human performance tracked the accuracy of the AI model there, and that large differential that we saw is what we call cognitive surrender."

Shaw said participants with both AI systems reported feeling more sure about the accuracy of their responses than the control group.

"Just having access to AI made people more confident," Steve Shaw, assistant professor of marketing at Kings College in London, said of his study. "About 10% were more confident, despite the fact that half the time they're being given incorrect information."

Sycophantic AI

Chesterfield said the positive sentiment described by AI users is a feature of the technology, not a bug. She noted that many chatbots default to supporting the ideas fed into them by humans — a practice that makes users more inclined to continue using them.

In her study, Chesterfield, who also works as director of human-AI collaboration and the tech firm Salesforce, examined how this dynamic impacted financial decisions. 

She, too, had three groups of participants, one armed with a highly sycophantic chatbot, another given a standard chat bot and a third with a less sycophantic — more critical — chatbot. Each group was then instructed to discuss what to do with a surprise extra $10,000. 

Chesterfield said the highly sycophantic chatbot group invested, on average, $447 more than they would without consulting AI. The less sycophantic chatbot group invested $345 less than they would have otherwise. 

Chesterfield said the disparate results do not necessarily suggest sycophantic AIs steer users toward better or worse outcomes — in some cases, investing more might be the optimal choice, she said — but rather it demonstrates the lack of objectivity in the technology.

While this dynamic can be problematic at the individual and firm level, its risks are greatest when looking across industries and even across whole swaths of the economy that rely on the same base AI models. Chesterfield said the more groups rely on the same systems for analysis, the more similar they are likely to become.

"With different judgments, so without AI, mistakes can cancel each other out," she said. "But with a shared algorithm, a shared recipe for making decisions, those mistakes can become correlated."

Chesterfield said many of these issues can be avoided by opting for more challenging settings within AI systems, namely by instructing them to be more critical and to be oriented toward challenging inputs rather than supporting them. She also noted that setting up proper institutional controls can also mitigate these risks.

"This is a design choice that firms can make," she said. "Once we get the models, we can then instruct them to have particular personalities. We want the devil's advocate, we don't want the sycophant."

Placido said BNP Paribas has developed its own LLM and encouraged employees to use both the proprietary system as well as other generally available tools. He said the focus has been on ensuring that bank employees are not afraid to use the technology. 

In light of what he heard at the conference, Placido said the bank will place a greater emphasis on teaching its employees about the pitfalls of outsourcing thinking to AI.

"Now, we'll have a section on cognitive surrender to adapt how we train," he said. "All of the information you see may be too good to be true. That second sense of checking and challenging your own work is something we need to keep teaching everyone along the way."