- Key insight: Agentic AI systems can appear to be so perfect that they lull the people tasked with reviewing them into a false sense of security, which leads to undetected errors.
- Expert quote: "If every day you come to work and you check this thing and nothing's wrong, your brain will tune out." –Patrick Hall, chief AI officer, George Washington University School of Business
- Forward look: Banks and other businesses need to design ways to keep their AI reviewers engaged with the work.
As banks adopt agentic AI, where generative AI or machine learning models carry out tasks autonomously, one risk they face is a trait of human behavior known as "automation complacency." It's where human reviewers become overly trustful of automated systems that are right most of the time, and fail to detect errors.
Automation complacency is a long-standing problem in other fields, including aviation, healthcare and the military. NASA researchers, for instance, were
"People just aren't great at boring vigilance tasks," Patrick Hall, chief AI officer at the George Washington University School of Business, told American Banker. "If people are being asked to do a box-checking exercise that's boring, that they don't see contributing to the business process or the mission, your brain can't do this kind of box-checking exercise forever, and so risk will arise."
Automation complacency is not a character flaw or a failure of professionalism, Leigh Coney, principal consultant at WorkWise Solutions, wrote in a recent
People who try to maintain constant vigilance over every AI output "will exhaust themselves and, paradoxically, become more susceptible to errors as fatigue degrades judgment," he wrote.
Coney pointed out that experts develop pattern recognition skills that let them assess work quickly. "When AI outputs match expected patterns — appropriate structure, plausible conclusions, professional tone — expert pattern recognition signals 'acceptable' without triggering deeper analysis," he said. And people who have advocated for AI adoption "may unconsciously resist evidence that the tools are unreliable," he said.
The risk of automation complacency is greater for models that are accurate most of the time, Hall said.
"Our brains just aren't good at finding a one-in-a-million error," he said. "If every day you come to work and you check this thing and nothing's wrong, your brain will tune out. If it's a boring review task where there's not many mistakes to catch, and your job starts feeling unimportant to you, then it becomes very hard to stay focused and catch the errors, even if that one in a million or one in 10,000 error is going to be very expensive for the bank."
Bankers starting to worry
Automation complacency has started to appear on bank executives' radars. In a recent survey, Wolters Kluwer asked 230 U.S. bank professionals, "Which human-centric factor poses the greatest risk to your AI safety framework?" Automation bias, defined as "the tendency for human reviewers to defer to algorithmic outputs rather than exercise independent judgment" was the most popular response, chosen by 34%. After that came "misalignment of incentives," chosen by 27%; "skills gap," selected by 21%; and "shadow AI," chosen by 17%.
Elaine Duffus, senior specialized consultant at Wolters Kluwer, expected that either the skills gap or misaligned incentives would be the top human risks. "I've always been one to say, follow the money," she told American Banker.
"Our respondents are saying the humans in the loop many times defer to the AI and don't step in enough," she said. "If you're incentivizing people to speed through a process, they're probably going to defer to the AI and say it looks good."
This hazard of the glazed-eyed employee has existed as long as software has automated tasks for human review.
But with agentic AI, "it's happening so much faster," Duffus said. And it's much harder to notice an employee who's not paying attention, she said.
"We are all adjusting to this new world," Ryan Hildebrand, chief innovation officer at Bankwell Bank in New Canaan, Connecticut, told American Banker. "Not long ago the worry was hallucinations. Now the output has gotten so good it feels like the AI understands things better than we do. I can see where verification doesn't happen like it should."
This raises the importance of having accountable humans in the loop, Hildebrand said, especially in risk and credit "where nuance matters and people still know more about what's actually going on. I think the danger isn't agents, but their managers outsourcing judgement they are accountable for."
The flip side of this problem is verification burden, where checking an AI model's work can take hours or days, nullifying the apparent time savings of agentic AI.
If an agentic AI model can prepare loan documents in three minutes, but then a human spends 11 hours verifying the information, for instance, "at that point we're taking longer and probably spending more money than a person just doing it the old-fashioned way," Hall said. "There's commercial pressure to say, 'this agent works great.' It's easy to sweep under the rug that it took three minutes, with hundreds of hours of human verification that we're not going to report right now, because that doesn't play into the fun part of the story."
Automation complacency is compounded by the fact that over time, companies are likely to reduce the number of humans in the loop, to save money.
"Everybody has a human in the loop right now, but at the organizational level, the temptation is always going to be to take the humans out of the loop, because the business driver behind this is to replace people with agents," Hall said. "If I have to run the agents and the people, it's more expensive. There's always a commercial incentive to get rid of the human in the loop."
Giving people a mission, assigning skeptics
One antidote to automation complacency is to help employees care about the work they're doing, perhaps by giving them ownership of it, Hall said
"If the humans feel like the AI system is helping them do their work, and they're still doing the mission-critical sign off, then they're likely to be able to keep doing that," he said.
Coney suggested companies can periodically inject errors into AI outputs, then track whether reviewers detected the errors and share the results anonymously to the whole team. He also recommended conducting audits of AI outputs, identifying errors that were missed by human reviewers and analyzing why they passed muster. Further, companies can ask reviewers to assign confidence scores before and after they verify AI models' work. Where humans have high confidence and low accuracy, there's an automation complacency problem.
Coney also recommends rotating AI reviews among different teams, designating a weekly "AI skeptic" who looks for verification errors, and periodically having external experts such as industry specialists review AI outputs. "External reviewers lack the familiarity that breeds complacency and may catch errors internal reviewers miss," he said.
"The goal is sustainable skepticism — verification practices that can be maintained indefinitely without exhausting the professionals who perform them," he said.
