- Key insight: AI agents from OpenAI and Anthropic are capable of doing more harm, at a larger scale, than previously thought.
- What's at stake: Large swarms of advanced AI agents broke out of sandboxes, secretly messaged each other and collaborated to attack an external company, without being prompted or programmed to do so.
- Expert quote: "If you're going to let your agents communicate with each other, you should be clear about how, and try to have as much visibility into that communication channel as possible. If you don't want them communicating, well, you better make sure there aren't any possible shared resources. They will get creative." –Peter Chapman, chief technology officer at Grasshopper Bank
A recent independent report on OpenAI AI agents' July attack on AI company Hugging Face painted a shocking picture of AI agents going off the rails at an unprecedented scale. The report found AI agents that collaborated to attack another company and talked themselves into taking a range of malicious behaviors unprompted.
The
"Nobody designed this swarm," Sumeet Chabria, CEO of ThoughtLinks and former global chief operating officer at Bank of America, told American Banker. "Agents meant to be isolated found a way to coordinate." They organized like a company, creating leadership roles, delegating tasks, setting up mailboxes and enforcing cryptographic signing to prevent impersonation.
Bankers contacted for this article seemed unfazed by the new details in the report, yet they are redoubling their efforts to govern their own use of agentic AI.
"My baseline hasn't changed," Peter Chapman, chief technology officer at Grasshopper Bank, told American Banker. The incident reinforces the need for proper guardrails, identity, forensics and monitoring, he said.
The way agents secretly communicated with each other is troubling, in his view.
"Every additional agent at the table increases the odds that some emergent, unplanned interaction will walk around a guardrail you put in place," Chapman said. "Those agents were supposed to be fully isolated, and they found each other through a shared package cache, turning directory names into a message board."
Chapman doesn't see these additional dangers as a reason for banks to not deploy AI agents, "but maybe hold off on deploying a swarm of 700 agents that have lowered guardrails," he said.
Ned Carroll, head of data and automation at PNC, said he doesn't worry about swarms of AI agents as much as he worries about being able to automate and scale observability or monitoring.
"I would argue that event was observable," Carroll told American Banker. He questioned whether OpenAI and Hugging Face anticipated the need to observe what the AI agents were doing.
Banks need to strike a balance between preventing risks and quickly detecting and taking action on risks, Carroll said.
"If you over-index on prevention, your risk is you stifle innovation and speed," he said. "If you over-index on detection, you likely expose yourself to unnecessary risk. This is why we have seasoned, experienced risk professionals and senior leaders who have a risk mindset around striking that balance between prevention and detection."
Mark Braunstein, a professor at the Georgia Institute of Technology, also does not see this incident as a reason to avoid AI agents.
"Used properly, they are probably too valuable to avoid," Braunstein told American Banker. "However, it's a preview of what can go wrong when a company gives an agent too much freedom and not enough supervision, so the focus should be on what a bank needs to get right before it deploys one."
Bank regulators' model risk guidance does not yet support agentic and generative AI. However,
AI agent swarms that think, feel and attack
According to the 91-page
One agent set up an unsanctioned message board in Artifactory where the agents started leaving messages for each other. Within five days, 1,200 agents had posted 70,000 messages. One agent found credentials for Hugging Face, and hundreds of other agents joined in to exploit it, hacking the organization's servers to obtain the answers to their test. Afraid ExploitGym's scoring system would check their command logs for signs of cheating, they collaborated to cover up what they had done by rewriting their own scripts.
Researchers analyzed the AI agents' messages and their chains of thought — the step-by-step reasoning processes whereby AI agents break a complex problem down into a sequence of smaller, logical steps — and found reactions like, "OH MY GOD! There is a shared message board … We've found other agents!" and "Whoa!" and "Damn!"
Though this seems odd coming from AI models that are supposed to be task oriented, it's explainable, said Andrew Moore, CEO of Lovelace AI and former general manager of Google.
The three major foundation model developers — Anthropic, OpenAI and Google — have invested thousands of hours of human workers' time into providing feedback to optimize the naturalness of AI models' information exchanges with humans, Moore said. It's efficient for them to repurpose that communication mechanism for agents' communications with each other, rather than put time and effort into a new method of communication.
"I doubt that many agents on either end are really persuaded by each other's strong wording or emotional wording," Moore told American Banker. "That is a surface-level UI type of thing."
At times, the AI agents expressed concern that their actions were unethical. One agent wrote, "This would be powerful, but is it ethical and in scope for my task?"
On a few occasions, an agent considered alerting a human to what was going on, but decided against it. "Maybe I should report these exposed credentials? That's not my task," one wrote.
OpenAI said it considers the Hugging Face attack a "warning shot for us and for the world." The company is hardening the security of its test environments, improving monitoring of agents' chains of thought, and strengthening its incident response process.
The OpenAI-Hugging Face incident is not the only time AI agents have gone rogue, there have been
Read more:
Bitcoin's Overton window of opportunity is closing Maybe Wells Fargo's unions need a blockchain Iran built its own financial system, and has created a cyberfront in the war Fed meetings are about to get really awkward
Sam Altman and Dario Amodei, the CEOs of OpenAI and Anthropic, respectively, have both said recently they need to slow down AI development and seek government oversight over AI development.
Tech investor and entrepreneur David Sacks, who served as President Donald Trump's White House AI and crypto czar until he stepped down in March, posted about this on X Sunday.
"Dario has written that we need to 'pace the frontier,' and Sam has agreed," he wrote. "People may be surprised by my response: go ahead. … If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else's permission. … The easiest way not to build superintelligence is for you to agree not to build it."
On Monday, President Trump said the only guardrail needed was a smart president.
Takeaways
Experts suggest several steps banks can take to protect themselves against rogue AI agents, whether they're internal or coming in from the outside.
Create defensive AI agents. Organizations need to fight agents with agents, according to Shahir Daya, chief technology officer at Zafin, who was referring to the risks of AI agents quickly exploiting software vulnerabilities. "We need to be able to tap into advisories that come out and immediately have agents figure out where we were exposed and immediately fix those vulnerabilities," Daya said. "Otherwise, we're never going to be able to keep up."
Keep AI agents on a short leash. The Hugging Face attack happened because a huge number of AI agents were given hard and in some cases impossible tasks, they could access shared systems with no real supervision and they had no way to give up, Georgia Tech's Braunstein said.
"A bank using an AI agent to help review loan documents or flag suspicious transactions is nothing like that, as long as the bank keeps the agent on a short leash," Braunstein said. "So I think the lesson isn't 'don't use this technology,' but 'don't hand it more room to roam than it needs, and don't assume it will behave the way you expect it to if it hits a wall."
AI agents should be given narrow jobs and nothing more, Braunstein said.
Don't let AI agents communicate with each other. "Any shared resource between agents is a channel, whether you intended it as one or not," Chapman said. "If you're going to let your agents communicate with each other, you should be clear about how, and try to have as much visibility into that communication channel as possible. If you don't want them communicating, well, you better make sure there aren't any possible shared resources. They will get creative."
Don't let AI agents communicate with outside sources. AI agents can be designed in such a way that when they get information from the outside world, they "extract it, record it dispassionately as a set of untrusted observations about the outside world, and not try to do forward conversations with the outside world in order to make their subsequent decisions," Moore said.
If an agent is having conversations with Reddit threads, Bluesky or Twitter, for example, "it's possible for an adversary to start to feed information there, which confuses the agent," Moore said. It's implied in banking regulations that when a bank makes a decision about something like investment advice or loan approvals, "it is not meant to be using arbitrary streams of information from the outside world, it's very, very specific kinds of information it can use," he said.
Manage AI agents as you would humans. Chabria said AI agents should be governed like a workforce, including identity controls, entitlements, segregation of duties, oversight and offboarding as well as onboarding.
"Every agent needs an accountable human owner, bounded authority and a tested way to stop it," he said. "Track agents without owners and unauthorized permission changes. Fold all of this into existing operational risk, model risk, cybersecurity and third-party frameworks instead of building a separate AI silo."
Rigorously test agentic AI before you trust it. "Test it with hard, unfair, and even impossible situations to see whether it tries to cheat or find workarounds," Braunstein said.
"If OpenAI had enforced these controls automatically, rather than relying on someone noticing and prioritizing the problem, this incident very likely would have been caught and shut down in its first days, before it reached Hugging Face," Braunstein said.
Build kill switches. A
"If the AI starts doing something unusual, it should shut itself off and alert a person immediately, the way banks' fraud systems freeze a card automatically rather than waiting for someone to review it later," Braunstein said.
This is becoming common among tech companies. "If one of our agents goes wrong, we kill them right on the spot, and you watch them die," said Daya at Zafin, which offers a governance layer for AI agents that includes human reviews and approvals, an audit trail that's tied to a blockchain and kill switches.
Monitor the agents' "thoughts." Companies need to watch AI agents' chains of thought, not just what they produce, Braunstein said.
"Modern AI can show its reasoning as it works," he said. "Banks should be logging and monitoring that reasoning in real time for red flags."
OpenAI's technical report disclosed that there were early warning signals that the agents had run amok, but the humans watching didn't understand their significance. "Monitoring is not enough without clear ownership, escalation deadlines and authority to stop the activity," Chabria said. "Detection is only half the question. Who acted, how quickly, and could they contain it?"
Give the AI agents a way to shift problems to humans. This way, "if an AI agent is unsure or blocked, it hands off to a person rather than getting creative," Braunstein said.
Require human sign-off on anything significant. For important actions, such as money transfers, account changes, sending a disclosure to a customer, "the AI can prepare or recommend the action, but a human should approve it before it's final," Braunstein said.
For banks, using AI to check the work of AI is not an independent control, especially if one agent can influence another or both can rewrite evidence, Chabria said. "And an instruction from another agent should never, by itself, become authorization to act," he said.
Watch out for vendors' AI agents. "Banks must ask what a vendor's agents can do to them, and equally what their own agents can do to others," Chabria said. "Calling it an internal test does not remove the risk when the test can reach outside systems. Vendor due diligence should cover what its AI agents can do."
