Citi trader got 711 warning messages before sparking flash crash

Victor J. Blue/Bloomberg

(Bloomberg) --For one Citigroup trader in London, the morning of May 2, 2022 went from bad to worse.

It was a public holiday in the U.K., so it should have been a quiet day in the markets. A little before 9 a.m., the staffer on Citigroup's Delta One trading desk — who was working from home — began putting together a trade that would hedge the bank's exposure to the MSCI World Index.

A tool that employees usually used for such a transaction wasn't available that morning, so the trader had to manually build the basket of stocks. That's where things started to go wrong.

In Citigroup's systems, traders have the option of either entering the notional value of a trade they want to make or entering the quantity of index units they want to trade. On that day in May, the trader meant to create a basket of equities valued at $58 million, but accidentally entered that 58 million into the quantity field instead, creating a mammoth $444 billion basket containing 349 stocks from 13 different countries.

The Wall Street giant's systems immediately fired off hundreds of warnings, ultimately blocking some — but not all — of the trade from going through. Still, about $1.4 billion worth of equities began to be sold across European exchanges.

Markets immediately started going haywire. Within minutes, the trader realized the mistake and canceled the order. But the damage was done: The blunder had sparked a five-minute selloff in European stocks, wreaking havoc in bourses stretching from France to Norway.

Two years on, U.K. regulators on Wednesday revealed the result of their long-running probe into Citigroup's actions that day as they saddled the bank with £61.6 million ($78 million) in penalties for the mistake. Their findings offer the first window into how a fat-finger glitch — along with a bevy of risk management lapses — turned into a flash crash that at one point wiped €300 billion ($325 billion) off European stocks.

It's the latest blow for Chief Executive Officer Jane Fraser, who has spent years trying to shore up the lending giant's underlying risk management systems.

"These failings led to over a billion pounds of erroneous orders being executed and risked creating a disorderly market," Steve Smart, joint executive director of enforcement and market oversight at the Financial Conduct Authority, said in a statement. "We expect firms to look at their own controls and ensure that they are appropriate given the speed and complexity of financial markets."

The trader has since left Citigroup, according to people familiar with the matter. A spokesperson for the bank declined to comment.

Fifteen Minutes

Just before the trader began putting together the errant transaction, a separate team at Citigroup was figuring out how to handle its responsibilities for the day.

The bank's algorithmic service desk, which normally oversees the real-time monitoring of internal executions, had decided to transfer those responsibilities to the electronic execution desk because it had staff out on scheduled leave that day.

When the trader first input the erroneous trade, he was met with a wall of 711 warning messages. He quickly overrode the ones he could and the order was placed at 8:56 a.m.

Trades began being executed across exchanges in Austria, Belgium, Denmark, Finland, France, Germany, Italy, the Netherlands, Norway, Portugal, Spain, Sweden and Switzerland, sparking a sudden drop in European indexes. Inside Citigroup, executives were puzzled by the dip and consulted newswires to try to figure out what was behind it.

By 9:10 a.m., the trader had canceled the order, saddling Citigroup with a $48 million loss.

The electronic execution desk had been receiving hundreds of alerts about the erroneous trade but failed to escalate any of them. A separate team known as the e-trading risk and controls team also escalated the incident to the electronic execution desk but not until 9:31 a.m.

"The immediate cause of the trading error was a manual input error by the trader," the Bank of England's Prudential Regulatory Authority said Wednesday in its findings. "The error was then not identified by either of the firm's risk functions dedicated to real-time monitoring of the firm's trades, but by the trader some 15 minutes after the trade was entered into the firm's systems."

Hard, Soft Blocks

Citigroup's systems had two lines of defense against these kinds of errant trades: Soft blocks and hard blocks.

The bank sets a series of thresholds for each type of block. If a trade triggers one of these, a pop-up appears. Soft blocks can be overridden, but not hard blocks.

Citigroup had increased some of these thresholds to account for the periods of heightened volatility during the pandemic. But two years on, it hadn't reset those limits.

Still, it was these hard blocks that stopped some of the trade from going through. But the U.K. regulators on Wednesday noted that in the U.S., Citigroup had rules in place since 2013 that would have stopped all of the trade from going through.

"We are pleased to resolve this matter from more than two years ago, which arose from an individual error that was identified and corrected within minutes," Citigroup said in a statement. "We immediately took steps to strengthen our systems and controls, and remain committed to ensuring full regulatory compliance."

The penalties come as a blow to Citigroup's equities trading unit, led by Fater Belbachir. The division has spent years trying to climb the ranks of stock trading, but it remains far behind rivals like Goldman Sachs Group or JPMorgan Chase & Co.

When regulators were formulating how big of a penalty to assess on Citigroup for the failures, they said that they considered that the trading desks within the bank's Delta One division that used the order management system at the heart of the blunder had generated roughly $612 million in the nine years leading up to the errant trade, or an average of about $68 million a year.

That would mean that between the fines and the trading losses from the day, the erroneous trade cost those desks nearly two years of revenue.