Could an AI Start a Financial Crisis Before Humans Could Stop It?

Written by James Taylor
AI, Borrowing and Panic: How the Next Financial Crisis Could Spread

How an AI Mistake Could Become a Global Financial Emergency

The Trade Nobody Could Stop

A financial system can survive a wrong prediction; the danger grows when that prediction makes institutions act together before anyone can check it.

On 1 August 2012, Knight Capital’s automated trading system malfunctioned. In approximately 45 minutes, it accumulated billions of dollars in unwanted positions. The firm ultimately lost more than $460 million.

That was a software failure, not a modern AI deciding to attack the economy. It nevertheless established something relevant to today’s debate: permission to trade can turn a technical error into a financial emergency before people regain control.

Could artificial intelligence start a financial crisis before humans could stop it? Yes, under particular conditions. AI could initiate or accelerate disruption faster than human review, especially where institutions grant systems substantial trading authority. But a rapid market fall becomes a wider crisis only when losses spread through borrowing, cash demands, essential services or confidence.

The crucial question is what remains after somebody presses stop.

What Would Count as an AI Financial Crisis?

A falling share price is not automatically a financial crisis. Investors lose money whenever businesses disappoint. Even a dramatic market plunge can remain a temporary repricing if institutions can absorb losses and essential financial services continue.

A systemic crisis involves disruption spreading far enough to threaten the financial system’s ability to provide those services. Businesses struggle to borrow. Institutions doubt whether counterparties can pay. Assets that normally raise cash become difficult to sell without accepting punishing discounts.

For AI to cause that kind of event, it needs a route into financial decisions. A chatbot producing a mistaken market forecast has little direct power. A system authorised to place orders, alter lending limits or move collateral has a different relationship with the world.

Access matters as much as intelligence. An ordinary model connected to a large balance sheet could cause more immediate damage than a sophisticated model restricted to drafting research.

There is also a separate possibility: investors might overpay for AI companies and suffer losses when expectations collapse. That would be a crisis associated with investment in AI. It would not, by itself, demonstrate that an AI system had autonomously caused one.

Keeping those scenarios separate prevents every technology-stock downturn from becoming supposed evidence of machines escaping control.

What Automation Has Already Demonstrated

The Knight Capital incident is useful because it requires no speculation about machine consciousness. The US Securities and Exchange Commission found shortcomings in the firm’s safeguards, software deployment and response procedures. The company agreed to settle the charges without admitting or denying the findings. sec.gov

The practical lesson is about the distance between authority and supervision. Software was allowed to create exposures that the organisation’s controls failed to contain promptly. Stopping further orders did not automatically remove the positions already acquired.

Modern AI introduces different uncertainties, but it enters a world where execution can already outrun organisational understanding. A language model need not produce each individual exchange order. It could supply a judgement that another, faster system converts into many orders.

Consider the distinction between deciding to reduce exposure to an industry and carrying out the resulting trades. The first task might involve interpreting an earnings report. The second might involve established execution software. A mistaken judgement can travel through that connection without the AI itself operating at the fastest trading speeds.

That makes the interfaces important. Which conclusions become executable instructions? What checks sit between them? How much exposure can accumulate before those checks intervene?

Calling the entire arrangement “AI trading” obscures the places where control actually succeeds or fails.

When Sensible Decisions Become a Collective Problem

Imagine several investment funds holding similar assets. Each has a system instructed to limit losses and preserve cash. A sharp price fall makes every system recommend selling.

For one fund, selling early may be prudent. Across the market, simultaneous sales can push prices lower and trigger further defensive decisions. An action that protects an individual institution can make the environment worse for everyone else.

This problem predates AI. The potential change is how quickly institutions interpret information and act on it, and whether their systems behave similarly under stress.

The International Monetary Fund’s October 2024 analysis identified both sides of that possibility: AI could improve risk management and market liquidity, while also increasing market speed and volatility during stressful periods. Greater efficiency in normal conditions does not establish resilience in abnormal ones.

Nor does a common model guarantee identical trading. Different firms have different holdings, instructions, funding arrangements and constraints. One may buy an asset that another must sell.

The risk therefore depends on observable behaviour. Do systems repeatedly identify the same assets as dangerous? Do they withdraw at similar thresholds? Are the buyers able to keep buying when lenders become cautious?

A useful test would compare those responses during stress, rather than assuming that different company names represent genuinely independent decisions.

How a Sell-Off Could Become a Cash Emergency

Borrowing provides one route from falling prices to forced action.

Take a deliberately simplified fund with £100 million of investments, financed by £80 million of borrowing and £20 million of its own capital. If its assets fall by 10%, their value becomes £90 million. Assuming its debt remains unchanged, its own capital falls to £10 million.

The investments have lost a tenth of their value; the fund’s capital has halved. This illustration leaves out hedges, fees and contractual details, but shows why leverage makes relatively modest price movements consequential.

Now suppose the borrowing is secured against those investments. The lender may demand extra collateral, depending on the agreement. The fund must find acceptable assets or cash. If it has insufficient reserves, it may sell investments into the falling market.

Britain’s 2022 liability-driven investment episode provides a documented comparison. Following a sharp repricing of government bonds, some leveraged funds faced collateral demands and could not obtain additional capital quickly enough. Their bond sales intensified market pressure, prompting Bank of England intervention. This was not an AI crisis.

For an AI scenario, the important inference is that an initial machine-driven sale could activate ordinary financial obligations. Once cash is contractually due, switching off the model that initiated the sale does not settle the bill.

A Plausible Chain of Events

The following scenario is illustrative, not a prediction or a description of an observed AI crisis. It assumes substantial automated authority, overlapping exposures and insufficient cash buffers. Remove those assumptions and the outcome could be much less serious.

An investment firm’s AI system misreads an ambiguous announcement as evidence of acute credit trouble. Its permissions allow it to reduce exposure within a pre-approved limit. It sends instructions to an execution system.

Other firms see the resulting price movement. Some react to prices without reading the announcement. Others use similar information sources and reach comparable conclusions. Sellers become more urgent while prospective buyers demand lower prices.

At this point, independent verification could break the chain. A trusted correction might arrive. Buyers with available capital might step in. Hard position limits might keep the original error small.

Suppose those defences prove insufficient. Falling prices generate losses for leveraged holders. Lenders ask for collateral. Funds sell other assets to obtain cash, spreading pressure beyond the original market.

Human managers then stop the faulty AI. That prevents new instructions from that system, but other firms still face losses and outstanding obligations. Some may reduce lending because they no longer trust counterparties’ financial positions.

The final stage would unfold through institutional decisions over hours or days, potentially longer. The initiating error could be fast; the wider crisis would involve a continuing struggle to allocate losses, obtain funding and establish which organisations remained sound.

“Before humans could stop it” can therefore mean before humans prevent the first irreversible commitments. It need not mean an entire global recession materialises in seconds.

Why Correcting False Information May Not Be Enough

A false claim does not need to remain believable forever to cause damage. It needs to affect decisions before verification catches up.

In the illustrative scenario, the original announcement might eventually be clarified. An investor who bought at a depressed price might benefit from the correction. Someone forced to sell because a collateral deadline arrived first might have already crystallised a loss.

This creates two clocks. One measures how quickly the truth becomes established. The other measures how quickly institutions must meet their obligations. If the second expires first, a correct explanation arrives too late for some participants.

Generative AI could contribute to such a problem through fabricated material or unreliable summaries. The Financial Stability Board has identified AI-enabled disinformation and fraud among potential financial vulnerabilities. That warning establishes a risk category, not proof that any particular fabricated message could topple a bank.

Credibility, distribution and the target’s financial condition would matter. A doubtful rumour about a well-funded institution may fail. The same rumour about an institution already struggling to raise cash could find a more receptive audience.

The familiar question of whether a computer’s representation matches reality also runs through WarGames. In finance, however, participants can create real obligations while acting on a false representation. Establishing the truth does not necessarily reverse what they have done.

The Weak Point Could Be a Shared Supplier

A crisis need not begin with a sell instruction. It could begin when several institutions lose access to a service needed to operate safely.

The Financial Stability Board’s 2024 assessment highlighted concentration among technology providers and dependencies on external services. If many firms rely on one critical provider, an outage or compromise can affect them together.

The consequences would depend on the service. Losing a writing assistant is different from losing a tool embedded in payment screening or risk controls. AI use alone does not make a supplier systemically important.

Consider a hypothetical bank whose payment checks become unreliable. Continuing to process transactions could expose it to fraud; pausing them could delay legitimate payments. If customers cannot tell whether the problem is technical or financial, uncertainty may spread beyond the original malfunction.

A backup helps only if it can perform the required work. Another interface connected to the same underlying service is not an independent recovery route. Nor is a manual procedure credible merely because somebody documented it: staff need the capacity, information and authority to use it.

This is the practical version of the dependence explored in The Machine Stops. The relevant question is how much useful activity survives when a central service becomes unavailable.

Would a Kill Switch Solve the Problem?

A kill switch can be valuable. It can stop new orders from a particular system or revoke its authority. Its effectiveness depends on its coverage, reliability and speed.

However, stopping a program does not ordinarily cancel completed trades. It does not restore the value of assets sold at a loss, repay borrowing or automatically release collateral. Those consequences sit outside the program.

Market-wide circuit breakers address a different layer. The New York Stock Exchange’s February 2026 guidance describes US equity-market thresholds linked to S&P 500 declines of 7%, 13% and 20% from the previous close. Subject to timing rules, the first two trigger pauses of at least 15 minutes; the third ends trading for the day.

Such mechanisms create time for assessment. They do not guarantee willing buyers at the previous price or solve every problem in lending, payments and privately negotiated contracts.

A pause also needs a reopening plan. If institutions still need to sell and buyers remain uncertain, the underlying imbalance survives the interruption.

The useful design question is therefore specific: what does this switch stop, what continues running, and who manages the obligations left behind? A button labelled “emergency stop” is meaningful only when those answers are known and tested.

The Limits of a Human in the Loop

Human approval can prevent errors when the reviewer has time, relevant information and the power to refuse. It becomes less convincing when approval is reduced to clicking through recommendations that arrive faster than anyone can assess them.

In a June 2026 speech, Bank of England Deputy Governor Sarah Breeden said the evidence then suggested trading firms mostly used autonomous AI for lower-risk operational work, including research. She also warned that relying on human review for every action would become unrealistic as autonomy increased.

That combination matters. It argues against claiming that fully autonomous AI already controls the financial system, while identifying a plausible weakness in future oversight.

For a hypothetical trading operation, meaningful human control could involve approving an exposure ceiling before trading starts and requiring fresh approval before it increases. The reviewer controls the scope of action rather than attempting to inspect each transaction.

Responsibility would also need to survive disagreement. An operator should be able to stop a system because its behaviour is unexplained, without first proving that its recommendation is wrong. Otherwise, the organisation asks the human to defeat the model’s analysis before exercising authority over it.

An alert creates an opportunity to intervene. A workable procedure determines whether anyone can use it.

Could Better AI Prevent the Crisis Instead?

Yes. A capable system could identify an unreliable input, detect inconsistent positions or recognise that apparently separate exposures depend on the same underlying borrower. It could help supervisors see relationships that are difficult to track manually.

The IMF’s assessment explicitly recognises potential improvements in risk management and monitoring. The case against an inevitable AI crisis is therefore substantial: adoption could remove some existing human weaknesses rather than simply add new dangers.

There is also no reason every institution must respond identically. Different investment horizons, available cash and mandates can produce opposing actions. A long-term investor with uncommitted funds may treat a forced sale as an opportunity.

The harder question is whether that stabilising capacity remains available during stress. A bargain is of little use to an investor whose borrowing has just been withdrawn. A model can recognise underpricing without possessing permission or money to trade against it.

This suggests a test for optimistic claims. Demonstrate that the system improves decisions when data is incomplete, funding is constrained and other participants react unexpectedly. Strong performance on ordinary days cannot answer all three questions.

AI can help a financial institution understand its choices. Whether it can take the stabilising choice also depends on the balance sheet and rules surrounding it.

What Effective Protection Would Require

The most credible protections would limit the size of a mistake, slow its transmission and preserve ways to recover. These are different tasks, so no single model benchmark can establish that all have been achieved.

First, consequential authority should have explicit boundaries. A research tool should not acquire trading permissions simply because connecting the two systems is technically convenient. Position, borrowing and payment limits should remain enforceable when the AI behaves unexpectedly.

Second, organisations should test interactions. A fund might pass an isolated test by selling assets quickly. If many funds depend on selling the same assets, their individual recovery plans could conflict. Testing should ask who buys, with what financing, and under which constraints.

Third, recovery arrangements should distinguish stopping activity from restoring service. A firm may be able to disable an unreliable payment model within seconds while needing far longer to process the resulting backlog safely. Those are separate capabilities with separate consequences for customers.

Fourth, investigations need dependable records. Operators should be able to establish which information entered the system, what permissions applied and which instructions were executed. A fluent explanation generated afterwards is not a substitute for an audit trail.

Finally, someone must have authority to act across organisational boundaries. A problem affecting several institutions cannot always be resolved by each firm optimising its own position. Cooperation needs procedures agreed before emergency decisions become necessary.

How It Could Reach People Outside the Markets

For households, the first visible consequence might be a falling investment balance. The more damaging consequences would depend on whether market disruption affected lending, payments or employment.

A business with sound long-term prospects can still struggle if finance becomes unavailable when wages or suppliers need paying. A household can experience disruption through a delayed payment even when its bank ultimately remains solvent. Different failures require different responses.

Mortgage costs would not move in one guaranteed direction. Expectations of a weaker economy might pull some interest rates down, while lenders’ funding costs or caution push borrowing terms the other way. Claiming that an AI crisis would automatically make mortgages cheaper or dearer would hide those competing forces.

Pension effects would also vary by scheme, assets, funding and time horizon. A market price decline, a fund’s cash shortage and a permanent loss of retirement benefits are not interchangeable events.

The same distinction between wealth and money available now helps explain what happens when a country runs out of money. Financial trouble often becomes urgent because an obligation falls due before resources can be mobilised.

An AI-triggered shock would reach ordinary life through those familiar channels, even if its origin lay in unfamiliar software.

The Decision That Matters Comes Before the Emergency

There is no defensible countdown to an AI financial crisis. Public warnings establish plausible mechanisms and reasons to prepare; they do not supply a reliable date or probability for a global breakdown.

The strongest conclusion is conditional. AI becomes more dangerous when its authority grows faster than the ability to contain its actions, when institutions make overlapping decisions, and when losses activate obligations that cannot wait for an investigation.

Evidence of strict permissions, independent recovery systems and successful tests of interacting institutions would weaken the darker scenario. Evidence that firms cannot explain their exposures or operate without a shared service would strengthen it.

The decisive question for a board approving an autonomous financial system is concrete: if this makes the wrong decision at the worst available moment, how much can it commit before control returns?

That amount is the exposure humans choose in advance. By the time they are deciding whether to press stop, some of it may already belong on a balance sheet.

Sources

Next Reads

James Taylor
Next

What Actually Happens When a Country Runs Out of Money?