Skip to content
Tudor Hilcu THG Conseil

Knight Capital: one missed server, over $460 million lost

On 1 August 2012, in 45 minutes, Knight Capital, one of the major US equity brokers, built up billions of dollars of unwanted positions that would cost it more than $460 million. The cause, as established by the SEC: new code installed on seven servers out of eight, a deployment nobody checked, and error messages nobody acted on. The same mechanism can turn up in an SME’s smallest update.

Author
Tudor HilcuFounder & CEO, THG Conseil
Published
Reading time
10 min
Topics
Projects, Go-live, Crisis
Contents10

On 1 August 2012, as the market opened, the system that sends Knight Capital’s orders to the trading venues ran out of control. To handle 212 small customer orders, it sent millions of orders into the market. In about 45 minutes, it obtained more than 4 million executions in 154 stocks, for more than 397 million shares. By the time the flow stopped, Knight had unintentionally built up a net long position of around $3.5 billion in 80 stocks and a net short position of around $3.15 billion in 74 others. The final loss came to more than $460 million.

Knight was no small player: in 2011 and 2012, its trading generally accounted for around 10% of all trading in listed US equities. On 16 October 2013, the SEC, the US securities regulator, published an order describing the sequence of events in detail. Knight consented to it without admitting or denying the findings.

Some companies wear themselves down slowly. Carillion, a British construction and services giant, paid out more in dividends than its operations generated in cash for five and a half years, before collapsing in January 2018. At Knight, a single morning was enough to do serious damage to its capital. There is nothing specific to finance about the mechanism: it can turn up in an update to a website, a till system or a price list.

Eight servers, one missed

To take part in a new New York Stock Exchange programme for retail orders, due to start on 1 August 2012, Knight changed its order router, known as SMARS. From 27 July, the new code was deployed in stages, onto a few servers each day. One technician failed to copy it to one of the eight servers. Nobody reviewed the deployment, and no written procedure required anyone to. Nobody noticed that the eighth server still ran the old code.

That old code had a history. It belonged to a function called Power Peg, which Knight had stopped using in 2003 but had left on the servers, where it could still be called. In 2005, a change had moved the counter that stopped orders being sent once the customer’s order was filled, and Power Peg had not been retested. On top of that, the new code reused a flag that had once triggered Power Peg.

On 1 August, the seven updated servers handled orders correctly. On the eighth, the flag woke Power Peg up, and it fired off orders in rapid succession without regard to what had already been executed.

The warning signs were there before the open

From about 8:01 a.m. New York time, an internal Knight system sent automated emails that referred to SMARS and reported an error: “Power Peg disabled”. It sent 97 of them to a group of staff before the 9:30 a.m. open. These messages had not been designed as alerts, and staff generally did not read them. Yet they were caused, in real time, by the failed deployment. According to the SEC, they gave Knight a chance to find and fix the problem before the market opened.

Another warning dated back ten months. In October 2011, Knight had run a disaster recovery test over a weekend. When trading began on the Monday morning, one of its desks kept generating quotes from the test data: a loss of nearly $7.5 million. Knight fixed that specific problem, without asking whether its controls would stop erroneous orders coming from another system. The SEC notes that Knight reacted to incidents narrowly.

45 minutes to work out how to stop

At the open, senior staff saw positions building up in an internal account. But the risk monitoring tool relied entirely on people: it blocked no orders, sent no automated alerts and did not even display the limits. No control compared the orders leaving the router with those coming in, and there was no procedure for halting it if it behaved abnormally.

Nor did Knight have any procedure to guide its staff through a serious incident. The technology team hunted for the cause during live trading, while the system stayed connected to the market and kept sending orders. In one of its attempts to fix the problem, the team uninstalled the new code from the seven servers where it had been deployed correctly. That made matters worse: on those servers too, the next incoming orders triggered the Power Peg code, just as on the eighth. What was missing, according to the SEC: clear guidance telling technology staff when to disconnect a malfunctioning system from the market.

Afterwards: $400 million in five days, then the sanction

On 2 August, Knight announced that it had traded out of all its erroneous positions, with a realised pre-tax loss of around $440 million. It acknowledged that its capital base had been severely hit. On 6 August, it announced $400 million of equity financing from a group of Wall Street firms, to restore its capital and liquidity. On 1 July 2013, Knight Capital Group and GETCO combined to form KCG Holdings.

On 16 October 2013, the SEC found that Knight had broken the rule requiring brokers to control the risks of their market access. It was the SEC’s first enforcement action under that rule, adopted in 2010. Knight was censured and agreed to pay a $12 million civil penalty. It also undertook to have an independent consultant review, among other things, how it deployed software and how it responded to incidents. In the SEC’s view, a written procedure requiring a simple double-check of the deployment could have shown that a server had been missed, and averted the events of 1 August.

The same question, at SME scale

An SME does not route millions of orders onto a stock exchange. But it does install updates: the online shop, the till software in its stores, a new price list in the ERP, the rules in an invoicing tool. Whenever the same change has to reach several places, one of them can be missed: one till in eight, one shop in eight, one workstation in eight.

The Knight case also shows that one slip is not enough to cause a disaster. On top of the slip, it took an abandoned function left in place, error messages nobody was in charge of reading, and no rule for pulling the plug. An SEC official, Daniel M. Hawke, drew from it a question to ask about every component of every system: if it malfunctions, what happens, and what safety nets limit the damage?

Five rules for every go-live, however small

  1. Write down where the update has to land. The full list of places: every server, every till, every shop, every workstation. After the installation, tick each line with the version you actually see on screen, not the one you planned.
  2. Have a second person check it, list in hand. Not the person who installed it. They check what was installed, and where.
  3. Give every error message a recipient. Sort the automated messages your tools send: the ones that call for action go to a named person, who knows what to do with them. A message everybody receives is a message nobody reads.
  4. Decide in advance when to pull the plug, and who may do it. A simple criterion (if this sign appears, we stop), a name, a deputy, a number where they can be reached. Stop first, look for the cause afterwards. Prepare the rollback in advance too: Knight’s, decided during live trading with no procedure, made the damage worse.
  5. Remove what is no longer used. An old function, a discount rule, a user account, an automation that is switched off but still there: what lies dormant can wake up. When you change something an old function depends on, retest that function, or remove it.

Finally, after every incident, even a minor one, ask a broader question than the fix: would our controls have stopped the same error if it had come from somewhere else?

Fictitious example: the go-live sheet of an eight-shop chain changing its till prices
  • What. The new autumn price list, version 3.
  • Where. The eight main tills, one line per shop, ticked with the version shown on screen.
  • Who installs, who checks. The IT manager installs on Friday evening. The purchasing manager checks on Saturday before 8:30 a.m., list in hand, with a test receipt from each shop.
  • Who gets the errors. Error messages from the tills go to the on-call manager’s phone, and to nobody else.
  • When we pull the plug. One wrong price on one test receipt, in a single shop, is enough: the old price list goes back everywhere. The on-call manager decides alone, without waiting for the managing director.
  • Rollback. The old price list is saved, and putting it back was tested on Thursday on one till.

Six lines, one page, no tool to buy. The sheet forces you to answer, before the day, the questions Knight only discovered during live trading.

What I learnt

On my assignments for large groups, I run go-lives of critical projects. They are prepared with a daily steering committee, the critical path, mitigation and rollback plans, sign-off of critical tests, a checklist, crisis communications ready to go and a support team for the days that follow. On an ERP and WMS programme, I also ran remediation plans after major incidents.

For me, a critical go-live is decided before the day: criteria written in advance, a rollback plan ready, a daily committee in place.

The limits of this case

The facts about Knight come from the SEC’s order and press release of 16 October 2013, and from two Knight press releases filed with the SEC in August 2012. Knight consented to the order without admitting or denying its findings. The loss varies by source: around $440 million before tax in the figure Knight announced on 2 August 2012, and more than $460 million in the SEC’s order, which is the figure I use. The order names no Knight employee: I stick to the procedures it describes. In an SME, an error rarely costs that much in so little time, but the mechanism is the same. What happened to KCG Holdings afterwards has not been checked for this article.

Where to start this week

  • Note the next update planned in your company, however small: a website, a till, a price list, a piece of software.
  • On one page, answer five questions: where it has to land, who installs it, who checks it, who gets the errors, who can stop it and on what criterion.
  • Open the inbox where your tools’ automated messages arrive. Count the ones left unread, and name a recipient for each type of message.

Twenty minutes, and your next update has a safety net.

An update is only finished once someone else has checked that it has landed everywhere.

The Revlon and SAP case shows another go-live going wrong, this time over several months. What Knight had to settle over the next five days is in the first 48 hours of a crisis, on one page. To prepare yours, the free go-live checklist covers what must be ready, the committee’s decision and the rollback plan. The Crisis on one page module has you write down, on a calm day, who decides and who to call on the day everything stops. Both are free to use on the Resources page. If a go-live is coming up, Project leadership explains how I work; if an incident is under way, so does Coming through a crisis. The first conversation is free, by phone and in confidence, within 48 hours: call me or write to me, on +33 6 74 86 88 70 or at contact@thgconseil.fr. In a crisis, call directly: the conversation and the intervention contract happen within 2 hours.

Sources

Facts checked on 24 October 2026 against the original sources.

Is something holding you back?

Call me or write to me. The first conversation is complimentary, and I reply within 48 hours.

Take the rapid assessment

Four questions, two minutes. Nothing is sent or stored.

By phone:+33 6 74 86 88 70WhatsAppEmail