Another good software disaster

The good software disasters are the ones where nothing goes wrong and everything works.

Knight Capital’s suicide bid on 1 August 2012 was one of those. They have never admitted what caused them to buy shares at the offer price and immediately sell them back at the (lower) bid price, losing a little money each time, between 40  and 100 times a second, on many different shares, for almost half an hour, but the description at Nanex, although based on inference, seems close to the truth of how software development works:

  1. They developed some new market-making software: software to offer shares to people who want them and to bid for shares at a lower price from people who don’t.
  2. They had to test this.
  3. So they wrote a test program which placed orders to buy at the offer price, to see that the market-making software handled them properly, and which placed orders to sell at the bid price – very fast, to make sure that the software could handle high volumes of transactions.
  4. The tests worked.
  5. So they linked the market-making software into New York Stock Exchange’s systems, and at 9.30am on 1 August 2012, it went live.
  6. It worked.
  7. They accidentally included the test program in the software that went live.
  8. It worked too.
  9. So, many times a second, Knight Capital asked to buy a share at the offer price and immediately afterwards asked to sell it at the bid price, losing a little money every time and driving the prices wild.
  10. If the test order went to Knight Capital’s own market-making software, nothing much happened. Knight Capital was losing money to Knight Capital. But when it went to another market-maker, it was a real purchase or a real sale. Knight Capital lost a quarter of a million dollars every second. In half an hour, the rest of Wall Street was 440 million dollars richer at Knight Capital’s expense.

Because the test program was a test program and not doing real trades, there was no need for it to record the trades it was making, so it didn’t. The only way to tell what was happening while it was happening was to watch the monitors go wild. Here is a series of monitor pictures, at increasing levels of zoom, from the middle of the Knightmare. They are actually rather pretty.

But the thing to remember is that at all times and in all ways, all the software worked exactly as it was designed to work.