Software Updates and Chaos in the Stock Market

Software updates, though seemingly beneficial to the prosperity of a company, have been the cause of catastrophic mess-ups in financial markets: the Knight Capital (KCG) computer glitch that cost it $440 million is an example of one such failure.
In March, electronic exchange BATS had to cancel its own initial public offering (IPO) when a previously undetected bug in its new IPO auction software reared its head at the worst possible time, despite months of internal testing. In May, Nasdaq’s IPO software encountered a major error during the early stages of the hugely anticipated Facebook (FB) IPO. In this case, just like BATS, Nasdaq had performed “thousands of hours of testing, replicating “a hundred scenarios” aimed at anticipating problems,” Nasdaq Chief Executive Robert Greifeld said.

However, now it comes to this. Though Knight declined to comment, more and more market watchers believe the glitch occurred in a piece of code that Knight unveiled Wednesday morning to prepare for the New York Stock Exchange’s Retail Liquidity Program, also launched Wednesday morning. NYSE (NYX) says it made the program’s technical specifications available for 240 days before the launch, and its testing environment was open for six months. It was up to the trading firms, such as Knight, to adjust their software. “Clearly, some wrote good software, and some wrote bad software,” says Manoj Narang, founder and CEO of high-frequency trading firm Tradeworx.

The domain of automated trading programs essentially lies in today’s stock market. Every second, for every one thing that goes wrong, there are millions of things that go right. But the recent incidents in which software that supposedly has been rigorously tested crashes and burns raise an alarm. One such follow-up question is whether the testing that firms implement on new pieces of code and software is effective enough.

“They’re absolutely not being adequately tested,” says Eric Hunsader, CEO of market data service Nanex. “That’s the problem for sure.” Considering how intricate the markets are, it’s almost impossible to predict everything that can go wrong. No such simulation or trial will capture all the variables of live trading.

High-frequency trading has no industry-wide ideal practices that govern the internal testing of developmental systems and software. The industry should come up with protocols based on standards similar to those of the International Organization of Standardization, which has developed some 19,000 methods of standardized best practices covering almost all aspects of
technology and business. All this, except for high-frequency trading.

Rick Cooper and Ben Van Vliet, two professors at the Illinois Institute of Technology are proposing a plan for the development of industry standards. The two have written a piece outlining basic principles for monitoring the behavior of a high-frequency trading algorithm in real time and shutting it down if unusual glitches occur. Published this spring in the Journal of Trading, the paper suggests methods the industry could monitor itself. The paper argues for outside monitoring systems that act like a circuit breaker. If trading volume and profits and losses aren’t following a pattern of expected behavior, the “circuit breaker” can shut down the algorithm. “That way you dont screw up the marketplace,” says Cooper. In the case of Knight, you simply, utterly have to blow yourself up.

What is shocking about Knight’s glitch isn’t that it happened, but that it took the firm so long to respond. According to a person familiar with the situation, who would speak only on condition of anonymity, it was the New York Stock Exchange that alerted Knight to the problem early Wednesday morning. It took Knight 30 to 45 minutes to get things up to speed. “Any time you’re starting a new system, you have to be aware of the heightened possibility of something going wrong,” says Narang. “You have to watch it like a hawk.”