A report at Ars Technica compares the computerised financial markets to a vast and infernally complex piece of multi-threaded software running on hardware that was never designed to cope with it (or vice versa), before telling us what I suspect most of us have already guessed: it’s a gigantic house of electronic cards. But ironically enough, part of the problem stems from the very transparency that the shift to electronic trading was supposed to bring with it:

Unlike the market of an earlier era, where humans executed trades by talking to (and shouting at) one another, the electronic communication networks (ECNs) that emerged in the late 70s logged every detail of every trade for later auditing. No more “he said, she said” when resolving a dispute or ferreting out fraud—just go to the tape. But then came the flood.

After a solid decade of moving almost all trading activity onto electronic systems (the NYSE floor is just there for show at this point), the market generates so much data that it’s nearly impossible for a mere governmental agency like the SEC to analyze. There are literally tens of thousands of quotes per second in hundreds of thousands of symbols across multiple electronic exchanges—the SEC would need the brain and computer power of the NSA to even begin to do a credible job of crunching this many numbers for a credible post mortem.

[…]

The amount of data isn’t just a problem for regulators. Much of the report details how the systems of the market participants were themselves overwhelmed in real-time with the sudden surge of digital information. Processing began to slow, queues filled, backlogs developed, and machines were eventually pulled offline as the humans intervened and tried to sort out possible data integrity issues.

Beyond the challenges of reconstructing events, the traders also use some subset of the data firehose that the market’s machines throw off today as input to train the algorithms that will run the market tomorrow. So at some point, we’ll wake up and realize that it’s really turtles machines all the way down. Put that in your bong and smoke it, Keanu.

Ouch. And it gets worse, too; go read the whole thing. I think the best way to sum it up in layman’s terms is that we’ve turned the financial markets into something a little like one of those “game of life” software ecosystems… which would be quite a fascinating idea if it weren’t for the fact that unexpected interactions within that ecosystem can affect meatspace in a pretty serious way.

The more I learn about derivatives and futures and all that “clever” quant stuff, the more I think it’s a bunch of hubristic mathematical voodoo bullshit that we’d do well to get shot of sooner rather than later; the only people it really seems to benefit are the wankers who thought it all up in the first place.