Fast money: the battle against the high frequency traders

Michael Lewis - Flash Boys

Back in the 1990s, when doctors had all but given up on holding the American tobacco industry to account, the lawyer Michael Lewis found an ingenious way through their defences.

If juries couldn't see past individuals' responsibility for their habit, he reasoned, why not sue on behalf of individual states and their smoke-burdened Medicaid systems? Lewis gathered a team and tested the approach. In 1994, on behalf of the state of Mississippi, he sued 13 tobacco companies for the cost of treating cigarette-related illness; they were joined by another 38 US states and, under intense pressure, Big Tobacco agreed to pay $368.5bn. The industry's nemesis became legal legend.

A proud Mississippian, Lewis wears his intellect with quiet southern grace; he listens before speaking and never interrupts. In almost half a century of legal practice, he thought he'd seen everything, but over 14 years of retirement he watched the world lurch towards a new technology-driven feudalism, defined by disparities in wealth that were more keenly felt in his own state than most. He was feeling ready for another dragon to slay.

As is often the way, in the end the dragon found him. In April 2013, he was at a baseball game with his friend Professor Bonnie Van Ness, head of the University of Mississippi's school of business administration. Lewis asked after her professor husband, Robert, who was presenting a paper in Boston. What was the paper about?

"Quote stuffing."

"Quote stuffing?"

This is the practice by which stock market players called high-frequency traders slam vast numbers of orders into the system, cancelling them before anyone can react, with the aim of slowing the transit of information to competitors, or of creating confusion from which they can profit – all in the space of milliseconds.

"I'm certain you must have that wrong, Bonnie," Lewis said, "because that would be quite illegal. It would be market manipulation."

No, she said, quote stuffing was as real as the teams on the field. The stock market was a new wild west running on rocket science: the stuff going on there was mind-blowing. The rest of the game passed in a haze.

Van Ness mailed Lewis the paper. Over the coming weeks, he read everything he could on this phenomenon. Here was a market beyond human control, dominated by super-fast machines running complex computer algorithms that jostled and fought each other at the level of milliseconds, microseconds – and with no meaningful oversight. The familiar cliche of gaudily dressed men waving arms on a stock market floor was history: trading now happened within black boxes housed in highly secure, unmarked "data farms".

Not only that, the algorithms at the heart of this world were run not by finance or programming people, but by "quants": quantum physicists, climate scientists, theoretical mathematicians. Some of the most formidable minds in the world were now employed in a technological arms race, a hidden war stalked by million-dollar predator algorithms that could swarm those of the larger, slower players – typically, pension and mutual funds – in the same way a shoal of piranhas might an ox, cutting them to shreds and pocketing the profits. The regulators couldn't keep up. If they tried, the algos simply mutated.

To Lewis, it looked as though the finance industry had found a new way to fleece the public – the ordinary savers and workers with pensions. Already, the system had caused several big scares, most notably the "Flash Crash" of May 2010, in which $1 trillion was wiped off the value of markets in the space of 10 minutes. Yet, the deeper he delved, the more forcefully a depressing truth emerged: that a fortress of legislation had been built around the stock exchanges and powerful traders (mostly banks and hedge funds) by lobbyists and politicians. Immoral as these practices might be, they were no more illegal than had been the packaging of sub-prime mortgages into tradable securities prior to the crash of 2008-2009.

Lewis pressed on, but his mood grew bleak. Then he met Eric Hunsader.

Eric Scott Hunsader has lived in Chicago for years, but he's never been to a place quite like this – the Chicago Executive Airport, whose Comfort Inn frontage belies a runway full of Learjets out back. It is 6 May 2014 and for the past four years, Hunsader, one of the most gifted programmers in the country, has felt like a lone voice in the woods after stumbling across high-frequency trading (HFT) and being shaken by what he saw. Finance insiders have branded him a conspiracy theorist or – absurdly – a luddite, but now the world seems to be waking up. Next to him at a rosewood conference table is not just the renowned Big Tobacco slayer Michael Lewis, but Lewis's "dream team" of class-action lawyers, whom he must convince of the evidence – a tough job, given the complexity of HFT. He can see some of the team struggling with the material, having doubts, but then one by one getting it.

Hunsader tries not to feel excited, knowing what the stock exchanges and banks will do to protect their interests against any lawsuit. It helps that since he and Lewis first met a year ago, the bestselling book Flash Boys (written by the other Michael Lewis) has opened up HFT to wider debate. Lewis smiles as he invokes the movie Star Wars and the metre-wide ventilation shaft Luke Skywalker uses to destroy the Death Star, hoping to have found its stock market equivalent. He's even taken to calling this strategy "the Hunsader torpedo", after the man who pointed it out.

If the torpedo works, Hunsader will be at the heart of one of the legal battles of the fledgling century – The People versus Big Finance. He looks about the room full of top lawyers and – not for the first time – wonders: how the fuck did I get here?

6 May 2010 opens like any other day on the markets. Asia is quiet, the US and Europe jittery as UK electors trudge to the polls, while Greeks hit the streets to protest austerity. Stock prices have been rising through the year, buoyed by waves of cheap credit, but now the mood is darkening, and every time Athens hits the TV screen, another few points drift from the Dow Jones Industrial Average like teargas. By 2.30pm, it is down 2.5%: hardly catastrophic, but worth a weather eye.

And then something unexpected appears – a flutter in the price of E-mini futures contracts, an investment vehicle traded on the Chicago Mercantile Exchange and regarded as a bellwether of wider sentiment. Almost no one notices, until the flutter becomes a shiver, then a spasm, amid whipsawing prices as the E-mini's vertigo spreads to other stocks and exchanges, and indices begin to plummet.

Within seconds, the Dow has lost 100 points. Finance workers turn back to their screens. But seconds later, another 100 has been shed and managers fly from their offices, yelling, "Pull everything!" as traders hit buttons and hammer keyboards, cancelling orders in an attempt to limit damage. In horror, they gather in communal spaces and watch price lines dive with eerie, implacable momentum, like lines scratched by an angry child.

300 points down…

400 points…

500 points…

At 600 down, the Dow has fallen further than it did on news of Lehman Brothers' collapse in 2008. But that crash took a day: this spans minutes. At 5% down and in apparent freefall, traders will report feeling nauseous, sick; a sense of staring into oblivion. Even 9/11 failed to rock the market like this – which implies that something catastrophic has happened. But what? The CNBC pundit Jim Cramer, an experienced former hedge fund manager, roars: "The machines broke – these are not real prices!" and wonders why no grown-up is stepping in to shut the show down. But the trouble is, nobody can: circuit breakers designed to halt trading after unnatural price swings work only until 2.30pm and it is now 2.47pm, with the Dow racing towards an unprecedented 1,000-point loss and almost $1tn wiped from balance sheets.

Then something even stranger happens as, with Armageddon approaching, the market turns tail and begins to rise, just as impossibly as it fell.

600 down…

400…

300…

200…

The traders breathe again. The whole episode, the most dramatic in stock market history, has occurred within 10 minutes. Welcome to the world of HFT and the Flash Crash.

Eric Hunsader never played the market himself: programming was his thing. At school he had been funnelled towards a career in medicine as a kind of smart kid default, but when he discovered coding, it was like falling in love. He adored the way you did "x" to make a computer do "y", and if it didn't, you knew you'd made a mistake; you got to the bottom of the glitch and fixed it. Beautifully simple, and certain.

Back in the 1980s he'd written a program aimed at pre-empting stock market movements for profit, but trading was too nerve-jangling for him; he switched to designing such software for other people. From one such project he earned real money, which enabled him to build a house and write his magnum opus, a program he called NxCore. This used compression technology to collect, store and relay stock market data to clients on behalf of his own new company, Nanex. Hunsader never cared what the data he collected said: he just wanted it to be instant, accurate and easy for his clients to use. As a result, he saw the market differently from most. Where others obsessed over price, to him it was all about data.

Most days Hunsader would focus on coding tasks at his office in the bucolic Chicago suburb of Winnetka, while the software ran itself, but today had been different. As always, the blinds were drawn; there was no radio or TV, just a flood of US stock market data jinking colourfully across his array of screens. The software allowed him to see every stock and derivative traded on all 13 US exchanges in real time, down to the millisecond.

For all its jargon, the stock market works like any other: there are sellers and buyers. Owners of shares offer a specified number to market at a particular price (the offer price, or "ask"), while buyers bid for a stated quantity, at a named price. If there is a gap between them (the "bid/ask spread"), then one side has to move up or down for a trade to happen. If no one moves, then both sides remain on the system as "quotes".

Simple. Except that over the past couple of years, Hunsader had noticed a marked increase in the number of messages entering the system, mostly in the form of split-second changes to the price or size of quotes, despite the number of actual trades remaining constant. He didn't know what was generating this background chatter, but it was of little concern: so long as the market functioned and his clients were happy, so was he.

That day, Hunsader had watched the screens more closely than usual, until just after 2.30pm when, in the blink of an eye, the network exploded with the highest message rate he had seen. Worried about the strain on his system, he stopped looking at prices and began to monitor volume, so while the rest of the world watched prices fall off a cliff, Hunsader saw something different: a system overrun by information, swarmed like a wheat field by locusts.

News programmes spat theories ranging from Eurozone fears, slack monetary policy and collective memories of 2008, to Cramer's "broken machines" notion or a "fat finger" incident, in which some hapless trader typed "billion" instead of "million" and spooked the herd. None of which rang true, because who would sell Accenture or CenterPoint Energy – solid companies then trading at around $40 and $13 per share – at 5 cents? Or, albeit fleetingly, price Apple (then around $250) and Sotheby's (around $35) at $100,000 a share? Then, Reuters ran a leaked report that at 2.40pm a large institutional trader had placed an order to sell a whopping 75,000 E-mini contracts on the CME. Such a large order, executed suddenly by an automated program, had panicked others into selling. By the end of that first evening, the rumoured culprit even had a name: Waddell & Reed, an asset management firm based in Kansas.

The government-run Securities and Exchange Commission (SEC) announced an investigation, for which it would have to assemble mountains of data – a gargantuan task that would take months. But Hunsader already had the data and could do the job in a trice. "Let's just run it," he told his lieutenant, Jeffrey Donovan. "Let's commit one week to going through it and see what we find."

How did a bad day become the Flash Crash? Prices had begun to fall precipitously at 2.42pm, so Hunsader examined that minute and saw the message rates leap skyward in the 44th second. He extracted that second and stepped into it like a child into a derelict house, feeling his way, until at 75 milliseconds he found the point of departure: a starburst of data with message rates bumping against the exchange systems' capacity to deal with them. The computers betrayed their British heritage by forcing messages into an orderly queue: Hunsader could feel the machines groaning as the data cycled round.

And now a thought occurred. More a "what-if" than a solid theory. Hunsader called up video of CNBC's Flash Crash coverage and focused on the timeline at the bottom of the screen. Comparing the TV time stamp with Nanex's own, he made a remarkable discovery: that the price information going to the Securities Information Processor – the so-called "SIP" feed upon which participants rely for live market data – had been delayed. Meaning that when pundits and the public still thought the Dow was in freefall, it was in fact rallying hard. Anyone who sold near that false bottom would have lost their shirt. And anyone who bought made a killing. Hunsader had no way of knowing who was who, because this information was deemed confidential. It was like watching a game of poker between players wearing masks.

Cramer had been right. Overloaded with information, the exchange system had broken down. Quotes on General Electric shares experienced a 24-second delay at one point and other stocks hit 36 seconds, yet no one appeared to have noticed – or, if they had, they weren't saying. As Hunsader pulled his focus back, he saw something even more surprising: patterns, often repeating over hours or days or weeks, clearly aiming at stealth. Here were machines placing and cancelling the same order 10,000 times in a second, or stepping from one share all the way to 100, one at a time, and then marching back down again in milliseconds, over and over and over.

The Nanex software rendered these anomalies as shapes, so Hunsader and Donovan took to calling them crop circles, and even gave them names: Wild Thing, Zuma, The Click, Living On The Edge. The pair quickly realised that many were designed not just to manipulate the system, but to avoid detection by each other, even to lure slower algos into traps. Was it possible that the Flash Crash had been caused on purpose, by "stuffing" the New York Stock Exchange with messages in order to slow it down and create fleeting price discrepancies, upon which the fastest machines could pounce? When Waddell & Reed took the highly unusual step of asking Hunsader to examine their algorithm and its actions on 6 May, he quickly saw that it could never have caused the Flash Crash. The truth lay elsewhere.

A few insiders were already raising questions. Dave Lauer studied computer science, international finance and philosophy at graduate and postgraduate level, before entering finance on the technical side. A voluble character with a naturally inquiring mind, he soon moved sideways into HFT trading, but quickly developed misgivings. "I was making a lot of money, but I didn't like it," he explains. "At the time I made this rationalisation, saying: 'Well, high-frequency trading is providing the lubricant which allows the wheels of capitalism to turn.'"

This is the standard justification of HFT, which the Flash Crash finally demolished for Lauer, who followed his moral compass and left. Had he expressed his concerns to colleagues? "I tried to. One of my friends who was in HFT – who's just left – said: 'Well, what did you expect? Why did you think you were getting into this?'" Then my wife got pregnant, and that was another contributing factor. Because I didn't quite know how I could explain to my daughter how we got so rich – which we would have been if I'd continued on that path. But why? I couldn't quite figure it out.

"I think there are two major reasons people go into HFT. The first, and probably 95% of it, is the money. But the other is that scientists like a really hard problem, and you're trying to figure out these very difficult intellectual problems. And validation is immediate: you find out instantly whether you've figured it out or not, and are compensated very well when you have. That's probably why, from a scientific perspective, there's a lot more appeal to it than your traditional scientific research and publishing track."

Lauer put me in touch with Ezra Rapaport, who trades in New York. Rapaport studied computer science at Harvard and loves puzzles; he can solve the Rubik's cube in under 90 seconds. He describes his job as "being given access to a certain amount of capital, and then I have to put as much of that capital into the market as possible. I am basically trying to solve the world's most difficult puzzle every day, which is what the markets are." It's a high-adrenaline game. "You've got computers dealing with real money, placing orders and bonds. The capacity to bankrupt the entire operation – the capacity for loss – is immense. And the fact that it can happen faster than a human can step in and pull the plug is something that keeps me up at night."

It's the risk that keeps him up, rather than any personal scruple. Of HFT, he says, "It is absolutely something that was going to occur anyway, because technology will penetrate all areas of society. I compare it to Formula One: we seek to be innovative within the same market as the other teams. So you get very well-financed, talented teams seeking that edge." Is he motivated by greed? "Definitely. Do I have any existential or moral issues with that? I don't. We're not seeking to improve the state of technology – we push the envelope to seek profit. The way I redeem it is seeing the benefit it has for my child and my wife."

But while Rapaport is unrepentant, Lauer believes the root problems with HFT could be dealt with quickly and easily. Which raises the obvious question: why is no one doing anything about it? It's one Hunsader also found himself asking when the SEC, after a five-month investigation into the Flash Crash (in which he had proffered help), laid blame squarely at the feet of Waddell & Reed – the one market participant he knew could not be responsible.

In the last seven years, the trading environment has changed out of all recognition, spurred largely by the introduction of new regulation at the end of 2007. Designed to increase transparency, this forced brokers to offer clients the best possible price across all 13 US exchanges, meaning that price information had to be synchronised – a fiercely difficult technical challenge that rendered the market hard to monitor and abruptly brought Einsteinian physics into play, as data raced between exchanges at light speed.

Since then, big HFT firms have gone to great lengths to gain tiny time advantages, both through software (the algos) and infrastructure. Mountains have been drilled through; billions of dollars, euros and pounds spent constructing gargantuan automated data centres (some with duplicates as backup, so expensive is down-time) and microwave transmission systems, such as the one between Frankfurt and London, because light travels marginally faster through air than through fibre-optic cable.

The NYSE was the first stock exchange to build its own data centre, at Mahwah, New Jersey (with an identical European sibling in Basildon, Essex). Finding engineers to describe what goes on inside this 37,000 sq metre (400,000 sq feet) hangar is not easy, but once you have, the eccentricity of the microverse comes comically alive. NYSE began by building a room called a pod, in the centre of which was the "matching engine", a central server that gathers data and distributes it to market participants for a fee. For a higher fee, however, HFT traders could "co-locate" their own algorithmic computers inside the pod. And because space inside Pod 1 sold quickly, a second was built. When the traders in Pod 2 complained at being farther from the matching engine, a Pythonesque comedy began. "I always thought that was funny," laughs one engineer down the phone, "because a foot of cable equates to one nanosecond's difference – a billionth of a second. And when this first happened, the machines most of these guys used couldn't even differentiate nanoseconds."

But the NYSE centre is regulated, unlike private, third-party data centres, and has to be seen to be fair. So NYSE came up with an elegant solution, by measuring the distance to the farthest HFT server and giving everybody the same length of cable. This worked until engineers noticed that traders now wanted their servers located farther from the matching engine, because light signals bouncing around inside a long, coiled cable would in principle degrade and slow infinitesimally in comparison with a straight cable. In response, NYSE ran traders' cables around the walls to minimise bend. Laugh at this Marx Brothers farce, however, and systems people will remind you that computers don't work the way we do. Our "time" is meaningless to them: a nanosecond might as well be a second, or a century, because the message that arrives first gets the trade.

Traders' servers are housed in black cabinets whose contents are jealously guarded. Players pay for their feed, cabinets (one large trading firm is said to run 60), a power supply and cooling system, with some now graduating to huge "super-cabinets", which allow for yet further reduction in cabling. High-frequency traders buy an insuperable advantage over the public – and none of this is illegal.

Hunsader watched and blogged what he saw as anomalies. A major HFT firm called Knight Capital, the largest stock trader in the US, imploded when a test algo was mistakenly used for live trades, buying and selling 2.6m shares per second in an electronic fit lasting 45 minutes and costing $440m. On another occasion, a mysterious "ghost algo" appeared out of nowhere, placing and cancelling orders in 25-millisecond furies up to 10.30am each morning for a week, accounting for 4% of traffic on the US stock market without executing a single trade – then vanishing. A hoax tweet claiming that President Obama had been injured in twin explosions at the White House caused another terrifying slide, which became known as the Hash Crash – because the machines were now reading Twitter. By the end of 2013, Hunsader counted an average of two dozen "mini-flash crashes" in individual stocks every day.

The numbers were now mind-boggling. In the 1960s, an average share in a company would be held for four years, a figure that had fallen to eight months by 2000, en route to two months in 2008. But now the average share was held for 20 seconds, with 10 on the horizon and the fastest machines able to fetch quotes in a millionth of a second. Within two years of the Flash Crash, HFT accounted for 70% of market activity in the US and almost 40% in Europe. Meanwhile, more than 50 so-called "dark pool" markets had risen to shelter slower traders behind anonymity, but most of these had been colonised by predator algos, with some deteriorating into "toxic dark pools", where no one was safe. Algo-designing quants now earned hundreds of thousands, even millions a year and NxCore was processing a trillion bytes of information per day. The faster the machines got, the less anyone could tell what was really going on.

Some things were clear, though. Hunsader could never attribute motives to the machines, but could see the way large pension and mutual fund bots were lured into traps: the algos probed them for signs that they were about to buy or sell, and then used superior speed to get them to sell lower or buy higher. High-frequency traders referred to their prey as "whales", "low-hanging fruit", "dumb money" or "dinner". The stock market had been designed as a means of allowing capital to flow where it could be most useful, allowing individuals with excess money to invest in the energy and ideas of others. But HFT was nothing to do with value assessment or creation: the machines' sole aim was to use speed to game the market. Ordinary savers were being robbed and traditional investors were leaving the market, forcing capitalism farther away from anything either Adam Smith or the NYSE founding fathers would recognise as virtuous.

Some trading firms were even operating from unregulated jurisdictions such as Russia and the Czech Republic. What was to stop terrorists entering the frame? On the face of it, nothing. In Money And Speed, a 2011 documentary by the Dutch film-maker Marije Meerman, the SEC's Gregg Berman made this extraordinary confession: "There is no air traffic control for the market… there is no one person or group that in real time sits and watches. Oversight would cost government and the taxpayer too much, so it won't happen."

Not only that, but the invader algos were multiplying. Nanex detected one it took to calling The Disruptor, because it slammed so many millions of orders into the system that the whole market shook. How could this be allowed? The Nobel Laureate economist Michael Spence called for an outright ban in 2011, and European MEPs threatened action; their proposals were rejected by the UK's Coalition government. When Hunsader's finance friends pointed out that nobody was driving busloads of children over cliffs, he would grab their wallet and remove a $20 bill, then hand the wallet back. "Does anyone in the world really care what just happened there?" he would ask. "It makes no difference to anyone but you, and even then not much. It's just that in a civilised society, we don't tolerate that. Civilisation breaks down when people don't follow the rules, because nobody can trust anybody else."

Or as Thomas Peterffy, the Republican billionaire founder of Interactive Brokers, put it: "Basically, the people whose money is in the large pension funds and investment accounts, they do not understand what is happening. If they understood that it is their money that the banks are basically taking from them, then they would do something about it. But the problem is that it's too complicated."

Some people think the stock market has become almost impossible for anyone to understand. Professor Neil Johnson specialises in complex systems and has been studying financial markets for years. A British academic working at the University of Miami, he previously subscribed to the conventional view that, if you wanted to understand markets, you examined long-term trends; that anything occurring on the scale of hours, minutes or seconds was just noise – statistically insignificant. When Johnson heard about Hunsader's inquiries, however, he was intrigued: he flew to Chicago, where the Nanex data thrilled and unsettled him in equal measure. Most unexpected among his discoveries was that in the runup to the 2008-2009 financial crisis, a concentration of "mini-flash crashes" occurred in banking stocks, for the likes of Goldman Sachs, Morgan Stanley, JP Morgan and Lehman Brothers.

"To be honest, that still shocks me," he says, "because it suggests a link between what goes on at a sub-second level and what happens on the scale of months. At that point it started to look like an ecological system. Because in an ecological system, you have predators of all sizes… And I think this is what we're seeing. The algorithms are all looking for and picking up some kind of weakness in those particular bank stocks, picking away at them, picking away. So I'm not saying they caused the crash, but they were like sensors of the impending bigger weakness."

What are the implications? "It's fascinating," Johnson says. "I mean, people have talked about the ecology of computer systems for years in a vague sense, in terms of worm viruses and so on. But here's a real working system that we can study. The bigger issue is that we don't know how it's working or what it could give rise to. And the attitude seems to be, 'out of sight out of mind'."

Significantly, Johnson's paper on the subject was published in the journal Nature and describes the stock market in terms of "an abrupt system-wide transition from a mixed human-machine phase to a new all-machine phase characterised by frequent black swan [ie highly unusual] events with ultrafast durations". A scenario complicated, according to the science historian George Dyson, by the fact that some companies are now allowing the algos to learn – "just letting the black box try different things, with small amounts of money, and if it works, reinforce those rules. We know that's been done. Then you actually have rules where nobody knows what the rules are: the algorithms create their own rules – you let them evolve the same way nature evolves organisms."

Could the system collapse in a monumental, global Splash Crash? The inventor and thinker Ray Kurzweil, now head of engineering at Google, has long predicted a moment he calls "the singularity", when the machines take over. Could the incubator for such an event be the unregulated stock market? The thought seems outlandish, almost too exotic to voice, and yet not one of the experts approached for this article would dismiss it out of hand.

The British quant Paul Wilmott, a crack algoman turned HFT critic, is that rarest of things: an expert who doesn't rely on the industry for a living and is free to speak his mind. He has been alarmed by this brand of trading for years, he says, but is now more concerned with the broader context. "My own stance has changed, in that I'm becoming increasingly cynical and focusing less on the mathematical modelling, worrying more about the human nature side of the thing – the way the banks find some bandwagon and try to exploit it as much as they can. And when people get worried, they claim there's nothing wrong with getting two or three years of profit out of this socially useless bandwagon, then moving on to another… We're all just servicing the banking industry now. This is what I find upsetting."

But chinks have appeared. Haim Bodek was also an elite quant, rich and successful, when the tide inexplicably turned and he started haemorrhaging money. He spent an anguished year pulling apart his algorithms, trying to work out why competitors were suddenly beating him to every trade, until, drowning his sorrows at a party one night, an amused insider let slip a secret: a cabal of favoured traders were being given preferential treatment through "special order types" – equivalent to a different code or language – which enabled them to jump the queue. The insider suggested Bodek do the same, because his algos were running through treacle compared with these favoured few.

Bodek's Jewish family had been caught up in the Holocaust, where scientists turned a blind eye to how their technology was being used, so he accepted his invitation to join the club, but was plagued by conscience and finally went to the SEC, knowing he would become a pariah. When, towards the end of 2013, a public outcry greeted news stories that the same subset of traders could pay for a private data feed that was faster than the one everyone else used (something traders had known for years), Bodek could only laugh. Nothing, as usual, was done.

The son of an experimental particle physicist, Bodek now lives in Connecticut, and is still angry. He thought long and hard before going public with his discovery and expected the issue to blow up quickly once he had. Instead, traders and exchanges argued that these secret "undocumented features", immoral as they might appear, were perfectly legal. "They don't think of themselves as criminals," Bodek says. "They think of themselves as the good guys. They don't want to believe that in the last 10 years they've just been doing securities fraud. They want to believe they've outwitted everybody."

He claims to have been effectively blacklisted from the industry for two years, and the hardest part to take was that the people from the mutual and pension funds were the most dismissive, despite being the victims of HFT. "Apparently, the idea that I saw something bad and called it out is not allowed. Acknowledging what I said would be admitting they didn't know this stuff. It's like I was this inconvenience to the victims of Wall Street." Only this year has the tide begun to turn, with Bodek's actions increasingly seen as those of a leader within the industry.

The lawyer Michael Lewis's first thought was to build a case for market manipulation and insider trading against the HFT firms. But the more he spoke to Hunsader, the more he began to accept that this would be too complicated for judges to follow. Instead, Hunsader nudged Lewis in the direction of the two most serious advantages HFT traders had been handed by exchanges: the faster language or secret order types, and the private data feeds. US stock exchanges promise to supply subscribers with timely and accurate information via their standard SIP feed, for which users pay a total of $500m a year. It appeared to Lewis that the exchanges were in breach of that contract. "The information they were providing was not timely or accurate, and wasn't fairly distributed. Here's what's happening: the real market is represented by these private data feeds. The illusory market, the market that the investor sees when he looks at his monitor, is anywhere from 1,500 to 900 milliseconds old. That doesn't sound like much, because the blink of an eye is 300ms. But that's a long, long time in the world of HFT."

Should it be allowed? "How can it possibly be allowed? The profits being made are coming out of ordinary investors' pockets."

Lewis pauses. For him, more is at stake than just the future of HFT and he will later characterise the impending litigation as "a small skirmish against the larger backdrop of the vast accumulation of wealth and political power" that is currently undermining democracy in the west. "In many ways we've all become vassals again," he says. "And it's this simple: the people who are writing the rules are winning the game. This is just one example. But we have to start somewhere."

For now, that "somewhere" is a court in the southern district of New York state, where on 22 May a class action complaint was filed against all 13 US stock exchanges and their subsidiaries, a prelude to what could be one of the most significant courtroom dramas of the century so far. The first line of the lawyers' 40-page document reads: "This is a case about broken promises."

@wiresmith

• The headline of this article was updated on 7 June 2014. It was originally "Inside the murky world of high frequency trading".

Powered by Guardian.co.ukThis article was written by Andrew Smith, for The Guardian on Saturday 7th June 2014 09.00 Europe/London

guardian.co.uk © Guardian News and Media Limited 2010

 

Best place to workThe Best Firm of the Last Decade is...

Register for Financial Markets News Alerts