VHAYU BLOG

Aite Group Report: Managing the Market Data Explosion

An independent Aite Group research study advises that financial services firms confront the shortcomings of their existing enterprise data architectures today--before market data volumes once again ascend to the limits of technology's capacity to support them when better markets return.

For this report, Aite Group conducted interviews with market data heads at 10 firms ranging from small, high frequency trading shops to some of the largest buy-side and sell-side firms in the world. They also spoke with several exchanges, hosting providers, and market data providers.

Key findings include the following:

• Aite Group expects U.S. equities message traffic to double from current volumes in the next two years.
• U.S. equity data storage needs are currently growing 18% per month.
• Market fragmentation in Europe (and eventually Asia) will cause multiple repeats of current U.S. equities market data issues.
• Foreign exchange (FX) storage requirements are growing at twice the rate of equities storage requirements.
• There is a high level of interest by electronic trading firms to add unstructured content into trading models.

The white paper examines electronic trading trends associated with market data volume, and considers how those trends may evolve in the future.

DOWNLOAD NOW

Market data platform crash?

On September 18, 2008, the US equity markets established a new single day record, both in terms of trade and share volume, with over 60 million trades and over 600 million Level 1 quotes disseminated. Did your firm's infrastructure keep up or crash?

Most Vhayu customers had no issues recording the US equity market on 9/18 - the few that did have problems were not Vhayu-related but because of telecom and/or market data fat pipe provider issues. Vhayu installations, even one that was running on a dual processor machine with only 2Gb of memory, had no problems recording upwards of 700 million pricing events from 9:30 - 16:00 on 9/18.

Vhayu's core competency is providing firms with an industrial strength market data platform that records market data with zero tick loss in a fault tolerant, highly reliable, highly scalable manner.

With the demise of the investment banking model, and thus the prominence of the leveraged proprietary trading desk on the decline, the emphasis for market data platforms in the near-term will shift from the race to zero in terms of latency to the circle the wagon's bunker mentality of regulation, oversight and providing price transparency and accountability to all phases of trading operations.

Is your market data platform handling the explosive volumes we are seeing during the credit crunch? In less than a year, we will see well over 1 billion market data prices in a single trading day for US equities - Vhayu will handle it - what about your market data platform?

Market Data Gravity

The sheer gravitational force of the market data plant is pulling the rest of the trading assembly line into its complex. Applications such as OMS's, CEP platforms, P&L blotters, portfolio risk analysis, TCA, quantitative modeling and compliance monitoring all have an ever increasing need to quickly and efficiently marry global market data to their siloed application data.

In the past, market datasets were small enough to allow them to be pulled in a reasonable amount of time to each of these trading applications independently, or if there was enough system downtime between successive business days, it did not matter if it took a few hours every night to pull the data. Now a few hours have grown into at times half a day to get, for example, all North American equity quote data.

The gravitational pull of the market data plant will continue to strengthen and force more and more trading applications to import some or all of their data and/or analytics into a market data platform such as that provided by Vhayu. For most of the applications previously mentioned, the growth rates of the siloed datasets pale in comparison with the growth rates of the market data universe. The only way that trading houses can hope to keep up with explosive market data volumes is to integrate the trading desk data silos into the market data plant.

More and more of Vhayu's equity customers are bringing their internal order flow into our market data platform for analysis on a historical basis. The next logical step is to do this on a real-time basis. On the fixed income side, Vhayu customers have imported bond reference data history, futures contract information and spot FX trades directly into the system. Prospective customers have approached Vhayu recently about how to integrate a historical bond (govies and corporate) ratings database into their platform, in order to do global bond portfolio re-balancing and risk assessment analysis.

As many of the competitive and collaborative players such as exchanges, broker/dealers, banks, hedge/pension/mutual funds, regulatory bodies and government banks face the ever increasing volumes of trade data, which of these entities will continue to resist the pull of the market data plant versus those who embrace it and adopt platforms such as Vhayu remains to be seen.

Market Data Monopoly

The goal of the popular board game Monopoly is to bankrupt the other players. By owning the high rent properties Park Place and Boardwalk (if only the game's originators could see Atlantic City today), a player's odds of draining the most funny money out of the competition are pretty good.

A recent report generated by SIFMA titled "An Economic Study of Securities Market Data Pricing by the Exchanges" (http://www.sifma.org/legislative/financial_services/pdf/SLCG-Market-Data-Study.pdf) challenges the SEC to investigate whether NYSE Euronext and NASDAQ OMX are exerting monopoly pricing power by charging brokers and retail investors fees that are well above the cost of consolidating and distributing data.

This brouhaha caught my eye because I thought the report would shed light on market data infrastructure costs, which would be useful to how we position our software for analyzing and storing data. One of the main arguments SIFMA uses to make its case concerns the use of depth-of-book data and whether NYSE and NASDAQ have the right to charge as much as they do for it, since they conclude "there is an inelastic demand" for it and the markup price is exorbitant.

I agree that in order to gain access to depth-of-book, the two major U.S. exchanges are pretty much the only places to access the data. What I'm curious about is a rebuttal from the exchanges that states their technology infrastructure costs for processing, normalizing, cleansing and storing this vast amount of information. Reg NMS put the burden of storing market depth for each exchange or ECN upon the market centers, not the brokers.

Brokers are well aware that having the necessary technology to avoid missing ticks can be worth millions of dollars on one wrong trade. Then there's the whole storage issue. We're talking terabytes of order book data per week! Are brokers really prepared to spend millions of dollars per year to process and store the data themselves? It also raises questions about the future intention of the major exchanges. Will they soon compete with the brokers they serve today? Do they possess the necessary technology pieces to become full-fledged market data providers? The next few years should be very interesting watching this drama unfold. Until then, Park Place and Boardwalk will continue to command the most rent for data.

Next-generation architecture for market data

The current market data architectures are based on a theory and approach that is no longer valid.  Current systems were conceptualized and implemented when message rates were orders of magnitude less than they are today.  The underlying architectural premise was that market data was moved to the applications and/or the analytics that required it.  This approach tacitly assumed that the increase in market data rates would approximate Moore's law, which if true would have meant that the increase in processing power and network bandwidth would have supported the architecture.  The facts are that market data rates are doubling every year, while Moore's law is delivering processing capability that doubles every 18 months.  On a compounded basis, market data rates over a five-year period between 2003 and 2008 have grown 32X while processing power has only grown 8X.  So the conclusion is not debatable.  The architectures must change.

There are two major considerations for the next-generation architecture.  One is the sheer mass of data and the other is the speed at which that data can be analyzed. 

Mass of data

With certainty we can project daily market data volumes of a terabyte per day in the near future and then growing well beyond that.  Applications will continue to want to analyze data to figure out what has happened in the past, whether that was a second, a minute, a day, week, month, or a year ago.  It is not possible to move the sheer mass of data to the application for analysis.  The analytics need to move to the data.

Speed of processing - programming models

Moore's law has not delivered the processing speed increases in a single processor that are required to support the explosion of market data volumes.  To deal with this failure, the processing complex has evolved to include numerous single processors that can operate in unison.  Multiprocessor servers are commonplace and now multi-core processors are becoming the standard in high-performance computing.  The major problem is that the vast majority of software has been written using compilers and languages that were designed when the available hardware was single processor and the predominant software architecture was appropriately single-threaded.  To take full advantage of multiple cores, software needs to be written from the ground up to be multi-threaded. This is very difficult because the languages and compilers are basically optimized for single-threaded use.  The software needs to be multi-threaded.

Speed of processing - hybrid hardware/software systems

The speed at which general purpose processors and software written for those platforms can perform certain operations is orders of magnitude slower than special purpose hardware tightly integrated with application software.  In order to deal with the explosion of market data, software-only solutions that run on general purpose processors will not work.  Field Programmable Gate Arrays (FPGAs) are one class of specialized hardware being used to perform tasks like compression, message parsing, and other jobs significantly faster than is possible with software-only solutions.

Next topic - gravitational pull of data sucking in analytics