In the 1980's and early 90's graphics were all the rage. Machines were offering GUI's in one form or another. Action games were being offered that had better quality images and far more realistic blood splatters. Digital photography and scanners were starting to sell. From the user point of view this was great.
Programmers had a different point of view, mainly one of stress. Drawing ever higher resolution pictures was taxing the hardware of the time to the limit. Machines that did 50 million instructions per second didn't seem very fast when trying to draw pictures with a million 3 byte pixels 15 or more times per second. High level languages frequently couldn't generate code that was efficient enough for the demands of the application. Hand written assembler that had to fit in a cache line was often required to make an application perform adequately. Programmers who thought that they had left the days of counting instructions behind (the 8088/80286 period) were back to doing it again on the 386 and 486 even though those processors were 10 to 100 times faster. Working on a big, price is no object, graphics machine was suddenly like working on a barely adequate chip designed for embedded processing. There just didn't seem to be enough speed or memory available to comfortably do the job.
Flash forward 20 years. Machines have gigs of RAM, multi-gigahertz clocks and many cores. Graphics is easy. No one worries too much about the application overwhelming the hardware anymore, unless they are involved with market data.
Any programmer working with market data in 2008 has a lot in common with graphics programmers of 20 years ago. Once again there is just way too much data for the hardware to deal with comfortably. Nowadays there are machines with 16+ cores, hundreds of gigs of RAM, 10 gigabit networks and SANs with fiber channel connections. That kind of hardware sounds very impressive until you look at the data that needs to be processed, which can peak at over a million ticks a second and requires sub-millisecond response times on analysis and queries. At those rates a program has just one microsecond, or less, to do real-time analysis, normalize and store it and do this while analytics are publishing updates to subscribing consumers and queries are pouring in for previously stored ticks. Suddenly the many cores and gigabytes of memory doesn't seem that impressive, in fact it seems barely adequate. Programming for a "finance" application suddenly looks more like programming an extreme real-time system. So once again worrying about instruction counts and memory constraints are an issue.
The universe, currently in the form of market data, maybe sending a message that no matter how fast your hardware gets there are always going to be uses for which it is barely adequate.
Comments