Today I attended a conference in New York called Accelerating Wall Street. It was an opportunity for technologists like myself to hear about how others in the financial industry are handling the ever growing volume and speed of order and information flow on the electronic markets. The main topics of discussion were:
- the true meaning of “ultra low latency” today, versus a couple years ago,
- efforts to utilize multiple cores,
- exploring complex event processing (CEP), and
- hardware acceleration.
As I expected, due to Wall Street’s highly competitive and secretive nature, the participants did not share many actual experiences or plans. Instead, most of the conversations were high level and forward looking. A number of intelligent and influential people were there though, so I did leave with some fresh ideas.
First, technology on Wall Street is changing extremely rapidly. You cannot build up a system to implement a trading strategy, and then sit back, light your cigar, and count the money as it rolls in. To succeed you must continually improve your system and look for new opportunities. You must be highly agile so opportunities don’t disappear while you build your solution. For example, two years ago the highest latency traders were concerned about measuring and saving milliseconds, and now they’re concerned about 100s of microseconds. Similarly, be cautious of anybody touting results or best practices from even one or two years ago — they are likely outdated and largely irrelevant.
As a result, only a few elite firms can compete in the latency race. To win the race, you must optimize all of your system components and all of their interactions. At the conference, several stories were tossed around about shops that focused on microseconds within a system that included a hop that takes 10s of milliseconds. In a sense, this realization questions the relevance of the conference’s topic.
Recently I have researched some of the major complex event systems. I have a hard time differentiating the various offerings or identifying the best product. This experience was echoed by others at the conference. Further, all of the participants who spoke directly about their experience using CEP said that they used Esper, which is open source. Another common observation, which I share, was that CEP products need to offer tighter integration with messaging bus products.
Finally, I didn’t hear any lucid comments about multi-core processors or grid computing. Instead of taking the fresh development perspective that’s needed to utilize parallel processing, Wall Street seems to be stuck in the mode of optimizing their legacy techniques and products. For example, some attendees discussed using hardware acceleration for Java components. The general mindset is still, “well, I code it up using threads in Java, and if that’s too slow I either recode it in C++ or accelerate it with special hardware.” The firms that manage to embrace more modern techniques for developing concurrent software systems will be the ultimate winners.
Edit: here is a post from one of the panelists at the conference.
