|
|
|
|
|
|
 |
Case Study: Realtime Information Data LoggingActivity Summary Our client (a researcher from a leading European institute of economics) was doing a research on realtime market data. Delera Systems developed a .NET-based solutions to communicate with a commercial realtime data server, parse and clean up incoming transaction data and place it in a MS SQL Server database for subsequent analysis. Problem Processing realtime market data is a complex topic. Using a high-bandwidth connection, a server can be made to listen to this data and process or display it. There are certain times during the daily cycle - such as when certain markets open - when thousands of separate transactions are transmitted in a few seconds. Due to the nature of the research performed, heavy computations had to be done on continuously streaming data. The exact timing of market events was a key factor, so a database buffer between the data server and the processing engine was needed. This software buffer had to operate flawlessly in a 24x7 mode, while allowing for the inevitable brief periods of database inavailability. The software had to cope with an evolving protocol, and to take care of daily database recycling, so as to keep data volumes manageable. A key requirement was the completely automated operation. The software and database would run in a 24x7 environment and no data loss was allowed, as this would have adverse effects on analysis results. The system was to take care of inevitable disconnections, connection resumes, and keep a full log and performance statistics for review at a later time. Solution Delera Systems built a custom Windows-based front end application using the Microsoft .NET Framework and the C# and Managed C++ languages. The application would interface with an operator, allowing them to control the process of data collection, formatting and retrieval. At the same time, the application would use a complex pipelined architecture to provide a RAM-resident buffer, thus capturing 100% of the incoming protocol data while also allowing for brief database downtime periods, and for database recycling. The application proved to be more than capable of handling large volumes of data. It demonstrated great resource efficiency when working with a sustained data throughput, while also coping with enormous peak load levels when those occurred - certainly more data than SQL Server would be able to handle out of the box on modern midrange hardware. Service Offerings The following Delera Systems service offerings were applied to this project:
Technologies
|
|
|
|
