Energy & commodities trading
Collecting 120 million market data records a day
120M+records collected per day
A high-volume collection system for order books and trades that gives algo traders the history they need to design and backtest strategies.
Context
The algo trading team needed complete historical order book and trade data from European energy markets to design and backtest strategies. The data was available through the Trayport Data Analytics REST API, but it had to be collected at volume, stored in several systems and kept complete.
The challenge
- Sustain more than 120 million records a day through a REST API.
- Write the same data to an analytical tick store, a time-series database and a data lake.
- Detect and recover gaps without duplicating records.
- Run and upgrade the supporting infrastructure without manual steps.
What we did
- Built the collectors in ASP.NET Core 8, with Hangfire for scheduling, retries and backfills.
- Ingested into Kx Insights Enterprise through an EMQX MQTT cluster, into TimescaleDB, and into AWS S3 as Parquet.
- Deployed the EMQX cluster on AKS, fully automated with Azure DevOps.
- Wrote CI/CD pipelines for Kx Insights Enterprise installs, upgrades and licence renewals.
Outcome
- More than 120 million order book and trade records collected every day.
- One consistent dataset for strategy research and backtesting.
- Pipeline-driven operations for the brokers and databases.
Technologies
Next case study
Ultra-low-latency tick capture with the Disruptor pattern
Ultra-low-latency tick capture with the Disruptor pattern