Energy & commodities trading
Audit and fills data for trading surveillance
Dailyaudit and fills data, queryable in SQL
A daily pipeline that gives the trading surveillance team queryable audit and fills data from Trading Technologies.
Context
The trading surveillance team needed audit trails and fills from the Trading Technologies platform in a form they could query and retain.
The challenge
- Collect files delivered daily over SFTP, reliably.
- Transform vendor formats into a consistent schema.
- Make years of data queryable without running a database cluster.
What we did
- Built a scheduled collection and transformation pipeline.
- Stored the data on AWS S3 as Parquet, partitioned for the surveillance team’s queries.
- Exposed it through AWS Athena for SQL access.
Outcome
- Daily audit and fills data available to surveillance through standard SQL.
- Serverless storage and query, with no database cluster to operate.
Technologies
Next case study
Real-time pricing and analytics for an energy sales desk
Real-time pricing and analytics for an energy sales desk