Context
At the University of Toronto Institute for Aerospace Studies, a research group led by Prof. Craig Steeves was building a Next.js web platform for drone simulation. Our principal joined as the data engineer — because a simulation platform is only as fast as its slowest query.
The problem
Simulation output is awkward data: large, semi-structured, and schema-drifting as the research evolves. Dumped naively into tables, it makes every dashboard query a table scan and every schema change a migration crisis.
What we built
A normalized relational model designed from the platform’s actual access patterns — entities, relations, and types mapped before a single table was created; then an indexing and caching strategy, with views encapsulating the platform’s recurring analytical queries.
Ingestion runs through dynamic Python ETL scripts that tolerate schema evolution: new simulation parameters flow through transformation into the model without hand-editing pipelines each time the researchers iterate.
Key decisions
Dynamic schema handling in the ETL layer — accepting a little ingestion complexity to keep the relational core clean — is what let the platform speed up while the research kept changing underneath it.
Outcome
Faster queries, calmer migrations, and a pipeline the research team could keep feeding — plus the origin of JES’s conviction that physical-AI platforms are data platforms first.