#datafusion

3 posts · all posts

Related: performancetrinorusttpc-dsquery-optimizationicebergopen-sourcearchitecture
· Jacob Verhoeks

Our Nemesis: TPC-DS Query 72 and the Limits of a Custom SQL Engine

One query. Ten tables. Twelve times slower than Trino. Everything we tried, what worked, what didn't, and where the ceiling is.

performancetpc-dsdatafusiontrino
· Jacob Verhoeks

DataFusion 53, a Vendored Fork, and 40% Faster Queries

We upgraded SQE from DataFusion 52 to 53 by forking and rebasing iceberg-rust ourselves. The result: 27-40% faster across every benchmark suite.

datafusionicebergperformancerust
· Jacob Verhoeks

We Replaced Our Trino Fork with a Rust SQL Engine

How we went from maintaining a 2M-line Java fork to shipping a 50MB binary that runs every query as the authenticated user.

rustdatafusiontrinoarchitecture