SQE now reads Hive-style external tables over CSV, line-delimited JSON, and partitioned Parquet on any S3-compatible store, using Athena's own CREATE EXTERNAL TABLE syntax. There is no Hive Metastore, no Glue API call, and no catalog service of any kind behind the table: the definition is a JSON manifest next to the data, shaped like a Glue TableInput. This post is what emulating Athena and Glue actually requires, which parts we copied deliberately, the four Athena behaviours we refused to copy, and the credential difference that is a real change from how SQE treats Iceberg.
SSB at SF10 was the last benchmark where Trino beat us, and the gap was one serial scan feeding eight idle cores. Closing it took three fixes: parallelize the probe scan, assign byte-range splits instead of whole files, and stop trusting a flag that had never actually fired. Along the way the row counts started lying. Three TPC-DS queries returned exactly eight times their LIMIT, and the mechanism is a lesson in what parallelism does to a plan that was optimized while it was still serial.
Reads on SQE spill to disk when they run out of memory. Writes did not. A large CTAS, a wide MERGE, or an oversized client upload could balloon a coordinator buffer past its memory limit and get the process OOM-killed, which takes every other query on that node with it. We gave the write path the same memory discipline the read path already had: pool-track the buffers that must exist so an oversized write fails as one typed error, and stream the ones that never needed to buffer at all.
SSB is the one benchmark suite SQE loses to Trino, 2.5x at scale factor 10, while it wins everything else. The obvious cause was a single-threaded fact-table scan, so we built a planner rule to parallelize it. A correctness smoke caught the rule returning 240 million rows from a 60-million-row table, a latent 4x duplication bug we then fixed. Then the parallel scan turned out to change nothing: SSB did not move. The real cause is a dynamic filter that is a huge win on clustered fact tables and pure overhead on SSB's uniformly-distributed one. A slow number is a hypothesis, and the fix you are sure of can be perf-neutral.
DataFusion 54 landed and we bumped to it. The port was mostly mechanical, with one real behavioral change in the shuffle hasher. The interesting part is what the release notes implied and the engine did not deliver: LATERAL joins are logical-plan only, array lambdas still fail, and the one compatibility win we found was a documentation bug, not a DataFusion feature. We tested every claim before we wrote it down.
A two-table TPC-H join that Trino ran in 2.2s took us 161s at SF10. We blamed partition layout, then single-node joins, then a subquery pattern. All three were wrong. A CPU profile and two timers found the truth: a runtime filter we pushed to the probe scan was getting re-snapshotted once per batch, and each snapshot rebuilt a 300,000-node expression tree. The fix snapshots it once. q12 went 161s to 2.7s, q17 176s to 7.1s, q10 from a 300s failure to 3.3s, with the result rows unchanged and no knob touched.
TPC-DS q95 was our worst query: 18 seconds against Trino's 1.3, a 14x loss that justified building a whole optimizer feature. Before we wrote a line of it, we pulled the plan and the profile. The 12-million-row self-join the feature was meant to shrink did not exist, the engine ran the query in under half a second, and the 18 seconds lived only in a benchmark harness running both engines on one starved host. On a clean rig SQE runs q95 in 240ms and beats Trino 12x. A slow benchmark number is a hypothesis until you reproduce it in isolation.
On SSB at SF1 Trino ran scan-heavy queries about 2x faster than SQE, even though we pruned just as well. A new line in our query profile found it: a 151MB lineorder file decoded on a single thread, 94% of a 969ms query spent waiting on one scan. The obvious fix (more partitions) was a trap that regressed q72 from 17s to 100s once before. The safe fix parallelizes decode inside the scan without changing the plan the optimizer sees.
Our SF0.1 compare run looked great: zero mismatches across seven suites. Then we asked DuckDB to check the data and found that 16 'passing' TPC-DS queries had never selected a single row, TPC-C had zero warehouses, and the one real disagreement between SQE and Trino was Trino's fault. Plus: the dynamic filter that shipped 6 million rows because nobody carried it across a node swap.
DuckDB 1.5 ships a wire protocol called Quack. We re-implemented it in pure Rust, turned SQE into both server and client, and proved you can JOIN an Iceberg table with a remote DuckDB table in a single SELECT. Two waves of bugs, one federated query.
MR #220 wired runtime filters into iceberg-rust's scan path and dropped TPC-DS 67%. SSB looked like it regressed 6%. Two failed heuristic attempts, one parquet-trace session, and ten warm passes later, the regression turned out to be measurement noise. The fix-the-fix that wasn't, and the data-clustering insight that explains why two suites with the same code path behave nothing alike.
TPC-DS q72 sat at 10 seconds while every other query ran in under 1.4. Five days of investigation chased scan parallelism, range-based NDV, iceberg-rust upgrades, and the RisingWave fork. None of those were the bug. The bug was a silently-skipped Err arm in our dynamic-filter evaluator that swallowed every Int32 vs Int64 type clash. Fixing it: 15.5s to 0.77s. q72 now beats Trino.
Modern object-store abstractions unify the filesystem and HTTP behind a single URL. That's a feature for ergonomics. It's a security trap when the URL comes from a user. SQE shipped that trap and then closed it. The IMDS pivot is the part worth telling.
SQE now ships DuckDB-style ATTACH / DETACH and CREATE / DROP SECRET. The story of building it covers parser extension, credential hygiene, a lifecycle bug we found in the integration tests, and a state-store pattern that is starting to repeat.
SQE now emits OL 2-0-2 events with column-level lineage on every write. Here is what they look like, why we walked the LogicalPlan to build them, and the disk spool we did not want to write.
SQE started as a distributed Iceberg query engine. Five MRs later it queries CSVs from disk, Parquet from S3, and Parquet from HuggingFace. We did not plan that.
We built SQE for distributed Iceberg, but most of the time you just want to look at a parquet file. Here's how we made the engine work both ways without forking the codebase.