A working catalogue of the access control SQE and Spark share through Apache Ranger: object grants at the catalog, five column masks, row filters, tags that span tables, four personas built from the same primitives, and the two places the engines still render differently. Everything here is a line from a run that finished 43 of 43, on an EU bank fixture rather than a toy table. Includes the reason there are two enforcement points rather than one, and what changed when we moved every privilege write onto an authenticated endpoint.
A complete, executable account of how SQE and Spark query the same Iceberg table under the same Apache Ranger policies: Polaris object grants, row filters, column masks, tags, precedence, writes, views, and the differences that remain.
Spark was connecting to our catalog as root, which meant every access-control policy we had written applied to exactly one engine. Fixing it took no engine code: Polaris already authorizes against the OIDC identity, so a per-user token was the whole change. Then we ran both enforcement tiers together for the first time and found they compose in an order nobody documents, with Spark's plugin answering before the catalog is asked. Along the way two of our own new tests passed for the wrong reason, the two engines turned out to disagree about which mask wins, and renaming a column silently removes its protection. Here is the full evaluation order, what we measured, and why Databricks and Snowflake do not have this class of problem.
Access control was the last subsystem whose end-to-end behaviour we checked by grepping CLI output. The denial test matched the string 'not found', which is also what a typo'd table name prints. We replaced that harness with twenty assertions on decoded Arrow values against a live Apache Ranger, and the exercise found a feature that had never once worked: every tag-based column mask had been restricting the column instead of masking it since the day we shipped it. Two of the new tests then passed for the wrong reason, and only mutation caught them. Here is what fine-grained access control in SQE actually does, and how we know.
SQE's founding rule is no service account: every query runs as the authenticated user, tokens passed through to Polaris and S3. Background compaction has no user to be. Here is how we gave Iceberg tables a self-healing maintenance path while keeping that rule intact: a structurally isolated principal, deny-by-default grants, and a commit authority that never leaves the coordinator, even when the rewrite itself is distributed across the worker fleet.
A Python ingestion job should not have to choose between an open Iceberg catalog and a governed SQL engine forever. We connected dlt directly to Apache Polaris through Iceberg REST, then sent the same full, incremental, and SCD2 loads through SQE over Trino HTTP and Arrow Flight SQL. The result is one service-principal identity, one catalog, three front doors, and a test suite large enough to expose the bugs hiding at batch boundaries.
Tag-based masking is two systems wearing one name: a rule that says what a tag means, and an association that says which columns carry it. We put the rule in Apache Ranger, where Spark can share it, and the association in the Iceberg table metadata, where it survives rename, replication, and federated catalogs that no policy store can see. Here is the split, the precedence contract, and the identity bug the tests now pin down.
Reads on SQE spill to disk when they run out of memory. Writes did not. A large CTAS, a wide MERGE, or an oversized client upload could balloon a coordinator buffer past its memory limit and get the process OOM-killed, which takes every other query on that node with it. We gave the write path the same memory discipline the read path already had: pool-track the buffers that must exist so an oversized write fails as one typed error, and stream the ones that never needed to buffer at all.
SQE got a second access-control backend: write GRANT/REVOKE as Apache Ranger policy and let Polaris enforce it. The protocol took an afternoon. The identity model took the week, because Polaris federation does not work the way the token suggests.
SQE's fine-grained Ranger backend does row filters, column masks, role-conditional masking, and tags. The kicker: the same Ranger policy on the same Polaris catalog produces byte-exact identical masked output in SQE and in standard Apache Spark. We proved it live, and the first run failed for a reason no amount of reading could have predicted.
Snowflake gives you masking policies, row access policies, object tags, and a GRANT model. SQE gives you the same primitives on Apache Ranger and open Iceberg, enforced by plan rewrite and shared across engines. Here is the mapping, the one real edge SQE has, and the gaps we have not closed yet.
DataFusion 54 landed and we bumped to it. The port was mostly mechanical, with one real behavioral change in the shuffle hasher. The interesting part is what the release notes implied and the engine did not deliver: LATERAL joins are logical-plan only, array lambdas still fail, and the one compatibility win we found was a documentation bug, not a DataFusion feature. We tested every claim before we wrote it down.
A two-table TPC-H join that Trino ran in 2.2s took us 161s at SF10. We blamed partition layout, then single-node joins, then a subquery pattern. All three were wrong. A CPU profile and two timers found the truth: a runtime filter we pushed to the probe scan was getting re-snapshotted once per batch, and each snapshot rebuilt a 300,000-node expression tree. The fix snapshots it once. q12 went 161s to 2.7s, q17 176s to 7.1s, q10 from a 300s failure to 3.3s, with the result rows unchanged and no knob touched.
On SSB at SF1 Trino ran scan-heavy queries about 2x faster than SQE, even though we pruned just as well. A new line in our query profile found it: a 151MB lineorder file decoded on a single thread, 94% of a 969ms query spent waiting on one scan. The obvious fix (more partitions) was a trap that regressed q72 from 17s to 100s once before. The safe fix parallelizes decode inside the scan without changing the plan the optimizer sees.
Our own README promised fine-grained Lake Formation in the SQE Glue quickstart. The engine does not do it. Here is what Lake Formation actually enforces when SQE reads S3 directly, and where SQE's real column and row masking lives.
DuckDB 1.5 ships a wire protocol called Quack. We re-implemented it in pure Rust, turned SQE into both server and client, and proved you can JOIN an Iceberg table with a remote DuckDB table in a single SELECT. Two waves of bugs, one federated query.
MR #220 wired runtime filters into iceberg-rust's scan path and dropped TPC-DS 67%. SSB looked like it regressed 6%. Two failed heuristic attempts, one parquet-trace session, and ten warm passes later, the regression turned out to be measurement noise. The fix-the-fix that wasn't, and the data-clustering insight that explains why two suites with the same code path behave nothing alike.
TPC-DS q72 sat at 10 seconds while every other query ran in under 1.4. Five days of investigation chased scan parallelism, range-based NDV, iceberg-rust upgrades, and the RisingWave fork. None of those were the bug. The bug was a silently-skipped Err arm in our dynamic-filter evaluator that swallowed every Int32 vs Int64 type clash. Fixing it: 15.5s to 0.77s. q72 now beats Trino.
SQE now ships DuckDB-style ATTACH / DETACH and CREATE / DROP SECRET. The story of building it covers parser extension, credential hygiene, a lifecycle bug we found in the integration tests, and a state-store pattern that is starting to repeat.
SQE now emits OL 2-0-2 events with column-level lineage on every write. Here is what they look like, why we walked the LogicalPlan to build them, and the disk spool we did not want to write.
SQE started as a distributed Iceberg query engine. Five MRs later it queries CSVs from disk, Parquet from S3, and Parquet from HuggingFace. We did not plan that.
We claimed the engine was catalog-agnostic. Time to prove it. One branch, five live integration tests, one small AWS SigV4 patch, and a matrix score that moved from 153 to 158.
Sixty-three capabilities, three levels, no marketing. The Iceberg Matrix is what compatibility looks like when the rubric is public and the evidence has to land in code. Here is why it works, what we learned from sitting on it, and why every open standard needs one.
We thought the V3 path worked. The unit tests said it worked. The matrix called it 'partial' and we agreed. Then we wrote eleven end-to-end tests and discovered Polaris had been silently rejecting every V3 column type for months.
An admin page that generates TPC benchmark datasets on-demand. Pick a benchmark, pick a scale, click Generate — sqe-bench does the heavy lifting in a Docker container while SQE loads the results into Iceberg tables under your identity.