A working catalogue of the access control SQE and Spark share through Apache Ranger: object grants at the catalog, five column masks, row filters, tags that span tables, four personas built from the same primitives, and the two places the engines still render differently. Everything here is a line from a run that finished 43 of 43, on an EU bank fixture rather than a toy table. Includes the reason there are two enforcement points rather than one, and what changed when we moved every privilege write onto an authenticated endpoint.
A complete, executable account of how SQE and Spark query the same Iceberg table under the same Apache Ranger policies: Polaris object grants, row filters, column masks, tags, precedence, writes, views, and the differences that remain.
A Python ingestion job should not have to choose between an open Iceberg catalog and a governed SQL engine forever. We connected dlt directly to Apache Polaris through Iceberg REST, then sent the same full, incremental, and SCD2 loads through SQE over Trino HTTP and Arrow Flight SQL. The result is one service-principal identity, one catalog, three front doors, and a test suite large enough to expose the bugs hiding at batch boundaries.
SQE got a second access-control backend: write GRANT/REVOKE as Apache Ranger policy and let Polaris enforce it. The protocol took an afternoon. The identity model took the week, because Polaris federation does not work the way the token suggests.
We thought the V3 path worked. The unit tests said it worked. The matrix called it 'partial' and we agreed. Then we wrote eleven end-to-end tests and discovered Polaris had been silently rejecting every V3 column type for months.