Spark was connecting to our catalog as root, which meant every access-control policy we had written applied to exactly one engine. Fixing it took no engine code: Polaris already authorizes against the OIDC identity, so a per-user token was the whole change. Then we ran both enforcement tiers together for the first time and found they compose in an order nobody documents, with Spark's plugin answering before the catalog is asked. Along the way two of our own new tests passed for the wrong reason, the two engines turned out to disagree about which mask wins, and renaming a column silently removes its protection. Here is the full evaluation order, what we measured, and why Databricks and Snowflake do not have this class of problem.
Tag-based masking is two systems wearing one name: a rule that says what a tag means, and an association that says which columns carry it. We put the rule in Apache Ranger, where Spark can share it, and the association in the Iceberg table metadata, where it survives rename, replication, and federated catalogs that no policy store can see. Here is the split, the precedence contract, and the identity bug the tests now pin down.
A lakehouse rarely has one query engine. Spark writes the tables, SQE serves the interactive queries, and both touch the same Iceberg data. The usual failure is policy drift: each engine has its own access-control plugin, the same masking intent gets translated twice, and the two translations disagree. We avoided that by not translating twice. SQE and Spark read the same Apache Ranger hive service, so one policy written once enforces the same way in both. The SSN reads xxx-xx-1111 whichever engine ran the query.
Multi-tenancy on a data platform is usually a ticket and a week of console clicking. We made it a Terraform apply: a workspace, its catalog, its groups, its grants, and its dbt project — isolated by construction, reviewed in a pull request.
Data quality belongs in the table's contract, not scattered across dbt tests and cron jobs. We build on the Open Data Contract Standard, compile quality rules to portable SQL, and run them on the live engine under your identity — plus an honest accounting of what's wired and what isn't.
A Terraform provider for the platform's own resources — warehouses, namespaces, grants, dbt projects, contracts, ontologies — managed in HCL, in a repo, in a pull request, with a plan you can read before you apply.