What the Data Platform Actually Does
The map of the platform: an assembly of open-source parts behind a Backend-for-Frontend that holds your token and enforces your identity at every hop. What each surface does, how a request flows, and where the cliffs are.
Most “data platforms” are one vendor’s product with a marketing page. This one is an assembly. Every piece is open source — Keycloak, Apache Polaris, Trino (or our own Rust engine), Iceberg, OPA, an S3-compatible store, dbt, Langflow — and the platform is the glue that makes them behave like one system instead of nine.
The glue has a name and a shape: a Backend-for-Frontend. A single FastAPI service that the browser talks to, which then talks to everything else on your behalf, holding your token and enforcing your identity at every hop. You never hand a Polaris credential to a React app. You never let a query engine see anything but the JWT you already minted. The BFF is the seam, and most of what’s interesting lives in that seam.
This post is the map. What each surface does, how a request flows, and — because half the value of a map is knowing where the cliffs are — what’s solid and what’s still rough.
One process, three modes
The whole backend is one binary that boots into one of three modes, chosen by EXECUTION_MODE.
- bootstrap — the one-shot initializer. It creates the Keycloak realm and OIDC clients, bootstraps Polaris, creates the
main_warehousecatalog and theanalytics_dbdatabase, sets up the principal roles (service_admin,catalog_admin,data_writer,datareader), creates the seed users, and then validates that a token minted by Keycloak is actually accepted end to end. It runs once and exits. - backend — the API server. The thing that’s up all the time.
- cli — defined in the enum, referenced in URL selection, and otherwise not there. The executor doesn’t exist. If you reach for CLI mode today, you’ll find a stub.
In the Docker quickstarts, bootstrap is usually done by sidecar init containers — keycloak-config-cli for the realm, the Polaris admin JAR for the catalog — rather than the Python bootstrap path. The Python path exists for programmatic and Kubernetes deployments. Two ways to do the same thing, which is worth knowing before you go looking for where a realm got created.
Authentication: the token never touches the browser
Identity is OIDC through Keycloak, in a strict BFF pattern. The browser holds one httpOnly cookie. The actual tokens live server-side, in Postgres, encrypted with a key derived from SESSION_SECRET_KEY — and that key is checked at startup for length and for not being a placeholder, because a weak session key is the kind of thing that’s fine in dev and catastrophic in production.
Login is the standard authorization-code flow. There’s a direct-login (ROPC) path for CLI and tests. There’s silent refresh when the access token nears expiry. And there’s /auth/token, which hands the frontend the raw JWT so it can be passed through to the query engine and the catalog — the mechanism that makes per-user authorization possible all the way down.
Service-to-service calls use the client-credentials grant. The ontology subsystem, for instance, talks to the query engine under its own service identity when no caller token is present. Keycloak isn’t load-bearing in the sense of being special — OIDC_DISCOVERY_URL lets you point at Okta or Entra without touching code. It’s the default, not a dependency.
There are two dev-only escape hatches:
CHAMELEON_ALLOW_UNVERIFIED_JWT=1andDEV_JWT_SIGNING_KEY, for running without a live IdP. They’re off by default. If you ever see them on in a production config, that’s the finding.
The catalog, proxied
Tables live in Iceberg, catalogued by Apache Polaris, which speaks the Iceberg REST API. The platform doesn’t replace Polaris — it proxies it. /api/platform/v1/metadata/... forwards your JWT to Polaris, which validates it and asks OPA whether you’re allowed. Browse catalogs, namespaces, tables, views; read a schema; preview rows (that last one runs a real query through the engine).
Catalog, namespace, and table names are validated against ^[A-Za-z0-9_]+$ before they’re ever interpolated into SQL. Small thing. It’s the difference between a metadata browser and a SQL injection surface.
Query execution, three engines
A query comes in at /api/platform/v1/query. The JWT comes off the session. The SQL is validated — non-admins get an allowlist of statement prefixes (SELECT, SHOW, DESCRIBE) and a blocklist of write keywords; service_admin and catalog_admin bypass the filter. Then it’s dispatched to one of three engines, and the caller’s token goes with it so the engine authorizes as you:
- Trino / SQE — the primary path, over HTTP. SQE is our own query engine, written in Rust, exposing a Trino-compatible HTTP endpoint and Arrow Flight SQL, speaking the Polaris Iceberg REST API, authorizing through OPA. Same wire protocol as Trino, different engine.
- Arrow Flight SQL — for when you want columnar results without JSON serialization in the middle. The client hand-encodes the Flight SQL protobuf to dodge a native-library dependency.
- Spark Connect — gRPC to Spark 4.1. Off by default, behind a compose profile, for the scan-heavy work.
Results come back as JSON, and the query is written to a history table with metrics — rows, duration, peak memory, spill — kept for 30 days.
Authorization: table-grain, two generators
This is the part to understand precisely, because it’s powerful and it has a hard limit.
Authorization is done by OPA, with Rego policies generated from the platform’s own data. There are two generators because there are two auth modes. In principal mode you manage Polaris-native principals, principal roles, catalog roles, and grants. In grant mode you manage a simpler /access surface and the platform merges grants into compact Rego. Either way, the generated policy is pushed to OPA, and Polaris is configured to ask OPA for every decision.
The limit: authorization is table-grain. Catalog, namespace, table, privilege, allow or deny. There is no row-level filtering and no column masking anywhere in the auth stack. Grep the codebase for row_filter or column_mask and you’ll find them only in unrelated places. If you need “analysts see their own region’s rows” or “mask the SSN column,” that’s not here. It’s the single biggest functional gap versus the commercial warehouses, and it’s worth saying plainly rather than discovering in a security review.
Everything else the BFF carries
The seam is wide. A quick inventory of what else runs through it:
| Surface | What it does | State |
|---|---|---|
| Admin | Catalogs, users, principals, roles, grants, warehouses, storage — CRUD across Keycloak + Polaris + S3 + the engine, orchestrated so a “create catalog” wires all of them | Solid |
| Audit | Every API action recorded by middleware, 90-day retention, readable by admins | Solid |
| Security detections | A loop every 5 minutes runs rules (mass-delete burst, MCP rate abuse) against the audit log, writes alerts | Working, two rules |
| OCSF forwarding | Audit events forwarded to Splunk HEC, Azure Sentinel, or a webhook, as typed OCSF events | Working |
| text2sql | NL → SQL, OpenAI when a key is set, otherwise local Ollama. Caches schema DDL, keeps conversation state, can be grounded with the published ontology | Working |
| Lineage | OpenLineage 2.0 ingest (Marquez-compatible), dbt manifest upload, a bipartite dataset⇄job graph, source trust profiles | Working, role-gated |
| Data contracts | ODCS v3 contracts, quality checks compiled to SQL and run on the engine, a scheduler | Working core, several pieces half-wired — see the deep dive |
| dbt | A full browser IDE: git-backed projects, container runs, manifest lineage, quick-test — see the deep dive | Substantial |
| Ontology | A virtual semantic layer over Iceberg, queryable by agents over MCP — see the deep dive | Substantial |
| Insights | An AI canvas: ask a question, an agent answers with charts | In progress |
| MCP servers | Two — a platform server (catalog, query, contracts, schedules) and the ontology server (per-entity tools) | Working |
Two background schedulers run the recurring work: one for data-quality contracts (every 60s), one unified scheduler (every 30s) for Langflow flows, dbt runs, quality runs, and Iceberg maintenance. Plus the cleanup, detection, digest, drift, and forwarder loops. A lot happens on timers.
The frontend, in one breath
The React portal is the human face of all of it. Query editor with text2sql. A catalog explorer where every table has tabs for schema, sample rows, quality, lineage, contract, permissions, and audit. A multi-source lineage graph. The ontology designer and knowledge-graph browser. The dbt IDE. The contracts editor. And a govern/admin section gated by role — catalogs, namespaces, policies or access control (the UI hides whichever auth mode you’re not in), schedules, audit, security alerts, system health.
It’s a portal, not a dashboard. The distinction matters: it’s where you operate the platform, not just look at it.
The topology, and why it’s the whole point
Here’s the service list, and the column that matters most is the last one.
| Service | Role | What it replaces |
|---|---|---|
| PostgreSQL | Persistence for everything | Standard OSS |
| Keycloak | OIDC identity | Auth0 / Okta / Cognito |
| Apache Polaris | Iceberg REST catalog | Glue / Unity Catalog / Hive Metastore |
| OPA | Policy decisions | Cloud-native IAM |
| RustFS / MinIO | S3-compatible object store | AWS S3 / ADLS / GCS |
| SQE (or Trino) | Query engine | Athena / BigQuery / Trino |
| Spark | Heavy compute (optional) | Managed Spark |
| Ollama | Local LLM inference | OpenAI API |
| Langflow | Visual AI flows | — |
| OpenObserve | Traces + metrics + logs | Jaeger + Prometheus + Loki + Grafana |
Every one of those is open source and self-hostable. The compose files set ZO_TELEMETRY: false and DO_NOT_TRACK: true and configure no outbound egress. The only optional cloud dependency in the entire stack is OpenAI, and Ollama exists precisely so you can cut it. Nothing in this platform has to phone home, and nothing has to run in someone else’s account. That’s not a feature bolted on the side. That’s the reason the thing is an assembly of open components instead of a wrapper around a vendor API. The argument for why that matters is its own post.
Where the cliffs are
In the spirit of telling the whole story:
- No row/column security. Table-grain only.
- CLI mode is a stub. The enum lies.
- Postgres is a single point of failure. The quickstarts run one node with hourly
pg_dumpall. No HA out of the box. Fine for a demo, a decision for production. - The
/connectionsAPI is git-shaped. It’s positioned as general but the model lives in the dbt module and only dbt consumes it. The path made a promise the code hasn’t kept yet. - Several subsystems are partially wired — the insights canvas, contract profiling, contract Polaris-sync. Built, tested in isolation, not yet plugged into the live path. The deep-dive posts name each one.
None of that makes the platform less interesting. It makes it honest. You’re looking at a system that assembled the entire modern data stack from parts you can read, run, and own — and that’s far enough along to use, with edges sharp enough that you should know where they are before you lean on them.