Setting It All Up With Terraform

A data platform you can’t manage as code isn’t a platform — it’s a collection of consoles. So we built a Terraform provider for it. Not for the infrastructure underneath; that’s a separate concern. A provider for the platform’s own resources: warehouses, namespaces, tables, users, principals, grants, dbt projects, contracts, ontologies. The things you’d otherwise click through six admin screens to create. In HCL, in a repo, in a pull request, with a plan you can read before you apply.

The provider lives at infrastructure/data-platform-terraform-provider. It’s real — full Go source for every resource and data source, generated docs, acceptance tests, a tagged release. This post is how it works and how you actually use it, grounded in the examples that ship with it.

One important constraint, up front

The provider talks to one thing: the platform’s BFF REST API, at /api/platform/v1/.... It does not touch Polaris, Keycloak, or S3 directly. Everything it does, it does through the same API the web portal uses, with the same identity and the same authorization.

That has a consequence worth stating plainly: the provider needs a running platform. It manages content, not infrastructure. Standing the platform up — the Kubernetes cluster, then Keycloak, Polaris, OPA, the object store, the engine, the backend — is done elsewhere (a Helm/operator deploy in production, or the docker-compose quickstart locally). Once that’s running and bootstrapped, the provider takes over and manages everything inside it. Infrastructure, then services, then this. The provider is the third layer, and it’s the one you’ll touch every day.

Configuring the provider

Three ways to authenticate, in priority order. The real example file lays them out:

terraform {
  required_providers {
    chameleon = {
      source  = "registry.terraform.io/schubergphilis/chameleon"
      version = "~> 0.1"
    }
  }
}

# Client credentials — recommended for service accounts / CI
provider "chameleon" {
  endpoint      = "https://chameleon.example.com"
  oidc_endpoint = "https://keycloak.example.com"
  client_id     = var.chameleon_client_id
  client_secret = var.chameleon_client_secret
}

For a static bearer token, set token and drop the OIDC fields. For local dev, ROPC works — username, password, and ssl_verify = false against chameleon.test. The client caches the token and refreshes it 60 seconds before expiry, so a long apply doesn’t die mid-run on an expired token.

Client credentials is the one to use in CI. A service account in Keycloak, a client secret in your secret store, and the provider authenticates as a machine identity that you can grant exactly the platform roles it needs.

What you can manage

The resource surface covers the whole platform:

Plus data sources for nearly all of it, and one ephemeral chameleon_query that runs a SELECT and returns rows without ever writing state — for reading a value you need to wire into another resource, not for persisting anything.

A real apply, start to finish

The demo example is the one to read first, because it’s a complete, working setup and its comments are honest about every edge. Here’s the spine.

A bucket, then a warehouse pinned to it. S3 names can’t have underscores, so the bucket is tf- prefixed; everything else is tf_. The warehouse pins its base location to that bucket so Polaris doesn’t reject the create with a location overlap against an existing main_warehouse:

resource "chameleon_storage_bucket" "demo" {
  name = "tf-iceberg-warehouse"
}

resource "chameleon_warehouse" "main" {
  name = var.warehouse_name
  storage_config = {
    default_base_location = "s3://${chameleon_storage_bucket.demo.name}/"
    allowed_locations     = ["s3://${chameleon_storage_bucket.demo.name}/"]
  }
}

Users, and the role distinction that matters. Only the admin user gets realm roles — service_admin, catalog_admin — which is what grants Polaris-level admin. Writer and reader are plain Keycloak users whose data access is expressed entirely as grants, not roles:

resource "chameleon_user" "demo" {
  for_each   = local.demo_users
  username   = each.key
  email      = each.value.email
  roles      = each.value.roles   # admin: [service_admin, catalog_admin]; others: []
  password   = each.value.password
}

Principals mirror the users. A Polaris principal per Keycloak user, with an explicit depends_on so the principal isn’t created before the user exists.

Grants, SQL-style. No catalog roles, no groups in the simple path — each user gets the verbs they need, on the catalog and on the namespace, as direct grants. The example builds them with a flatten over a privilege map, which is the idiom you’ll reach for constantly:

resource "chameleon_grant" "namespace_access" {
  for_each     = { for g in local.namespace_grants : g.key => g }
  privilege    = each.value.privilege          # USE / SELECT / MODIFY / CREATE / ALL
  catalog      = chameleon_warehouse.main.name
  namespace    = chameleon_namespace.demo.name
  grantee_type = "USER"
  grantee_name = chameleon_principal.demo[each.value.user].name
  effect       = "allow"
}

Note the grantee references the principal resource, not the username string. The name is identical, but the reference is what tells Terraform’s dependency graph “the principal must exist before this grant.” Small detail, correct dependency ordering.

Group-level rules, including a deny. The example also seeds OPA-layer grants for groups — and shows a namespace-scoped deny, the platform’s way of saying “analysts can read everything except PII”:

deny_pii_to_analysts = {
  privilege    = "SELECT"
  namespace    = "pii_data"
  table        = null            # applies to every table under pii_data
  grantee_type = "GROUP"
  grantee_name = "SG-DataAnalysts"
  effect       = "deny"
}

A table, declared by columns:

resource "chameleon_table" "demo_t" {
  catalog   = chameleon_warehouse.main.name
  namespace = chameleon_namespace.demo.name
  name      = "tf_demo_user_summary"
  columns = [
    { name = "id",          type = "long",        required = true },
    { name = "email",       type = "string",      required = true },
    { name = "signup_at",   type = "timestamptz", required = true },
    { name = "total_spend", type = "double",      required = false },
  ]
}

And the data-engineering path: a git connection holding a PAT, and a dbt project that references it and targets the warehouse and namespace above, so dbt runs land right beside the declaratively-provisioned schema.

The fullstack example goes further — five composed modules (storage, catalog, ontology, access, dbt, contracts) that build eight ontology entity types with subtypes and join mappings, a full role hierarchy, and an inline ODCS contract with a quality-run schedule. One apply against a running platform gives you a populated, governed, queryable warehouse with a semantic layer and a contract on top.

The edges, from the comments themselves

What I like about this example is that it documents its own scars, and they tell you real things about the platform.

No write path through chameleon_query. The /query endpoint is read-only by contract, deliberately, and the provider won’t grow a DDL/INSERT path through it. The comment is explicit: for declarative seeding use a chameleon_dbt_project; for ad-hoc setup, run a one-shot Trino script after apply. Tables get created declaratively; they get filled through dbt or a script. Don’t look for a Terraform resource that inserts rows — that’s not the shape.

Principal-role assignment was dropped on purpose. There’s a chameleon_principal_role_assignment resource, and the demo intentionally doesn’t use it. The admin’s roles come through Keycloak realm roles, which land in the JWT’s principal_roles claim — and that claim is what every downstream check (BFF, OPA, Polaris) actually reads. The Polaris-side assignment was a redundant second binding that tripped revoke-on-destroy bugs. The lesson encoded in the comment: express access via realm roles plus chameleon_grant, not via the second binding.

The S3 host-alias prerequisite. Polaris writes table metadata to S3, and the local quickstart only aliases iceberg-warehouse.s3 to the object store — there’s no alias for tf-iceberg-warehouse.s3. So a table create against the prefixed bucket fails with an UnknownHostException unless you add the alias or point at the main warehouse’s bucket. This is the kind of thing you only learn by hitting it, and it’s written down right next to the resource that hits it.

That last category — gotchas captured in comments next to the code that triggers them — is worth more than a wiki page. It’s the difference between an example that demos and an example that teaches.

Why a provider at all

You could manage all this through the portal. For a demo, you should. But a platform that runs real workloads needs its catalogs, grants, contracts, and ontologies to be reviewable, repeatable, and recoverable — and that means they live in version control, go through a plan, and get applied by a machine identity in CI, not clicked by whoever’s on call.

The provider makes the platform’s state a terraform plan away from auditable. You see the diff before it happens. You roll back by reverting a commit. You stand up an identical environment by pointing the same HCL at a different endpoint. That’s the payoff, and it’s why the third layer — the one that needs everything else already running — is the one that turns the platform from operable into governable.