dbt, Built Into the Platform

There are two ways to integrate dbt into a data platform. You can shell out to the dbt CLI and call it integration. Or you can build the IDE — git-backed projects, an editor, a runner, lineage, quick-test — and run dbt as a first-class citizen of the platform. We built the second one.

The bet behind it: dbt is where the transformations live, so dbt is where the analytics engineers live, so the platform should meet them there instead of making them context-switch to a terminal and a separate repo checkout. A browser tab, a git branch, a Run button, and the results in the same place as the catalog they’re transforming.

This is the deep dive. How a project becomes a workspace, how a Run becomes a container, how {{ ref() }} becomes a query you can test in one click, and where the regex shortcuts will bite you.

The shape: projects, sessions, runs

Three nouns structure everything, and they map to three tables.

A project is a dbt repo plus a target. It points at a git connection and a repo URL, and it carries two fields that decide where the SQL lands: warehouse (the catalog, default main_warehouse) and namespace (the schema). A project can also have no remote at all — you can scaffold one from a template and git init it locally.

A session is an ephemeral workspace. Opening a session clones the repo — shallow, depth=1 — into a per-session directory. The clone happens in the background; the API returns immediately with status: cloning, and the frontend polls for the flip to active. Sessions expire after 24 hours and get swept. This is the clone-on-open model: your edits live in a workspace that’s yours for the session and then goes away.

A run is one invocation — compile, run, test, build, seed — captured with its command, status, logs, and the parsed manifest.json and run_results.json.

The connection deserves a note, because it’s the source of a naming tension across the platform. A connection here is git credentials — a PAT or a GitHub App installation token, Fernet-encrypted at rest, decrypted only to clone. It is not a warehouse connection. The query target is wired separately, which matters for the next part.

From Run to container

Click Run. Here’s the path, end to end.

The request hits DbtService, which first writes a profiles.yml directly into the session workspace. That profile always targets Trino over JWT:

trino:
  outputs:
    dev:
      type: trino
      method: jwt
      jwt_token: "{{ env_var('DBT_ACCESS_TOKEN') }}"
      host: <trino_host>
      database: "{{ env_var('DBT_TARGET_CATALOG', 'main_warehouse') }}"
      schema: "{{ env_var('DBT_TARGET_SCHEMA', 'dev') }}"

Your OIDC token rides in as DBT_ACCESS_TOKEN. The catalog and schema come from the project. The git connection plays no part here — git got you the code; the token gets you the data.

Then DbtService builds the command and hands it to an executor. In production that’s the ComposeExecutor: it runs docker run --rm --network host, with sensitive env values redacted from the logs, and streams stdout line by line into a bounded buffer (the last 10,000 lines). It returns a container name as the execution ID and returns control immediately. The container runs in the background. The frontend polls every two seconds; the run status reads the live log tail while running, and on exit maps the code to success or failed and persists the logs.

There’s a LocalExecutor too — same interface, plain subprocess, no Docker — for development.

Two honest edges here. The container is fire-and-forget: process state lives only in memory, so on a backend restart, orphaned runs get marked failed in the database even though the Docker container may still be churning away. And --network host is hardcoded, which keeps the networking simple and the portability low. Both are fine for the deployment they were written for and worth knowing before you move it.

The quick-test trick

This is the most interesting piece, and the one with the sharpest trade-off.

A dbt model is full of {{ ref('other_model') }} and {{ source('raw', 'orders') }}. You can’t run that SQL directly — it’s Jinja, not SQL. Normally you’d dbt compile the whole project to resolve it. Quick-test lets you run one model’s SQL against Trino in a click, without a full compile, by resolving the refs in place.

The resolver is a regex. Not a Jinja interpreter — a set of six compiled patterns. {{ ref('x') }} gets looked up in the manifest and replaced with the fully-qualified relation name, or it falls back to catalog.schema.x, or — if it can’t resolve at all — it emits /* unresolved: ref('x') */ NULL. Sources resolve from sources.yml. Config blocks, comments, and other Jinja expressions are stripped.

The flow gives the user a chance to correct it. The frontend detects refs, asks the backend to resolve them, and if any are unresolved it opens a dialog where you confirm or override each mapping. The resolved SQL — yours to edit — goes to the test endpoint, which runs it against the engine under your token and returns columns and rows in the Results tab.

The trade-off is the regex. Macros, if blocks, loops, custom functions — all stripped to nothing. A model with non-trivial Jinja produces approximated SQL that may run and quietly compute the wrong thing, because an unresolved ref became NULL and the query still parsed. Quick-test is a fast feedback loop for straightforward models, not a substitute for dbt compile. Treat a green quick-test on a macro-heavy model with suspicion.

Lineage, twice

dbt manifests describe a graph, and the platform parses that graph in two completely separate places for two different purposes.

The first is the in-workspace lineage for the dbt UI. LineageParser walks the manifest into nodes (sources, models, tests, seeds, snapshots) and edges (from each node’s depends_on). Column-level edges come from sqlglot.lineage(), computed per model-column against a schema built from the manifest — and capped at 1,500 columns total, because the alternative is blocking the event loop for a minute on a big project. Above the cap, you get table edges only. Run results overlay onto the graph: a model node carries its run status, a column carries its test pass/fail. The result renders in the dbt lineage page, and a per-process cache keyed on file mtime keeps it from re-parsing an unchanged manifest.

The second is the OpenLineage integration for the platform-wide lineage system. A separate module converts the same manifest into OpenLineage RunEvents — one per model — with deterministic run IDs (UUID5 seeded from the node) and a deterministic event time (the manifest’s generated_at, not now()), so re-uploading the same manifest is idempotent. Those events feed the platform’s general lineage graph, the one that also ingests Spark and engine events.

Two parsers, one manifest, no shared code. It’s duplication, but they genuinely serve different graphs with different identity models, so it’s defensible duplication.

Source freshness, the source mapper, and scheduling

A few more pieces fill out the IDE.

Source freshness answers “is this table stale?” by combining the manifest’s configured thresholds with the last-run state in sources.json. The endpoint walks every active session to find one whose manifest knows the table — a linear scan, cached by file mtime, but still a scan. If no session knows the table, it returns is_dbt_source: false with a 200, so the UI just hides the widget instead of erroring.

The source mapper generates sources.yml from the live Polaris catalog. Pick a catalog, namespace, and tables; the backend fetches each table’s current schema from Polaris (using the current schema ID, so it’s not stale after a migration) and emits the YAML. It can also validate an existing sources.yml against the catalog and tell you which columns drifted.

Scheduling wires DbtService.run() into the platform’s unified scheduler. A Schedule button on the workspace toolbar pins a recurring build to the active session. The scheduler runs it on cron, under the schedule owner’s token.

What you see

The workspace is a real IDE in a browser tab. A collapsible file tree grouped by directory. A CodeMirror editor with Validate, Format, and Query for SQL files. A toolbar with branch info, Pull (with a badge counting remote changes), Seed, Compile, Run, Test — each with its own status dot — plus Commit & Push. A bottom panel with tabs for Logs (ANSI colors parsed), Validation, Results, structured Test results with pass/fail/error/skip badges, and Run history with durations and per-run log links.

The git workflow is all there: branch, diff, commit, push, pull, remote-status. You work the way you’d work in a local checkout, minus the local checkout.

OpenTelemetry: the orchestration, not the run

The dbt subsystem emits OpenTelemetry spans — dbt.open_session, dbt.execute, dbt.executor.start, dbt.test_query, and more — exported over OTLP to OpenObserve. But read the boundary carefully: these instrument the Python orchestration layer, the code that manages sessions and launches containers. The dbt process itself is not instrumented. The runner image installs dbt-core and dbt-trino and nothing else. There are no model-level spans, no per-node timing from inside the run.

The project goal was “dbt runs emit traces visible in Jaeger.” What shipped is orchestration spans to OpenObserve. The model-level tracing inside dbt isn’t there, and Jaeger isn’t configured anywhere — the trace target is OpenObserve. If you’re expecting a flame graph of a dbt run’s model DAG, that’s not what this produces today.

What’s there, and what isn’t

Implemented and working: git connections with encryption and GitHub App tokens; project CRUD with a scaffold path; the full session lifecycle; compile/run/test/seed via container or subprocess; quick-test with the confirmable ref dialog; SQL validation and Jinja-aware formatting; manifest lineage with column edges and run-result overlay; source freshness; the catalog-integrated source mapper; scheduled runs; orchestration-layer telemetry; the full browser IDE; and OpenLineage manifest conversion.

Defined but not wired: the AI “generate model” and “generate test” endpoints. The request models exist in the code. The service, the router, and the panel do not. The workspace can talk to the global AI chat (it passes the current SQL as context and can take SQL back), backed by a Langflow dbt-assistant flow — but the dedicated generate-model-from-a-prompt feature is a forward declaration, not a feature.

One more, for the deployment file: the generated profiles.yml sets cert: false and the container runs with HTTPS warnings suppressed. Correct for a self-signed local environment. A thing to revisit before production.

The honest summary: this is one of the most complete surfaces in the platform. A browser-native dbt IDE that clones, edits, runs, traces lineage, and tests — grounded in the same catalog and the same identity as everything else. The regex resolver and the fire-and-forget containers are the two places where the pragmatism shows. Everywhere else, it earns the word “integration.”