· Jacob Verhoeks · 4 min read

Bringing dbt to the Data Platform: A Browser-Based IDE

Integrating dbt Core into Chameleon with a full workspace IDE, git operations, lineage visualization, and AI-assisted model development.

dbtdata-engineeringdockerreact

We integrated dbt Core into Chameleon with a full workspace IDE, git operations, lineage visualization, and AI-assisted model development. All of it runs in Docker containers.

Why dbt belongs in the browser

Data engineers love dbt. It brings version control, testing, and documentation to SQL transformations. But the typical workflow means installing the CLI locally, managing Python environments, configuring credentials, and switching between terminal, editor, and browser.

We wanted dbt as a first-class citizen of Chameleon. Not a bolt-on. A native experience where data engineers write models, run builds, view lineage, and manage projects from the same browser tab where they browse catalogs and run queries.

The architecture

Workspace isolation

Each dbt session runs in an isolated Docker container with its own workspace. When a user opens a project, we either clone the git repo or copy the scaffold template into a fresh workspace directory.

  • No shared state between users. Two people can work on the same project simultaneously without conflicts.
  • Clean environments. Each session starts fresh. No leftover build artifacts.
  • Security boundaries. Workspaces run with the user’s own credentials, not a shared service account.

The backend manages workspace lifecycle: creation, file I/O, git operations, dbt command execution, and cleanup of stale sessions older than 24 hours.

The scaffold template

New projects start with a scaffold that includes sample files, not empty directories with .gitkeep. The template generates:

  • models/sources.yml with an example source definition pointing at the user’s catalog
  • models/staging/stg_example.sql with a staging model using {{ source() }} ref
  • models/staging/schema.yml with model tests (not_null, unique)
  • tests/assert_example_positive.sql as a custom test example
  • profiles.yml pre-configured for the platform’s Trino/SQE endpoint
  • macros/sqe_overrides.sql with compatibility macros for our Sovereign Query Engine

A new user can create a project, click “Compile,” and see it succeed immediately. No configuration required.

Git integration

Projects can be linked to a GitHub repository via our GitHub App OAuth integration. The workspace supports full git operations: clone, pull, commit, push, branch creation. Credentials are managed server-side through encrypted connections. The frontend never sees git tokens.

AI-assisted development

The dbt workspace integrates with our Langflow-based AI flow system. The “dbt Assistant” flow has access to the data catalog via MCP tools and can:

  • Generate dbt models from natural language descriptions
  • Explain existing models and their lineage
  • Suggest tests based on column profiling data
  • Help resolve compilation errors

The assistant activates automatically when you navigate to the dbt workspace. No manual flow selection needed.

Lineage visualization

The lineage page uses React Flow with dagre auto-layout to render the dbt dependency graph. Two view modes:

  • Compact shows simple nodes with status badges for run/test results
  • Detailed shows ER-diagram style nodes with columns and test status “traffic lights”

Column-level lineage, powered by sqlglot, draws dashed lines between specific columns across models. You see exactly how each field flows through the transformation pipeline.

We also bridged dbt lineage to the Data Explorer. When viewing any table in the catalog browser, the Lineage tab shows upstream and downstream dependencies from active dbt sessions.

What we learned

Docker-in-Docker is tricky. The backend runs as a container that needs to spawn dbt containers. We use Docker socket mounting with proper GID mapping. Template projects with no git repo required special handling: copying the scaffold instead of cloning from an empty URL.

Connection management matters. The connection_id foreign key on dbt projects had to be nullable for template-based projects that don’t need git. This sounds trivial but caused 500 errors until we fixed the constraint.

Stale sessions accumulate. Without cleanup, workspace directories grow forever. We added a cleanup loop that expires sessions older than 24 hours and removes their workspace files.

dbt Core integration was built in March-April 2026 as part of the Chameleon Data Platform. The full implementation plan is in docs/superpowers/plans/2026-03-27-dbt-core-integration.md.

All posts