Benchmark Data Generation: TPC at the Click of a Button

We added an admin page that generates TPC benchmark datasets on-demand. Pick a benchmark, pick a scale, click Generate. sqe-bench does the heavy lifting in a Docker container while SQE loads the results into Iceberg tables under your identity.

The problem

The Sovereign Query Engine ships with sqe-bench, a Rust-based benchmark tool that generates TPC-H, TPC-DS, SSB, TPC-C, TPC-E, TPC-BB, and ClickBench datasets. It’s fast — generates millions of rows in seconds, deterministically, with no external dependencies.

But using it means cloning the sqlengine repo, building the Rust binary, running CLI commands with the right flags, manually creating schemas, and loading Parquet files into the catalog. That’s fine for engine developers. It’s not fine for someone who just wants realistic test data in their warehouse for a demo or a performance evaluation.

We wanted this: open the platform, pick TPC-H at scale factor 1, point it at a catalog, click a button, come back a few minutes later to a warehouse full of properly structured benchmark data.

Two-phase architecture

The challenge was connecting sqe-bench (a Rust CLI that writes Parquet to disk) with the data platform (a Python backend that manages catalogs via SQE Flight SQL). We split it into two phases:

Phase 1 — Generate. A Docker container runs sqe-bench generate tpch --scale 1.0 --output /data/bench. It writes Parquet files to a shared host directory. No database connection, no authentication, no network calls to the catalog. Just fast, local data generation.

Phase 2 — Load. The backend reads those Parquet files into Iceberg tables via SQE’s read_parquet() function. This runs through Flight SQL using the user’s own JWT token. Tables are created under the user’s identity, respecting their RBAC permissions. If you don’t have CREATE TABLE on that namespace, it fails — exactly as it should.

sqe-bench container          SQE (Flight SQL)
───────────────────          ────────────────
writes /data/bench   →host→  reads /tmp/sqe-bench-data
                             CREATE TABLE ... AS SELECT *
                             FROM read_parquet('/tmp/sqe-bench-data/...')

The shared directory is the bridge. sqe-bench writes into it (container volume mount, writable), SQE reads from it (already mounted read-only in the compose stack). No S3 staging bucket needed.

The Docker image

sqe-bench wasn’t previously containerized. The main SQE Dockerfile builds sqe-server, sqe-worker, and sqe-cli — but not sqe-bench. We added a Dockerfile.bench that reuses the same cargo-chef multi-stage build pattern:

FROM deps AS builder
RUN cargo build --release --bin sqe-bench

FROM debian:bookworm-slim
COPY --from=builder /build/target/release/sqe-bench /usr/local/bin/
ENTRYPOINT ["sqe-bench"]

The result: a 39 MB image containing a single static-ish Rust binary with TLS support. It builds in a few minutes thanks to the cached dependency layer.

In the compose file it’s a build-only service:

sqe-bench:
  image: sqe-bench:latest
  build:
    context: ../../../sqlengine
    dockerfile: Dockerfile.bench
  profiles:
    - build-only

Build once with docker compose build sqe-bench, then the platform spawns containers from the image as needed.

The admin page

The /admin/benchmarks page has two sections:

A form for configuring the run:

• Benchmark type dropdown (all seven supported)
• Scale factor with live size estimates — when you pick TPC-DS at scale 10, it shows “~25 GB” so you know what you’re committing to before clicking Generate
• Catalog and namespace selection (create new or use existing)
• Overwrite checkbox for re-running on an existing namespace

A runs table showing active and past runs. Each row expands to show the log output. Status badges track progress through the phases: pending → generating → loading → completed. Active runs have a cancel button.

The frontend polls every 3 seconds while a run is active, then stops when it reaches a terminal state. One active run per user prevents accidental container storms.

How the backend orchestrates it

The BenchmarkService manages the lifecycle:

1. Validate — benchmark exists, scale is reasonable, user doesn’t already have an active run
2. Create staging directory — os.makedirs(/tmp/sqe-bench-data/{run_id}/, exist_ok=True)
3. Launch container — via ComposeExecutor.start(), same pattern as dbt jobs. The container gets the staging dir mounted at /data/bench
4. Poll — check container status every 2 seconds, capture logs
5. Load via Flight SQL — CREATE SCHEMA IF NOT EXISTS, then CREATE TABLE ... AS SELECT * FROM read_parquet(...) for each table
6. Cleanup — shutil.rmtree() on the staging directory

Error handling is straightforward. If the container fails, the run is marked failed with the last 500 characters of log output. If a table creation fails mid-load, partial tables remain — the user can re-run with overwrite enabled. If the JWT expires during load, it fails with an auth error and the user re-runs.

Why read_parquet() means Flight SQL, not Trino

This was a design trap we almost fell into. The original plan had the load phase going through Trino. But read_parquet() is an SQE-specific table-valued function — it doesn’t exist in standard Trino.

The data platform already has a Flight SQL client (clients/flight.py) that connects to SQE. It uses the same execute_query(sql, user_token, username) interface as the Trino client. We just pointed the load phase at SQE instead of Trino. The SQL is the same, the auth is the same, the behavior is the same — except SQE can read Parquet files directly.

The feature is gated on FLIGHT_ENABLED=true. If SQE isn’t in the stack, the benchmark service doesn’t initialize and the page isn’t accessible.

Scale estimates

Nobody wants to accidentally generate 250 GB of TPC-DS data. The scale dropdown shows estimated sizes per benchmark:

These are stored in benchmark/constants.py and served via the /benchmarks/metadata endpoint. The frontend updates the estimate dynamically when you change the benchmark or scale selection.

What we learned

sqe-bench already had S3 CLI flags, but they weren’t wired up for generate. The --s3-endpoint, --s3-bucket arguments exist in the clap definition, but the generate command ignores them and writes to local disk only. The shared volume approach turned out simpler than adding S3 write support — and faster, since it avoids a network hop.

Reusing ComposeExecutor from the dbt runner was trivial. Same pattern: start() returns a container ID, get_status() polls it, stop() kills it. The only difference is the image and the command. The executor doesn’t care what’s inside the container.

The read_parquet() path inside SQE matters. The SQE container mounts /tmp/sqe-bench-data at a specific path. The Parquet path in the SQL must match what SQE sees, not what the backend sees. This coupling is documented in the service but could bite if someone changes the SQE volume mounts without updating the benchmark config.

Benchmark data generation was built in April 2026 as part of the Chameleon Data Platform. Design spec: docs/superpowers/specs/2026-04-13-benchmark-data-generation-design.md. MR: !40.