Jacob's Blog

Scaling Terraform Across many Teams: A Native Framework for Platform Engineering

Mon, 12 Jan 2026 16:34:01 GMT

TL;DR: A pure Terraform framework that lets 50+ teams self-service infrastructure by...

How We Build Software with AI Assistants

Jacob Verhoeks — Sun, 22 Mar 2026 00:00:00 GMT

From brainstorm to production in four phases — structured AI collaboration that produces better software than either alone.

Making SQE Work Everywhere: Pluggable Auth and Catalogs

Jacob Verhoeks — Sun, 22 Mar 2026 00:00:00 GMT

How we're turning a single-vendor query engine into something that runs against any identity provider, any catalog, and any cloud.

When Your SQL Engine Understands Meaning

Jacob Verhoeks — Sun, 22 Mar 2026 00:00:00 GMT

SQL engines know table shapes. We're adding ontologies, property graphs, vector search, and AI-native interfaces.

We Replaced Our Trino Fork with a Rust SQL Engine

Jacob Verhoeks — Sun, 22 Mar 2026 00:00:00 GMT

How we went from maintaining a 2M-line Java fork to shipping a 50MB binary that runs every query as the authenticated user.

Building a Comprehensive SQL Benchmark Suite

Jacob Verhoeks — Tue, 24 Mar 2026 00:00:00 GMT

Seven benchmark suites, 222 queries, and the infrastructure to measure performance honestly.

Bringing dbt to the Data Platform: A Browser-Based IDE

Jacob Verhoeks — Fri, 27 Mar 2026 00:00:00 GMT

Integrating dbt Core into Chameleon with a full workspace IDE, git operations, lineage visualization, and AI-assisted model development.

AI Flows: Langflow as the Platform's Intelligence Layer

Jacob Verhoeks — Sun, 05 Apr 2026 00:00:00 GMT

Embedding Langflow as a managed AI service with auto-switching flows, MCP tool integration, and context-aware assistance.

A Visual Editor for Open Data Contracts

Jacob Verhoeks — Tue, 07 Apr 2026 00:00:00 GMT

A full ODCS v3.1.0 contract editor with tabbed forms, live YAML preview, catalog import with AI-suggested quality checks, and dbt export.

Docker Dev Mode: Hot Reload for the Full Stack

Jacob Verhoeks — Wed, 08 Apr 2026 00:00:00 GMT

A docker-compose.dev.yml overlay for instant feedback on frontend and backend changes without rebuilding containers.

Databricks-Style Column Profiling in the Data Explorer

Jacob Verhoeks — Thu, 09 Apr 2026 00:00:00 GMT

Inline column statistics with distribution sparklines computed from real SQL queries, not just Iceberg manifest metadata.

Dropping Radix UI: Going Fully Custom for React 19

Jacob Verhoeks — Thu, 09 Apr 2026 00:00:00 GMT

How a React 19 infinite re-render loop led us to replace all third-party UI primitives with 660 lines of custom components.

From 63% to 95%: Building Trino SQL Compatibility in a Single Day

Jacob Verhoeks — Thu, 09 Apr 2026 00:00:00 GMT

Implementing 70+ UDFs, Iceberg time travel, metadata TVFs, and engine-level SQL features for Trino drop-in replacement.

Streaming Writes, Sort Order Safety, and the IN (Subquery) Workaround

Jacob Verhoeks — Fri, 10 Apr 2026 00:00:00 GMT

Fixing OOM in CTAS, safe Iceberg sort order for mixed writers, and working around DataFusion limitations.

Five Layers of Caching and an 8.8x Speedup Over Trino

Jacob Verhoeks — Sun, 12 Apr 2026 00:00:00 GMT

How multi-layer caching took SQE from slower than Trino to 2.5-8.8x faster across every benchmark suite.

How Agentic AI Helped Us Beat Trino

Jacob Verhoeks — Mon, 13 Apr 2026 00:00:00 GMT

221 queries, 7 suites, one week — how an AI assistant running automated benchmarks drove a major performance breakthrough.

Auditing a Data Platform with AI: Iterative Security Hardening

Jacob Verhoeks — Mon, 13 Apr 2026 00:00:00 GMT

Four audit rounds with escalating AI personas found 42 security issues and removed 6,500 lines of dead code.

Benchmark Data Generation: TPC at the Click of a Button

Jacob Verhoeks — Mon, 13 Apr 2026 00:00:00 GMT

An admin page that generates TPC benchmark datasets on-demand. Pick a benchmark, pick a scale, click Generate — sqe-bench does the heavy lifting in a Docker container while SQE loads the results into Iceberg tables under your identity.

43 Findings, Zero Deferred: A Production Security Audit of a Rust SQL Engine

Jacob Verhoeks — Mon, 13 Apr 2026 00:00:00 GMT

We ran a full production sign-off audit against SQE and found 43 issues across security, runtime safety, logic bugs, and code quality. Then we fixed all of them in one session.

When the Ground Shifts Faster Than People Can Stand

Jacob Verhoeks — Mon, 13 Apr 2026 00:00:00 GMT

A companion piece to The Human Side of the Machine Shift — exploring why AI disruption hits different minds differently, and how leadership can turn discomfort into team strength.

DataFusion 53, a Vendored Fork, and 40% Faster Queries

Jacob Verhoeks — Tue, 14 Apr 2026 00:00:00 GMT

We upgraded SQE from DataFusion 52 to 53 by forking and rebasing iceberg-rust ourselves. The result: 27-40% faster across every benchmark suite.

Our Nemesis: TPC-DS Query 72 and the Limits of a Custom SQL Engine

Jacob Verhoeks — Thu, 16 Apr 2026 00:00:00 GMT

One query. Ten tables. Twelve times slower than Trino. Everything we tried, what worked, what didn't, and where the ceiling is.

The Iceberg Matrix and the Quiet Bug Hiding in V3

Jacob Verhoeks — Sun, 26 Apr 2026 00:00:00 GMT

We thought the V3 path worked. The unit tests said it worked. The matrix called it 'partial' and we agreed. Then we wrote eleven end-to-end tests and discovered Polaris had been silently rejecting every V3 column type for months.

SQE Talks to Five Catalogs Now: HMS, Nessie, Glue, JDBC, S3 Tables

Jacob Verhoeks — Wed, 29 Apr 2026 00:00:00 GMT

We claimed the engine was catalog-agnostic. Time to prove it. One branch, five live integration tests, one small AWS SigV4 patch, and a matrix score that moved from 153 to 158.

Why a Public Iceberg Matrix Beats Vendor Spec Sheets

Jacob Verhoeks — Wed, 29 Apr 2026 00:00:00 GMT

Sixty-three capabilities, three levels, no marketing. The Iceberg Matrix is what compatibility looks like when the rubric is public and the evidence has to land in code. Here is why it works, what we learned from sitting on it, and why every open standard needs one.

How we accidentally created a DuckDB

Jacob Verhoeks — Thu, 07 May 2026 00:00:00 GMT

SQE started as a distributed Iceberg query engine. Five MRs later it queries CSVs from disk, Parquet from S3, and Parquet from HuggingFace. We did not plan that.

One Binary, No Cluster: SQE Goes Embedded

Jacob Verhoeks — Thu, 07 May 2026 00:00:00 GMT

We built SQE for distributed Iceberg, but most of the time you just want to look at a parquet file. Here's how we made the engine work both ways without forking the codebase.

Shipping OpenLineage: column-level lineage for an Iceberg engine

Jacob Verhoeks — Sat, 09 May 2026 00:00:00 GMT

SQE now emits OL 2-0-2 events with column-level lineage on every write. Here is what they look like, why we walked the LogicalPlan to build them, and the disk spool we did not want to write.

Mounting catalogs from SQL: ATTACH, DETACH, and the registry pattern

Jacob Verhoeks — Sun, 10 May 2026 00:00:00 GMT

SQE now ships DuckDB-style ATTACH / DETACH and CREATE / DROP SECRET. The story of building it covers parser extension, credential hygiene, a lifecycle bug we found in the integration tests, and a state-store pattern that is starting to repeat.

Nine PRs, two merge conflicts, and the value of themed branches

Jacob Verhoeks — Wed, 13 May 2026 00:00:00 GMT

Sunday afternoon: an audit dropped eighteen issues into the tracker. Monday morning we had nine merge requests open. Two of them hit conflicts on rebase. Neither one mattered. Here is why that was deliberate, not lucky.

read_parquet shouldn't read /etc/shadow

Jacob Verhoeks — Wed, 13 May 2026 00:00:00 GMT

Modern object-store abstractions unify the filesystem and HTTP behind a single URL. That's a feature for ergonomics. It's a security trap when the URL comes from a user. SQE shipped that trap and then closed it. The IMDS pivot is the part worth telling.

npm install Shouldn't Run Your Code

Jacob Verhoeks — Thu, 14 May 2026 00:00:00 GMT

Supply chain attacks on package registries are surging across every ecosystem. Two controls block the majority: a quarantine window and disabled install scripts. The exact config for npm, pip, Go, Maven, NuGet, Cargo, and more — plus a one-command proxy and shell tests you can run today.

Nineteen MRs, four waves, and the failure modes of agent batches at scale

Jacob Verhoeks — Fri, 15 May 2026 00:00:00 GMT

Two days after the nine-PR audit pass, we ran a bigger one. 130 issues filed, 19 themed MRs merged across four waves. Same workflow, more failure modes. Watchdog stalls, a reboot mid-wave, a broken main, and config.rs as the conflict magnet. Here is what actually happened.

q72, our nemesis, and the Int32 that hid for a month

Jacob Verhoeks — Sat, 16 May 2026 00:00:00 GMT

TPC-DS q72 sat at 10 seconds while every other query ran in under 1.4. Five days of investigation chased scan parallelism, range-based NDV, iceberg-rust upgrades, and the RisingWave fork. None of those were the bug. The bug was a silently-skipped Err arm in our dynamic-filter evaluator that swallowed every Int32 vs Int64 type clash. Fixing it: 15.5s to 0.77s. q72 now beats Trino.

The SSB regression that wasn't

Jacob Verhoeks — Sun, 17 May 2026 00:00:00 GMT

MR #220 wired runtime filters into iceberg-rust's scan path and dropped TPC-DS 67%. SSB looked like it regressed 6%. Two failed heuristic attempts, one parquet-trace session, and ten warm passes later, the regression turned out to be measurement noise. The fix-the-fix that wasn't, and the data-clustering insight that explains why two suites with the same code path behave nothing alike.

Porting DuckDB's BinarySerializer to pure Rust

Jacob Verhoeks — Mon, 25 May 2026 00:00:00 GMT

Ten sub-MRs in a day. The wire format, the fixture-driven debugging loop, and two bugs that the C++ reference encoder ships without telling you: WriteListWithDefault elision, and uninitialised bytes at NULL VARCHAR positions.

The Last Mile Problem: Why 100% Supply Chain Coverage is Hard

Jacob Verhoeks — Mon, 25 May 2026 00:00:00 GMT

Every time a developer runs npm install, they're implicitly trusting hundreds of strangers. Intercepting that trust is straightforward in theory — and surprisingly subtle in practice.

Speaking Quack: SQE as a DuckDB server, a DuckDB client, and a federation engine

Jacob Verhoeks — Tue, 26 May 2026 00:00:00 GMT

DuckDB 1.5 ships a wire protocol called Quack. We re-implemented it in pure Rust, turned SQE into both server and client, and proved you can JOIN an Iceberg table with a remote DuckDB table in a single SELECT. Two waves of bugs, one federated query.

The type matrix as a roadmap: seven DuckDB types in two days

Jacob Verhoeks — Tue, 26 May 2026 00:00:00 GMT

We started with a markdown table tracking which DuckDB types we could round-trip. The table became the roadmap. DECIMAL, LIST, STRUCT, MAP, ARRAY, ENUM, UNION each got their own MR. The surprises came from how DuckDB models the relationships: MAP is LIST, UNION is STRUCT with a tag field, and DECIMAL packs four widths into one logical type.

Renovate + Supply Chain: What It Actually Intercepts (and What It Doesn't)

Jacob Verhoeks — Wed, 27 May 2026 00:00:00 GMT

We tested Renovate's registry behaviour for every ecosystem — npm, PyPI, Go, Cargo, Maven, NuGet, Composer — and found that minimumReleaseAge works for all of them, but how each datasource resolves its registry is surprisingly different. Here's what we found and how to build a complete defence.

Supply Chain Attacks Hit Every Ecosystem. Here's How to Defend Yours.

Jacob Verhoeks — Wed, 27 May 2026 00:00:00 GMT

A three-layer defense for developers and DevOps engineers: harden your dev machine in 5 minutes, lock down project configs for every ecosystem, and enforce policy org-wide with a proxy. Covers npm, pnpm, yarn, bun, pip, uv, Go, Maven, Gradle, NuGet, Cargo, and Composer — plus escrow, JFrog Curation, and Nexus.

Data Quality That Lives in a Contract