Sovereign by Design

Building a Production Query Engine on DataFusion

The complete journey of replacing Trino with a purpose-built Rust query engine. From first DataFusion query to distributed execution over Apache Iceberg, with OIDC auth passthrough, policy-as-plan-rewriting, and zero service accounts.

PDF (2.2 MB) EPUB (2.0 MB)

18 Chapters

~440 Pages

5 Parts

316 Commits in 15 days

Part I — Why Build

0. Preface: The Sovereignty Thesis
1. The Catalog Wars — From Hive Metastore to REST catalog. Why Polaris wins for sovereignty.
2. Tables Made of Files — Apache Iceberg from first principles. iceberg-rust in production.

Part II — First Query

3. The Engine You Already Have — DataFusion as a library, not a service.
4. You Are the Query — OIDC passthrough. No service account. Bearer token to Polaris and S3.
5. Speaking Arrow — Arrow Flight SQL protocol. The wire protocol as user experience.
6. The Catalog Is the API — information_schema, system.runtime.*, dbt integration.

Part III — Making It Real

7. The Write Path — INSERT, CTAS, MERGE, DELETE. Streaming writes and conflict resolution.
8. What You Can't See Can't Hurt You — Policy-as-plan-rewriting. Row filters, column masks.
9. Observability Without Surprise — Prometheus, OpenTelemetry, health endpoints.
10. Configuration Is the Product — TOML config, plugin points, twelve-factor applied.

Part IV — Going Distributed

11. Why Distribute at All — When single-node stops being enough. The decision framework.
12. Standing on Ballista's Shoulders — What we kept and what we replaced.
13. The Coordinator and the Worker — Plan splitting, fragment assignment, the trust boundary.
14. Failure Is a Feature — Worker crash, network partition, 50 concurrent clients.

Part V — Production

15. Deploying Sovereignty — Docker multi-stage (2.3GB to 47MB), Helm, rolling upgrades.
16. Benchmarks Don't Lie (But They Mislead) — TPC-H, TPC-DS, SSB. SQE vs Trino on same hardware.
17. What We'd Do Differently — The AI-assisted build: honest assessment.

Open source The SQE source code that accompanies this book is pending approval for open source release under the Apache 2.0 license. Once available, each chapter will reference specific commits, crates, and tests from the repository.