claudeinsights coaches me on my own Claude Code sessions by sending each rough one to a model. Every session used to cost a full claude -p call. Now a 421M-parameter encoder called Laya scores all of them locally first, and only the rough ones go to Opus 5.5 for quotes and advice. Go downloads and caches the checkpoint through go-huggingface, sharing Python's Hugging Face cache without a single extra byte, and its pure-Go tokenizer matches Hugging Face's Rust tokenizer id for id on Laya's vocabulary, except for one lstrip flag. The model itself still runs in PyTorch, and skipping one random-weight initialisation took its load from 46 seconds to 4.
We pointed SQE's MCP endpoint at an Okta tenant and asked Claude Code to query a Glue Data Catalog as a real person. The result is the Atlassian-plugin experience for a lakehouse: no token to paste, no client secret, no service account. Okta issues the access token in the browser, SQE verifies it, and Apache Ranger decides which tables that person's groups may read. Three Okta settings decide whether it works at all, and two configuration slips look exactly like authorization failures without being one. This is what we set, what broke, and what the System Log told us.
SQE now speaks Model Context Protocol over Streamable HTTP, and it does it as an OAuth resource server: no service account, no token minting, no credential proxy. The caller's bearer becomes an SQE session and travels to Polaris and S3 unchanged. Read tools are gated on one scope, the read-only SQL tool is enforced by SQE's own parser rather than a keyword regex, and the write tool needs three independent authorizations that do not substitute for one another. The runnable quickstart proves it with 14 checks against live Keycloak, Polaris and RustFS, including the one that matters most: a reader holding the OAuth write scope is still denied by the source.
Cycle 2 of SQE's Hive external tables replaces the per-query prefix LIST with a partition index persisted in the sidecar manifest, maintained by MSCK REPAIR TABLE and ALTER TABLE ADD/DROP PARTITION. Athena asks Glue for the partition list over an API. SQE reads one ETag-cached JSON object sitting beside the files. On a 449-partition slice of the public Bitcoin dataset a pinned query goes from 99 ms to 1.9 ms cold, and DuckDB on the same files stays at 44 ms because it re-globs every query. Partition projection is implemented too, and measurably slower than the index on the same pin.
SQE now reads Hive-style external tables over CSV, line-delimited JSON, and partitioned Parquet on any S3-compatible store, using Athena's own CREATE EXTERNAL TABLE syntax. There is no Hive Metastore, no Glue API call, and no catalog service of any kind behind the table: the definition is a JSON manifest next to the data, shaped like a Glue TableInput. This post is what emulating Athena and Glue actually requires, which parts we copied deliberately, the four Athena behaviours we refused to copy, and the credential difference that is a real change from how SQE treats Iceberg.
AMD, Apple and NVIDIA all sell a 128 GB local-AI box. The TOPS numbers are not comparable. Bandwidth is. Netherlands prices, real decode speeds, what 70B at 512K actually costs in memory, and how Qwen3.8-27B changes the 4/8/16-bit trade-off.
A working catalogue of the access control SQE and Spark share through Apache Ranger: object grants at the catalog, five column masks, row filters, tags that span tables, four personas built from the same primitives, and the two places the engines still render differently. Everything here is a line from a run that finished 43 of 43, on an EU bank fixture rather than a toy table. Includes the reason there are two enforcement points rather than one, and what changed when we moved every privilege write onto an authenticated endpoint.
A complete, executable account of how SQE and Spark query the same Iceberg table under the same Apache Ranger policies: Polaris object grants, row filters, column masks, tags, precedence, writes, views, and the differences that remain.
Spark was connecting to our catalog as root, which meant every access-control policy we had written applied to exactly one engine. Fixing it took no engine code: Polaris already authorizes against the OIDC identity, so a per-user token was the whole change. Then we ran both enforcement tiers together for the first time and found they compose in an order nobody documents, with Spark's plugin answering before the catalog is asked. Along the way two of our own new tests passed for the wrong reason, the two engines turned out to disagree about which mask wins, and renaming a column silently removes its protection. Here is the full evaluation order, what we measured, and why Databricks and Snowflake do not have this class of problem.
Access control was the last subsystem whose end-to-end behaviour we checked by grepping CLI output. The denial test matched the string 'not found', which is also what a typo'd table name prints. We replaced that harness with twenty assertions on decoded Arrow values against a live Apache Ranger, and the exercise found a feature that had never once worked: every tag-based column mask had been restricting the column instead of masking it since the day we shipped it. Two of the new tests then passed for the wrong reason, and only mutation caught them. Here is what fine-grained access control in SQE actually does, and how we know.
SQE's founding rule is no service account: every query runs as the authenticated user, tokens passed through to Polaris and S3. Background compaction has no user to be. Here is how we gave Iceberg tables a self-healing maintenance path while keeping that rule intact: a structurally isolated principal, deny-by-default grants, and a commit authority that never leaves the coordinator, even when the rewrite itself is distributed across the worker fleet.
A Python ingestion job should not have to choose between an open Iceberg catalog and a governed SQL engine forever. We connected dlt directly to Apache Polaris through Iceberg REST, then sent the same full, incremental, and SCD2 loads through SQE over Trino HTTP and Arrow Flight SQL. The result is one service-principal identity, one catalog, three front doors, and a test suite large enough to expose the bugs hiding at batch boundaries.
SSB at SF10 was the last benchmark where Trino beat us, and the gap was one serial scan feeding eight idle cores. Closing it took three fixes: parallelize the probe scan, assign byte-range splits instead of whole files, and stop trusting a flag that had never actually fired. Along the way the row counts started lying. Three TPC-DS queries returned exactly eight times their LIMIT, and the mechanism is a lesson in what parallelism does to a plan that was optimized while it was still serial.
Tag-based masking is two systems wearing one name: a rule that says what a tag means, and an association that says which columns carry it. We put the rule in Apache Ranger, where Spark can share it, and the association in the Iceberg table metadata, where it survives rename, replication, and federated catalogs that no policy store can see. Here is the split, the precedence contract, and the identity bug the tests now pin down.
A lakehouse rarely has one query engine. Spark writes the tables, SQE serves the interactive queries, and both touch the same Iceberg data. The usual failure is policy drift: each engine has its own access-control plugin, the same masking intent gets translated twice, and the two translations disagree. We avoided that by not translating twice. SQE and Spark read the same Apache Ranger hive service, so one policy written once enforces the same way in both. The SSN reads xxx-xx-1111 whichever engine ran the query.
Reads on SQE spill to disk when they run out of memory. Writes did not. A large CTAS, a wide MERGE, or an oversized client upload could balloon a coordinator buffer past its memory limit and get the process OOM-killed, which takes every other query on that node with it. We gave the write path the same memory discipline the read path already had: pool-track the buffers that must exist so an oversized write fails as one typed error, and stream the ones that never needed to buffer at all.
A BI tool connecting to a query engine is not one query. It is a scripted handshake of a dozen metadata calls, and every one has to look exactly like Trino or the tool gives up without an error. We pointed a real Metabase at SQE's Trino endpoint and watched it fail six different ways: a PREPARE the parser rejected, a SHOW TABLES column that collapsed every table into one, catalogs it never enumerated, quoted identifiers that matched nothing, and a timestamp type signature the JDBC driver refused to parse. None of them threw. They just showed nothing.
SSB is the one benchmark suite SQE loses to Trino, 2.5x at scale factor 10, while it wins everything else. The obvious cause was a single-threaded fact-table scan, so we built a planner rule to parallelize it. A correctness smoke caught the rule returning 240 million rows from a 60-million-row table, a latent 4x duplication bug we then fixed. Then the parallel scan turned out to change nothing: SSB did not move. The real cause is a dynamic filter that is a huge win on clustered fact tables and pure overhead on SSB's uniformly-distributed one. A slow number is a hypothesis, and the fix you are sure of can be perf-neutral.
S3 server access logs could only land in an AES256 bucket for years, and AES256 is exactly what Security Hub flags. The escape was never a bucket setting. It was a KMS key policy that lets the logging service write with your key.
SQE got a second access-control backend: write GRANT/REVOKE as Apache Ranger policy and let Polaris enforce it. The protocol took an afternoon. The identity model took the week, because Polaris federation does not work the way the token suggests.
SQE's fine-grained Ranger backend does row filters, column masks, role-conditional masking, and tags. The kicker: the same Ranger policy on the same Polaris catalog produces byte-exact identical masked output in SQE and in standard Apache Spark. We proved it live, and the first run failed for a reason no amount of reading could have predicted.
Snowflake gives you masking policies, row access policies, object tags, and a GRANT model. SQE gives you the same primitives on Apache Ranger and open Iceberg, enforced by plan rewrite and shared across engines. Here is the mapping, the one real edge SQE has, and the gaps we have not closed yet.
DataFusion 54 landed and we bumped to it. The port was mostly mechanical, with one real behavioral change in the shuffle hasher. The interesting part is what the release notes implied and the engine did not deliver: LATERAL joins are logical-plan only, array lambdas still fail, and the one compatibility win we found was a documentation bug, not a DataFusion feature. We tested every claim before we wrote it down.
A two-table TPC-H join that Trino ran in 2.2s took us 161s at SF10. We blamed partition layout, then single-node joins, then a subquery pattern. All three were wrong. A CPU profile and two timers found the truth: a runtime filter we pushed to the probe scan was getting re-snapshotted once per batch, and each snapshot rebuilt a 300,000-node expression tree. The fix snapshots it once. q12 went 161s to 2.7s, q17 176s to 7.1s, q10 from a 300s failure to 3.3s, with the result rows unchanged and no knob touched.
TPC-DS q95 was our worst query: 18 seconds against Trino's 1.3, a 14x loss that justified building a whole optimizer feature. Before we wrote a line of it, we pulled the plan and the profile. The 12-million-row self-join the feature was meant to shrink did not exist, the engine ran the query in under half a second, and the 18 seconds lived only in a benchmark harness running both engines on one starved host. On a clean rig SQE runs q95 in 240ms and beats Trino 12x. A slow benchmark number is a hypothesis until you reproduce it in isolation.
On SSB at SF1 Trino ran scan-heavy queries about 2x faster than SQE, even though we pruned just as well. A new line in our query profile found it: a 151MB lineorder file decoded on a single thread, 94% of a 969ms query spent waiting on one scan. The obvious fix (more partitions) was a trap that regressed q72 from 17s to 100s once before. The safe fix parallelizes decode inside the scan without changing the plan the optimizer sees.
TPC-H q01 returned six (returnflag, linestatus) combinations on our generated data. The spec defines exactly four. Both SQE and Trino agreed on the wrong answer, because both read the same broken tables. The bug class behind it: fields the spec derives from other fields, drawn instead as independent uniform random. Five defects in TPC-H, one in SSB, all the same shape.
Our SF0.1 compare run looked great: zero mismatches across seven suites. Then we asked DuckDB to check the data and found that 16 'passing' TPC-DS queries had never selected a single row, TPC-C had zero warehouses, and the one real disagreement between SQE and Trino was Trino's fault. Plus: the dynamic filter that shipped 6 million rows because nobody carried it across a node swap.
Differential testing against Trino caught a distributed-scan bug the whole test suite missed: every projected query failed once a scan actually distributed. The first fix restored correctness by disabling projection pushdown. Then a Claude agent on the Fable model found the real bug, one line in the worker's streaming rewrite, and got the speed back: 3.1x overall, 10x on the worst query.
Our own README promised fine-grained Lake Formation in the SQE Glue quickstart. The engine does not do it. Here is what Lake Formation actually enforces when SQE reads S3 directly, and where SQE's real column and row masking lives.
We turned every 'how to run SQE for X' into a self-contained run.sh that goes from clean state to captured output. The point was documentation. The payoff was a test suite that caught a missing metric, an overclaimed capability, and a benchmark that would have polluted our baselines.
Multi-tenancy on a data platform is usually a ticket and a week of console clicking. We made it a Terraform apply: a workspace, its catalog, its groups, its grants, and its dbt project — isolated by construction, reviewed in a pull request.
SQE already tracked every query and worker in memory, but there was no way to look at it without tailing logs or wiring Grafana. So we put a read-only dashboard on the health port: no login, no build step, no external assets. One HTML page over a small JSON API, with a tiny in-memory sampler so the charts move over time. Here is how it is built and why it stays deliberately small.
Data quality belongs in the table's contract, not scattered across dbt tests and cron jobs. We build on the Open Data Contract Standard, compile quality rules to portable SQL, and run them on the live engine under your identity — plus an honest accounting of what's wired and what isn't.
Not shelling out to the dbt CLI and calling it integration — a browser-native dbt IDE with git-backed projects, container runs, column-level lineage, and one-click quick-test. How a project becomes a workspace, and where the regex shortcuts bite.
A semantic layer that doesn't end in a triple store. The schema lives in Postgres, the data stays in Iceberg, and one query translator compiles every façade — MCP, GraphQL, SQL views — down to the same SQL.
A Terraform provider for the platform's own resources — warehouses, namespaces, grants, dbt projects, contracts, ontologies — managed in HCL, in a repo, in a pull request, with a plan you can read before you apply.
Every managed data platform asks you to put your data, identities, and queries in the vendor's plane. This one doesn't. Data, identity, catalog, compute, and policy — all five in infrastructure you control, by construction.
An honest read of where this platform sits against Databricks, Snowflake, and iomete — on the one axis that decides the choice: how much control you keep versus how much operation you hand off. Including where we lose.
The map of the platform: an assembly of open-source parts behind a Backend-for-Frontend that holds your token and enforces your identity at every hop. What each surface does, how a request flows, and where the cliffs are.
We tested Renovate's registry behaviour for every ecosystem — npm, PyPI, Go, Cargo, Maven, NuGet, Composer — and found that minimumReleaseAge works for all of them, but how each datasource resolves its registry is surprisingly different. Here's what we found and how to build a complete defence.
A three-layer defense for developers and DevOps engineers: harden your dev machine in 5 minutes, lock down project configs for every ecosystem, and enforce policy org-wide with a proxy. Covers npm, pnpm, yarn, bun, pip, uv, Go, Maven, Gradle, NuGet, Cargo, and Composer — plus escrow, JFrog Curation, and Nexus.
DuckDB 1.5 ships a wire protocol called Quack. We re-implemented it in pure Rust, turned SQE into both server and client, and proved you can JOIN an Iceberg table with a remote DuckDB table in a single SELECT. Two waves of bugs, one federated query.
We started with a markdown table tracking which DuckDB types we could round-trip. The table became the roadmap. DECIMAL, LIST, STRUCT, MAP, ARRAY, ENUM, UNION each got their own MR. The surprises came from how DuckDB models the relationships: MAP is LIST<STRUCT>, UNION is STRUCT with a tag field, and DECIMAL packs four widths into one logical type.
Ten sub-MRs in a day. The wire format, the fixture-driven debugging loop, and two bugs that the C++ reference encoder ships without telling you: WriteListWithDefault elision, and uninitialised bytes at NULL VARCHAR positions.
Every time a developer runs npm install, they're implicitly trusting hundreds of strangers. Intercepting that trust is straightforward in theory — and surprisingly subtle in practice.
MR #220 wired runtime filters into iceberg-rust's scan path and dropped TPC-DS 67%. SSB looked like it regressed 6%. Two failed heuristic attempts, one parquet-trace session, and ten warm passes later, the regression turned out to be measurement noise. The fix-the-fix that wasn't, and the data-clustering insight that explains why two suites with the same code path behave nothing alike.
TPC-DS q72 sat at 10 seconds while every other query ran in under 1.4. Five days of investigation chased scan parallelism, range-based NDV, iceberg-rust upgrades, and the RisingWave fork. None of those were the bug. The bug was a silently-skipped Err arm in our dynamic-filter evaluator that swallowed every Int32 vs Int64 type clash. Fixing it: 15.5s to 0.77s. q72 now beats Trino.
Two days after the nine-PR audit pass, we ran a bigger one. 130 issues filed, 19 themed MRs merged across four waves. Same workflow, more failure modes. Watchdog stalls, a reboot mid-wave, a broken main, and config.rs as the conflict magnet. Here is what actually happened.
Supply chain attacks on package registries are surging across every ecosystem. Two controls block the majority: a quarantine window and disabled install scripts. The exact config for npm, pip, Go, Maven, NuGet, Cargo, and more — plus a one-command proxy and shell tests you can run today.
Sunday afternoon: an audit dropped eighteen issues into the tracker. Monday morning we had nine merge requests open. Two of them hit conflicts on rebase. Neither one mattered. Here is why that was deliberate, not lucky.
Modern object-store abstractions unify the filesystem and HTTP behind a single URL. That's a feature for ergonomics. It's a security trap when the URL comes from a user. SQE shipped that trap and then closed it. The IMDS pivot is the part worth telling.
SQE now ships DuckDB-style ATTACH / DETACH and CREATE / DROP SECRET. The story of building it covers parser extension, credential hygiene, a lifecycle bug we found in the integration tests, and a state-store pattern that is starting to repeat.
SQE now emits OL 2-0-2 events with column-level lineage on every write. Here is what they look like, why we walked the LogicalPlan to build them, and the disk spool we did not want to write.
SQE started as a distributed Iceberg query engine. Five MRs later it queries CSVs from disk, Parquet from S3, and Parquet from HuggingFace. We did not plan that.
We built SQE for distributed Iceberg, but most of the time you just want to look at a parquet file. Here's how we made the engine work both ways without forking the codebase.
We claimed the engine was catalog-agnostic. Time to prove it. One branch, five live integration tests, one small AWS SigV4 patch, and a matrix score that moved from 153 to 158.
Sixty-three capabilities, three levels, no marketing. The Iceberg Matrix is what compatibility looks like when the rubric is public and the evidence has to land in code. Here is why it works, what we learned from sitting on it, and why every open standard needs one.
We thought the V3 path worked. The unit tests said it worked. The matrix called it 'partial' and we agreed. Then we wrote eleven end-to-end tests and discovered Polaris had been silently rejecting every V3 column type for months.
An admin page that generates TPC benchmark datasets on-demand. Pick a benchmark, pick a scale, click Generate — sqe-bench does the heavy lifting in a Docker container while SQE loads the results into Iceberg tables under your identity.
We ran a full production sign-off audit against SQE and found 43 issues across security, runtime safety, logic bugs, and code quality. Then we fixed all of them in one session.
A companion piece to The Human Side of the Machine Shift — exploring why AI disruption hits different minds differently, and how leadership can turn discomfort into team strength.
A technical analysis of AWS's latest database innovations, from Aurora's Graviton4 support to OpenSearch Serverless improvements. Learn how these changes are reshaping data architecture patterns and enabling more efficient, distributed database systems with real-world implementation strategies.
Explore how AWS's new zonal shift capabilities and Security Incident Response features are transforming traditional disaster recovery. This technical deep-dive reveals how automatic failover, self-healing systems, and integrated security responses are making DR more reliable and cost-effective than ever before.
An architect's perspective on AWS's latest customer experience innovations, including Amazon Connect's AI capabilities and Polly's new synthetic voices. Discover how these technologies are making digital interactions more natural while improving efficiency and customer satisfaction.
Discover how AWS's latest infrastructure updates, particularly the expansion of Graviton4 processors and R8g instances, are quietly revolutionizing cloud computing sustainability. Learn about real-world performance improvements, cost reductions, and environmental benefits of these strategic changes.