A live run against Databricks-hosted Unity Catalog: each engine writes Iceberg tables the other reads, both run DELETE, UPDATE and MERGE on the other's tables, and 25 reads are compared value by value. All 25 match in every SQE mode. The distributed run found a real bug first: workers ignored delete files, so rows Databricks deleted with a deletion vector came back.
SQE stores JSON as the Iceberg V3 variant type. VARIANT and, on a V3 table, JSON columns hold a binary-encoded value instead of a string, invalid JSON fails the write, and the json functions read both shapes. New tables are V3 by default. Spark 4.1 reads SQE's variants, SQE reads Spark's, and since this week that includes files Spark wrote shredded. SQE still writes unshredded variants and has no typed field access on shredded columns yet.