Narcis Miclaus

9 min read

Data & System Architecture, from the ground up Lesson 37 / 80

Lakehouses: Delta, Iceberg, Hudi

ACID transactions on object storage. The format wars of 2023-2025 and where the industry landed in 2026.

The previous lesson described how data is organised inside a lake (bronze, silver, gold). This lesson is about what the lake is actually made of in 2026. Specifically: what file format do those tables use, and why do we use anything other than plain Parquet on object storage?

The short answer is lakehouse table formats: Delta Lake, Apache Iceberg, and Apache Hudi. They are the layer that turns a folder of Parquet files on S3 into something that behaves like a database table, with ACID transactions, schema evolution, time travel, and concurrent reads and writes. They are the technology that made the “lakehouse” word stop being a marketing term and start being a real architectural pattern.

The three formats spent the years between 2019 and 2025 in a competitive scrum, with each vendor pushing one as the future. By 2026 the dust has mostly settled, and this lesson tells the story of what each one is, what they have in common, and what the 2026 picture looks like for someone choosing between them today.

Why plain Parquet on S3 is not enough

Parquet is a columnar file format. It is fast, compressed, and well-supported by every analytics engine. The instinct, when you first build a data lake, is to write Parquet files into S3 and read them back with Spark or Trino or DuckDB. For a single-writer, append-only workload this works fine.

The trouble starts the moment you have more than one writer, or any update that is not a clean append.

  • No transactions. If two jobs write to the same partition at the same time, they overwrite each other’s files. There is no atomic “add these new files and remove those old ones” operation.
  • No atomic partition replace. The “overwrite this partition” pattern (lesson 38) requires deleting old files and writing new ones. If a reader queries during the window between the delete and the write, they see a partial view of the data, or no data at all.
  • No row-level deletes. GDPR asks you to delete a specific user’s records. With raw Parquet you have to find every file that contains their rows, rewrite each one without the deleted rows, and swap them in. There is no built-in mechanism for any of this.
  • No schema evolution. Adding a column means rewriting every file in the table, or accepting a schema-on-read mess where some files have the column and others do not.
  • No time travel. “Show me the table as it was last Tuesday” is a question that raw Parquet has no answer to, unless you have happened to keep snapshots.
  • No statistics that update. Each Parquet file has its own statistics, but there is no table-level metadata layer that the query engine can use to skip files cleanly.

Lakehouse table formats are the layer that fixes all of this. They sit on top of Parquet files (the actual data is still Parquet), and they add a metadata layer that records which files belong to the table at which point in time. Reads consult the metadata to know which files to scan. Writes update the metadata atomically. Two writers do not corrupt each other; they queue up at the metadata layer and serialise.

That is the whole idea. The differences between Delta, Iceberg, and Hudi are about how the metadata layer is structured and what features it exposes.

Delta Lake

Delta Lake was open-sourced by Databricks in 2019. The metadata layer is a transaction log: a directory of JSON files (with periodic Parquet checkpoints) recording every commit to the table. Each commit is an atomic operation that adds and removes files. Readers consult the log to compute the current set of files; writers append to the log to make changes.

Delta’s strengths:

  • Tight Spark integration. Delta was designed alongside Spark Structured Streaming, and the streaming-to-Delta path is the most polished of the three formats. If your ingestion is “Kafka into a bronze table via Structured Streaming”, Delta is the easiest path.
  • MERGE syntax. MERGE INTO ... USING ... ON ... was the first to ship in Delta and is still the most ergonomic implementation. CDC-style upserts are first-class.
  • Vacuum and Z-ORDER. Delta has solid implementations of file compaction (OPTIMIZE) and multi-dimensional clustering (ZORDER BY). For Spark workloads, the maintenance story is good.
  • Backed by Databricks. The platform you are most likely to be running Delta on has the deepest possible integration. On Databricks, “use Delta” is the path of least resistance and almost always the right answer.

Delta’s historical weakness was that the open-source version lagged the Databricks version, and the engine support outside Spark was thinner than Iceberg’s. Both of these have improved through 2024 and 2025: the open-source version is closer to feature parity, and Delta UniForm is a compatibility layer that exposes Delta tables as Iceberg-readable, partly closing the cross-engine gap.

Apache Iceberg

Apache Iceberg came out of Netflix in 2018 and was donated to the Apache Foundation. It was designed from the start for query-engine independence. The metadata layer is more elaborate than Delta’s: a tree of metadata files (table metadata, manifest lists, manifests) that records the table’s snapshots, schemas, partition specs, and file lists. The structure is more verbose than Delta’s log, and that verbosity is the price of supporting a wider range of operations cleanly.

Iceberg’s strengths:

  • Engine neutrality. Iceberg is a first-class citizen in Spark, Trino, Flink, Presto, Dremio, Snowflake (since 2024), BigQuery, AWS Glue and Athena, and DuckDB. If you want to read the same table from multiple engines, Iceberg has by far the best story.
  • Hidden partitioning. Iceberg lets you declare partitioning as transforms on columns (days(event_time), bucket(16, user_id)) without exposing them in the user-facing schema. Queries do not need to know about the partition columns; the engine pushes filters down through the transforms automatically. This is one of the biggest ergonomic wins over Delta.
  • Schema and partition evolution. Iceberg can evolve the partition spec without rewriting old data. Old files keep their old layout, new files use the new layout, and reads work across the boundary. Delta cannot do this without rewriting.
  • Strong governance story. The catalog ecosystem (Nessie, Polaris, the AWS Glue catalog, Unity Catalog’s Iceberg support) has matured around Iceberg. Multi-engine governance is where Iceberg is genuinely the leader.

Iceberg’s historical weakness was that the streaming write story lagged Delta’s, and small-file management required more manual care. Both have improved.

Apache Hudi

Apache Hudi came out of Uber in 2017, the earliest of the three. Its design centre is upserts and incremental processing. Hudi is the format that was built around “I have a CDC stream from a database and I want a queryable table that mirrors it, with updates applied row-by-row”.

Hudi’s strengths:

  • Upsert-first. Hudi has two table types, Copy-on-Write and Merge-on-Read. Merge-on-Read maintains a base file plus a row-level update log, with periodic compaction. For workloads with high update frequency on individual records (CDC mirrors, slowly-changing dimensions), this is more efficient than rewriting whole files.
  • Incremental queries. Hudi has first-class support for “give me the rows that changed since timestamp T”, which is the natural primitive for downstream pipelines that want to consume only the deltas.
  • Streaming ingestion tools. The Hudi Streamer (formerly DeltaStreamer) is a packaged streaming ingestion tool for common sources (Kafka, JDBC CDC), with built-in deduplication and upsert semantics.

Hudi’s weakness, in 2026, is that the use cases it shines at are also the cases where Iceberg has caught up substantially, and outside those cases the broader ecosystem support is thinner. Hudi is the right pick for streaming-upsert-heavy workloads, especially when you already know that pattern dominates. It is rarely the right pick as a general-purpose default.

What they have in common

For the high-level architectural decision, the three formats are more similar than different. All three give you:

  • ACID transactions on object storage. Concurrent writes are serialised through the metadata layer. Readers see a consistent snapshot.
  • Schema evolution. Add a column. Rename a column. Change a column’s type within compatible families. The metadata layer tracks the history.
  • Time travel. Query the table as it was at timestamp T or version V. The metadata keeps old snapshots until they are explicitly expired.
  • MERGE / upsert semantics. Insert new rows, update existing rows, delete matched rows, all atomically.
  • Concurrent reads and writes. Long-running queries see a stable snapshot while writes proceed.
  • Row-level deletes. GDPR-friendly. Issue a DELETE FROM table WHERE user_id = ? and the format handles the file rewrites.

If you are choosing between them and any of these features tips the scale, you have probably misread the situation. The features are table stakes across all three formats now. The decision turns on ecosystem fit, not on capability.

The format wars and where they landed

Between 2022 and 2025, each vendor with a stake in the lakehouse pushed its preferred format. Databricks pushed Delta. Snowflake, AWS, and Confluent pushed Iceberg. Onehouse pushed Hudi. Customers ran proofs of concept on all three, wrote think-pieces, and wondered whether the format would still exist in five years.

By 2026 the picture has clarified, mostly in Iceberg’s favour. The pivotal moves were Snowflake’s deep Iceberg integration in 2024 (you can now read and write Iceberg tables directly from Snowflake), AWS Glue and Athena making Iceberg the default-friendly format, and BigQuery exposing Iceberg as a fully supported external table type. The combination meant that an Iceberg table could be queried from Spark, Snowflake, BigQuery, Trino, and AWS-native tools without anyone having to rewrite the data. That cross-engine story is the thing the original lake people wanted in 2018, and Iceberg is the format that delivered it.

Databricks has responded with Delta UniForm, which writes Delta tables in a way that exposes them as Iceberg-readable to other engines. The bet is that “Delta on Databricks, readable as Iceberg from outside” is enough for the cross-engine use case without abandoning Delta’s first-party features. As of 2026 it works well enough for most read scenarios, less smoothly for writes, and the trajectory is positive.

Hudi remains the best pick for the streaming-upsert workload it was built for, and a smaller ecosystem otherwise.

flowchart LR
    K[Kafka, CDC, files] --> I[Bronze: Iceberg or Delta]
    I --> SP[Spark transforms]
    SP --> S[Silver: Iceberg or Delta]
    S --> DBT[dbt models]
    DBT --> G[Gold: Iceberg or Delta]
    G --> Q1[Spark]
    G --> Q2[Trino]
    G --> Q3[Snowflake]
    G --> Q4[BigQuery]
    G --> Q5[DuckDB]

Where each fits in 2026

A practical rubric for 2026:

  • You are on Databricks. Use Delta. The integration is excellent, every Databricks feature assumes Delta, and UniForm gives you Iceberg-flavoured external readability when you need it.
  • You are starting a new lakehouse, multi-engine or vendor-neutral. Use Iceberg. It is the safest default, the broadest engine support, and the best long-term governance story. The “default of defaults” pick.
  • Your dominant workload is high-frequency CDC mirrors with lots of upserts. Look hard at Hudi. Run a benchmark on your actual workload before committing.
  • You already have Delta or Iceberg working. Do not switch for the sake of being on the “winning” format. The migrations are real engineering work and the differences are not large enough to justify them unless a specific feature is blocking you.

The PySpark course, in module 8, has a hands-on lesson that goes through Delta, Iceberg, and Hudi with code: setting up tables, performing MERGE, doing time travel, comparing the metadata layouts. If you want to work with the formats rather than just decide between them, that lesson is the next stop after this one.

The next lesson is about a property all three formats make easier than ever: making batch jobs idempotent.

Search