Why architecture matters here
Iceberg matters because it brings database-like table semantics to the data lake, solving the fundamental problems of the old Hive-tables-on-directories approach and enabling the lakehouse. The data lake (files on object storage) is cheap and scalable, but without proper table semantics it's problematic: no atomicity (partial writes visible -- inconsistent reads), no safe schema evolution (changing the schema risks breaking readers), painful partition management (partitioning baked into the directory structure and queries -- changing it means rewriting), no consistent reads during writes (readers see partial state), and the metastore as a bottleneck (listing directories, tracking partitions). Iceberg solves all of these: ACID atomicity (writes all-or-nothing, isolated), safe schema evolution (columns add/drop/rename safely), hidden partitioning (partition managed by the table, changeable without rewriting or query changes), consistent snapshots (readers see a consistent snapshot), and efficient metadata (manifest-based, no directory listing). This transforms the data lake into a lakehouse (a data lake with database semantics -- reliable, evolvable, performant). For any organization building on a data lake (most modern data platforms), a table format like Iceberg is increasingly essential (bringing the semantics that make the lake reliable and usable), and understanding Iceberg is understanding the modern lakehouse foundation.
The layered-metadata-plus-snapshots design is the architectural core, and it's what provides the ACID, time travel, and efficient metadata. Iceberg's metadata is layered: a catalog (pointing to the current table metadata file -- the atomic commit point), the table metadata (pointing to the current snapshot), a snapshot (pointing to manifest lists), manifests (listing data files with their metadata -- stats, partition values), and the data files themselves (Parquet/ORC). This structure provides several things. ACID via atomic metadata swap: a commit atomically swaps the catalog's pointer to a new table metadata (a new snapshot) -- so the write is atomic (readers see either the old snapshot or the new, never partial -- the pointer swap is the atomic commit). Immutable snapshots: each snapshot is immutable (a consistent version of the table) -- so readers get a consistent snapshot (isolated from concurrent writes -- they read a fixed snapshot), and old snapshots are retained (enabling time travel -- querying a past snapshot). Efficient metadata: the manifests list files with their metadata (stats, partition values) -- so query planning reads the manifests (metadata) to prune files (skip files not matching the query -- metadata pruning) without listing directories (the old bottleneck) -- fast planning. This layered structure (catalog -> metadata -> snapshot -> manifests -> data files, with atomic pointer swaps and immutable snapshots) is the elegant core providing ACID (atomic swap), isolation and time travel (immutable snapshots), and efficient metadata (manifest pruning) -- understanding it is understanding how Iceberg provides its guarantees.
And the engine-agnostic-plus-hidden-partitioning design is what makes Iceberg an open, usable format. Engine-agnostic: Iceberg is an open specification (not tied to one engine) -- so multiple engines (Spark, Flink, Trino, Hive, and others) can all read and write the same Iceberg tables (a shared, interoperable format). This is a major advantage (the data isn't locked to one engine -- different engines for different workloads, all on the same tables) and a reason for Iceberg's adoption (an open format for the lakehouse). Hidden partitioning: unlike the old Hive approach (where partitioning is baked into the directory structure and queries must reference partition columns -- and changing partitioning requires rewriting), Iceberg's hidden partitioning manages the partitioning internally (the table knows its partitioning -- queries don't need to reference partition columns, and Iceberg applies the partitioning transparently -- so a query on a timestamp is automatically pruned to the relevant partitions without the query mentioning partitions), and partitioning can be changed (partition evolution -- changing the partitioning without rewriting the data, since it's managed in metadata). This -- engine-agnostic (open, shared) and hidden partitioning (transparent, evolvable) -- makes Iceberg an open, usable, evolvable format (versus the engine-locked, rigid old approach), and understanding these advantages is understanding why Iceberg is transforming the lakehouse.
The architecture: every piece explained
Top row: metadata and semantics. Metadata layers: the layered structure -- catalog (pointing to current metadata -- the atomic commit point), table metadata (pointing to the current snapshot), manifests (listing data files with metadata), data files (Parquet/ORC). Snapshots: immutable table versions (each snapshot a consistent version) -- providing isolation (readers see a fixed snapshot) and time travel (past snapshots retained). Hidden partitioning: partitioning managed by the table (transparent to queries -- no partition columns needed in queries, Iceberg prunes automatically) and evolvable (partition evolution -- change partitioning without rewriting). Schema evolution: safe schema changes (add/drop/rename columns -- via column IDs, so changes don't break existing data or readers) -- reliable evolution.
Middle row: transactions and pruning. ACID transactions: atomic commits (a write atomically swaps the metadata pointer -- all-or-nothing), isolation (readers see a consistent snapshot, isolated from concurrent writes), and consistency -- database-like transactions on the lake. Time travel: querying past snapshots (the immutable snapshots retained -- query the table as of a past snapshot/timestamp) -- for auditing, reproducibility, recovery. Metadata pruning: query planning reads the manifests (file metadata -- stats, partition values) to skip files not matching the query (manifest-level file pruning) -- fast planning without directory listing (efficient, versus the old metastore bottleneck). Row-level deletes: efficient updates/deletes -- position deletes (marking specific rows in files as deleted) and equality deletes (deleting rows matching a condition) -- enabling row-level changes without rewriting whole files.
Bottom rows: openness and maintenance. Engine-agnostic: an open format read/written by multiple engines (Spark, Flink, Trino, Hive, etc.) -- the same Iceberg tables shared across engines (interoperable, not engine-locked). Compaction and maintenance: maintenance operations -- compaction (merging small files into larger ones -- addressing the small-file problem from streaming/frequent writes), expiring old snapshots (removing old snapshots and their unreferenced files -- bounding metadata and storage growth), and rewriting manifests (optimizing the metadata) -- the ongoing maintenance. The ops strip: metadata growth (the metadata -- snapshots, manifests -- growing with commits; expiring old snapshots and maintaining manifests to bound it), maintenance (compaction, snapshot expiry, manifest rewriting -- the regular maintenance keeping the table performant and bounded), and catalog choice (the catalog -- Hive metastore, AWS Glue, a REST catalog, etc. -- the atomic commit point and table registry, chosen for the environment).
End-to-end flow
Trace Iceberg's ACID and time travel. A table is updated (new data appended). The write: the new data files are written, a new manifest (listing them) is created, a new snapshot (including the new manifest plus the existing ones) is created, and the commit atomically swaps the catalog's pointer to the new table metadata (the new snapshot) -- an atomic commit (readers see either the old snapshot or the new, never partial -- the pointer swap is atomic). Concurrent readers: a reader that started before the commit sees the old snapshot (isolated -- it reads the fixed old snapshot, unaffected by the concurrent write); a reader after the commit sees the new snapshot -- consistent isolation (each reader sees a consistent snapshot). Time travel: to query the table as it was yesterday, the query specifies a past snapshot (or timestamp) -- Iceberg reads that immutable snapshot (retained) -- querying the past table version (for auditing -- what did the data look like then, reproducibility -- rerun on the same snapshot, or recovery -- revert to a past snapshot). The ACID (atomic commit, isolation) and time travel (immutable snapshots) come from the layered metadata and immutable snapshots -- database-like semantics on the lake.
The hidden-partitioning and schema-evolution vignettes show the usability. A hidden-partitioning case: a table is partitioned by day (hidden partitioning -- Iceberg manages it). A query filtering on a timestamp is automatically pruned to the relevant day partitions -- WITHOUT the query referencing partition columns (Iceberg applies the partitioning transparently, deriving the day from the timestamp) -- so the query is simple (just a timestamp filter) and efficient (pruned). And when the team wants to change the partitioning (e.g., from day to hour for finer granularity), they change it (partition evolution -- Iceberg changes the partitioning in metadata for new data, without rewriting the existing data) -- versus the old approach (changing partitioning meant rewriting everything). A schema-evolution case: the team adds a column (safe schema evolution -- Iceberg uses column IDs, so the new column is added cleanly, existing data reads with the new column as null, no breakage), and later renames a column (also safe -- the column ID unchanged, just the name) -- reliable schema evolution (versus the old approach's fragility).
The engine-agnostic and maintenance vignettes complete it. An engine-agnostic case: the team uses Spark for batch writes, Flink for streaming writes, and Trino for interactive queries -- all on the same Iceberg tables (the open format read/written by all three) -- the interoperability (different engines for different workloads, one shared table format) that Iceberg's openness provides. A maintenance case: frequent writes (streaming) create small files (the small-file problem) -- so the team runs compaction (merging small files into larger ones) regularly. And commits accumulate snapshots (metadata growth) -- so the team expires old snapshots (removing old snapshots and their unreferenced files -- bounding metadata and storage growth). The maintenance (compaction, snapshot expiry) keeps the table performant (larger files) and bounded (metadata not growing unboundedly). The consolidated discipline the team documents: use Iceberg (or a lakehouse table format) for database semantics on the lake (ACID, snapshots, schema evolution, hidden partitioning -- solving the old Hive-tables problems), leverage ACID (atomic commits, isolation) and time travel (past snapshots -- auditing, reproducibility, recovery), use hidden partitioning (transparent, evolvable) and safe schema evolution, exploit metadata pruning (fast planning via manifests) and row-level deletes (efficient updates), use the engine-agnostic openness (multiple engines on shared tables), run maintenance (compaction for small files, snapshot expiry for metadata/storage growth), and choose the catalog for the environment -- because Iceberg brings proper table semantics to the data lake, transforming it into a reliable, evolvable, performant, open lakehouse, solving the fundamental problems (atomicity, schema evolution, partitioning, metadata) of the old direct-on-directories approach.