🧊Apache Iceberg

Why Iceberg exists

For a decade the default way to keep a big analytic table on a data lake was the Hive convention: a table is a set of directories under a storage prefix, and partition values are encoded in the directory paths (/event_date=2026-06-01/…). That worked on HDFS and became the lingua franca of the lake. It also has a set of structural problems that get worse the larger the table and the more it lives on cloud object storage.

• Planning requires listing storage. To know which files exist, an engine has to LIST directories — slow, paginated, rate-limited, and historically only eventually consistent on S3/GCS. A table with millions of files plans slowly and can momentarily observe a half-written partition.
• No atomic commit across partitions. Writing several partitions, or replacing data, is multiple storage operations with no transaction around them. Two writers racing can clobber each other; a reader can see a partial result.
• Schema is fragile. Columns are matched by name or ordinal position, so renaming or reordering a column can silently misalign reads.
• Partitioning is welded to physical layout. The partition column is a real column you must store and must remember to filter on; forget it and you full-scan, change your mind and you rewrite the whole table.

Iceberg replaces "a table is whatever files are under this prefix" with "a table is the exact list of files named by this metadata tree." The current set of files is enumerated explicitly in metadata, so query planning never lists storage. A commit is atomic because it swaps a single pointer. Schema, partitioning, and sort order are tracked as versioned metadata keyed by immutable column IDs, so they evolve without touching data. The result, in the project's own words, is a high-performance format for huge analytic tables that lets many engines — Spark, Flink, Trino, Dremio, Snowflake, BigQuery, and more — safely work on the same tables at the same time.

The metadata tree

An Iceberg table is a layered tree of files. Reading top-down, a query resolves the catalog pointer to one metadata file, opens the current snapshot's manifest list, prunes to a handful of manifests, prunes those to a handful of data files, and only then touches Parquet. Writing bottom-up, an engine writes new data files, then manifests describing them, then a manifest list, then a new metadata file, and finally swaps the pointer. Every layer below the catalog is immutable and written once.

Layer by layer, from the top:

• Catalog. A small transactional service that maps namespace.table to the location of its current metadata file. It is the one mutable thing in the whole design, and its one job is the atomic pointer swap. (Catalogs get their own section below.)
• Table metadata file (vN.metadata.json, JSON). The table's complete logical state: all current and historical schemas (with column IDs), the partition specs, the sort orders, table properties, the list of all valid snapshots, and which one is current. Each commit writes a brand-new metadata file; old ones are kept until expired.
• Manifest list (one per snapshot, Avro). Names the manifest files that make up that snapshot and, for each, stores partition-value ranges and file counts. This is the first pruning layer: an engine can skip a whole manifest whose partition range can't match.
• Manifest file (Avro). Lists actual data files and delete files, one row each, with rich per-file stats: record count, per-column lower/upper bounds and null/NaN counts, the file's partition tuple, and its sequence numbers. The second pruning layer.
• Data files (Parquet, ORC, or Avro). The rows themselves. Iceberg never rewrites them to change schema or partitioning. Alongside them, Puffin files hold deletion vectors and statistics blobs.

On disk a table is just two folders — a metadata/ tree and a data/ tree — in your bucket:

# s3://warehouse/db/events/
metadata/
  00002-9a1f…-metadata.json        ← current (the catalog points here)
  snap-5181947…-1-b3c0…​.avro       ← manifest list for the current snapshot
  8a21…-m0.avro                    ← manifest
  9f03…-m1.avro                    ← manifest
data/
  event_date=2026-06-01/0000-…​.parquet
  event_date=2026-06-01/0001-…​.parquet
  …

Snapshots, commits & ACID

A snapshot is the complete set of files that make up the table at one instant, captured by exactly one manifest list. Each snapshot records a snapshot-id (a unique long — not derived from the clock), a timestamp-ms, the parent snapshot id, a summary (the operation — append, overwrite, delete, replace — plus row/file metrics), and the manifest-list location. Snapshots chain into a lineage: a log of every committed state the table has ever had.

A commit is optimistic. There are no locks; instead:

1. The writer reads the current metadata (its base version), writes new data files, new manifests, and a new manifest list, and assembles a new metadata.json with its snapshot as current.
2. It asks the catalog to swap the pointer from base → new, conditional on base still being current — a compare-and-swap.
3. If another writer committed in the meantime, the CAS fails. The writer re-reads the new base, re-applies its changes on top, validates its assumptions still hold, and retries.

The atomic swap is what delivers serializable isolation: a reader loads the metadata once and sees a single consistent snapshot for the life of its query, unaffected by concurrent writes until it refreshes. Writers choose what to validate at commit time (for example, "no new files appeared in the partitions I deleted from"), which lets a given operation target either serializable or snapshot isolation.

The machinery that keeps deletes correct under all this concurrency is the sequence number. Every successful commit gets the next number; each data and delete file inherits the sequence number of the commit that added it. The rule: a delete file applies to a data file only when the data file's sequence number is ≤ the delete file's. So a delete committed at sequence 7 removes matching rows from data written at sequence ≤ 7 but never touches data appended later at sequence 8 — no accidental deletion of future inserts.

Every write is observable as a row in the table's snapshots metadata table:

INSERT INTO db.events VALUES (…);   -- commits a new snapshot

SELECT snapshot_id, committed_at, operation, summary['added-records']
FROM   db.events.snapshots          -- Iceberg metadata table
ORDER BY committed_at;

Time travel, branches & tags

Because old snapshots stay valid until they're explicitly expired, you can read the table as of any of them — by snapshot id or by timestamp:

SELECT * FROM db.events FOR VERSION AS OF 5181947822281…;
SELECT * FROM db.events FOR TIMESTAMP AS OF '2026-06-01 00:00:00';

Rolling back is a metadata-only change — it just repoints current-snapshot-id; no data moves:

CALL system.rollback_to_snapshot('db.events', 5181947822281…);

Branches and tags are named references to snapshots, stored in the table metadata:

• A tag is an immutable pointer to one snapshot, with its own retention — an audit or compliance marker you want protected from expiry.
• A branch is a mutable line of snapshots you can write to independently, then fast-forward into main.

Together they enable write-audit-publish: a job writes into a staging branch, a validation job audits the data there with ordinary time-travel reads, and only on success is the branch published (fast-forwarded) into main. Readers on main never see un-audited rows.

ALTER TABLE db.events CREATE TAG `eod-2026-06-01` RETAIN 90 DAYS;
ALTER TABLE db.events CREATE BRANCH staging;
-- write into the branch, audit it, then publish:
CALL system.fast_forward('db.events', 'main', 'staging');

Evolution without rewrites

The two changes that force a full table rewrite in a Hive-style lake — altering the schema and altering the partitioning — are both metadata-only in Iceberg. That property falls out of two design choices.

Schema evolution by column ID

Every column gets a unique, permanent integer field ID when it's added; names and positions are just labels over IDs. Data files tag their values by field ID, and readers match columns by ID, never by name or ordinal. That makes the full set of schema changes safe and metadata-only — no data rewrite:

• add a column (new ID; old files simply lack it, so reads return its default or null),
• drop a column (its ID is retired; the bytes are ignored on read),
• rename (the label changes, the ID does not),
• reorder (display order changes, the ID does not),
• widen a type along allowed promotions (int→long, float→double, decimal precision up),
• give a column a default value (initial-default applies to existing rows, write-default to new writes).

ALTER TABLE db.events ADD COLUMN region STRING DEFAULT 'unknown';
ALTER TABLE db.events RENAME COLUMN region TO geo_region;   -- safe: ID unchanged
ALTER TABLE db.events ALTER COLUMN id TYPE BIGINT;            -- widen int → long

Hidden partitioning & partition evolution

You partition by a transform of a column, not by a separate hand-maintained column. Because Iceberg computes and stores the partition value itself, queries filter on the source column and the engine derives the partition predicate automatically — there is no shadow partition column to name, and no way to "forget" the partition filter and accidentally full-scan:

CREATE TABLE db.events (
    id        BIGINT,
    event_ts  TIMESTAMP,
    payload   STRING
) USING iceberg
PARTITIONED BY (days(event_ts), bucket(16, id));

SELECT count(*) FROM db.events
WHERE event_ts >= '2026-06-01';   -- pruned to days(event_ts) partitions; no partition column named

Partition evolution changes the spec later — say from days to hours — and records it as a new partition spec. Existing data keeps its old spec and is not rewritten; new data uses the new spec; and a single query reads across both, because every data file records which spec wrote it. The same is true of sort orders, which Iceberg tracks and engines honor when clustering data within files.

Row-level deletes & updates

Iceberg is append-mostly but supports full UPDATE, DELETE, and MERGE. Each table (and each operation) picks one of two strategies via table properties (write.update.mode, write.delete.mode, write.merge.mode):

• Copy-on-write rewrites every data file that contains an affected row — minus the deletes, plus the updates — and commits the new files. Reads stay simple and fast (nothing to reconcile), but changing one row in a big file rewrites the whole file. Good for read-heavy tables with infrequent changes.
• Merge-on-read leaves data files in place and writes a compact delete record that the reader applies on the fly. Cheap writes, a little more work on read, reconciled later by compaction. Good for frequent, scattered changes — CDC, streaming upserts.

Merge-on-read deletes take two forms:

• Position deletes / deletion vectors. Mark specific rows of a specific data file as deleted by position. In the current spec this is a deletion vector: a compact bitmap — one bit per row — stored in a Puffin file, at most one per data file, that the reader ANDs against the file. Small, one-per-file, fast to apply; the preferred representation.
• Equality deletes. Remove every row matching a set of column values (e.g. id = 42) without knowing where those rows live — ideal for streaming upserts where the writer knows the key but not the file or position.

A MERGE upsert maps cleanly onto these: matched rows become deletes (a CoW rewrite, or a MoR delete record) plus new data for the updated values, and unmatched source rows become inserts.

ALTER TABLE db.events SET TBLPROPERTIES (
    'write.delete.mode' = 'merge-on-read',
    'write.update.mode' = 'merge-on-read',
    'write.merge.mode'  = 'merge-on-read'
);

MERGE INTO db.events t
USING staged_changes s ON t.id = s.id
WHEN MATCHED AND s.op = 'D' THEN DELETE
WHEN MATCHED               THEN UPDATE SET t.payload = s.payload
WHEN NOT MATCHED           THEN INSERT *;

Iceberg can also track row lineage — a stable _row_id and a _last_updated_sequence_number per row across snapshots — so an UPDATE preserves a row's identity and you can build change-data-capture consumers that see exactly what changed between two snapshots. And because MoR leaves deletes uncompacted, the read cost grows until compaction (next section) merges the deletes back into clean, delete-free data files.

How a query stays fast

The same tree that makes commits atomic makes scans cheap, because every layer carries the statistics needed to skip the next. A filtered query prunes in three passes before reading any Parquet data:

1. Manifest-list pruning. Each manifest entry in the manifest list carries partition-value ranges; the engine drops whole manifests whose ranges can't satisfy the query (wrong day, wrong bucket).
2. Manifest pruning. The surviving manifests are read; each data-file entry carries its partition tuple plus per-column lower/upper bounds and null counts; the engine drops data files that can't match the predicate.
3. Row-group / page pruning. For the few surviving Parquet files, the engine uses Parquet footer statistics (and any Puffin stats) to skip row groups and pages.

Crucially, there is no storage LIST anywhere in that path: the file set is read from manifests, not discovered by listing object storage. That, plus hidden partitioning deriving the partition predicate automatically, is the core planning-speed advantage over Hive — selective queries touch a tiny fraction of the table without anyone hand-writing partition filters.

Keeping tables healthy

Every write is a new immutable snapshot plus new files, so tables naturally accumulate small files, dead snapshots, delete files, and metadata over time. Iceberg ships maintenance actions — exposed as Spark stored procedures, with equivalents in other engines — to keep them in shape:

• rewrite_data_files — compaction. Bin-packs many small files into fewer large ones, optionally sorting or z-ordering rows for better clustering and pruning, and applies pending merge-on-read deletes so the output is clean. The single biggest lever for read performance.
• expire_snapshots. Drops snapshots older than a cutoff, and the data/manifest files only they referenced — bounding metadata growth and reclaiming storage. This is what eventually deletes the data you "deleted."
• remove_orphan_files. Deletes files in the table's storage that no live snapshot references — debris from failed commits or aborted jobs.
• rewrite_manifests. Re-buckets manifest entries (e.g. by partition) so manifest pruning stays effective as the table grows.

CALL system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort', sort_order => 'event_ts DESC');

CALL system.expire_snapshots('db.events', TIMESTAMP '2026-05-01 00:00:00');
CALL system.remove_orphan_files(table => 'db.events');

Catalogs & the REST standard

The catalog is the one stateful, transactional piece of the system: it maps namespace.table to the current metadata-file location and performs the atomic compare-and-swap that turns an upload of files into a committed table state. Point several engines at the same catalog and they all see the same tables with the same guarantees.

The Iceberg REST Catalog is the part that matters most today: an OpenAPI-defined HTTP protocol for catalog operations. Implement that one API and any compliant engine — Spark, Flink, Trino, Dremio, PyIceberg, DuckDB — connects without an engine-specific plugin. It also standardizes what used to be per-vendor: server-side commit and conflict handling, OAuth2 authentication, and credential vending — the catalog hands the engine short-lived, scoped storage credentials instead of long-lived cloud keys. Increasingly, "an Iceberg catalog" just means "a REST endpoint."

How companies use Iceberg

Iceberg's promise is that your tables outlive any one engine: the data is open Parquet in your own bucket, and the catalog is a standard API. In practice the platforms differ in which pieces they own — who runs the compute, who is the catalog, whose storage the data sits in, and whether they can write or only read.

Dremio

A lakehouse query-and-compute engine that treats Iceberg as its native table format. It offers full SQL DML (INSERT, UPDATE, DELETE, MERGE, COPY INTO) plus table-management commands — OPTIMIZE for compaction and VACUUM for snapshot and orphan cleanup. Its catalog is built on Apache Polaris (the Dremio Catalog) with Project Nessie-style git semantics: branches and tags across the whole catalog so you can isolate, validate, and merge data changes like code. The data stays in your S3/ADLS/GCS.

Snowflake

Exposes Iceberg tables with data in a customer-owned External Volume — your own S3/GCS/Azure bucket — rather than Snowflake's internal storage. Two flavors: Snowflake-managed, where Snowflake is the Iceberg catalog and you get full read/write, most platform features, and automatic maintenance; and externally-managed, where an outside catalog (AWS Glue or an Iceberg REST catalog) owns the table and Snowflake is read-mostly. Snowflake Open Catalog is a managed Apache Polaris service, and the Horizon Catalog exposes Snowflake-managed Iceberg tables over the Iceberg REST API so external engines can read and write them. Compute is Snowflake's virtual warehouses.

Databricks

Manages Iceberg through Unity Catalog. UC can host managed Iceberg tables with full Databricks features, and it exposes a Unity Catalog Iceberg REST Catalog endpoint so outside Iceberg engines can read and write those tables. It can also attach foreign Iceberg catalogs (e.g. Glue, Snowflake) for read-only federation, and UniForm lets a table written as Delta publish Iceberg metadata over the same Parquet — so Iceberg readers (Snowflake, BigQuery, Trino…) can consume it without a copy. Compute is Spark/Photon.

Beyond the three: AWS positions Iceberg as the way to build a transactional lake on S3, with read-write support across Athena, EMR (Spark), and Glue, plus querying from Redshift, all coordinated by the Glue Data Catalog. Google offers BigQuery tables for Apache Iceberg (managed, read-write, data in your GCS bucket, mutable via GoogleSQL DML) alongside read-only external BigLake Iceberg tables, with the BigLake metastore speaking the Iceberg REST API to Spark, Flink, and Trino.

Where Iceberg sits

Iceberg is one of three open table formats, alongside Delta Lake and Apache Hudi. All three add ACID transactions, time travel, and schema evolution over columnar files in object storage, and the practical differences are in metadata structure, delete handling, and ecosystem breadth. As AWS's and Google's "what is Apache Iceberg" overviews frame it, what distinguishes Iceberg is its hierarchical metadata tree — snapshot → manifest list → manifests → data files, with statistics at each level — which lets engines prune to the relevant files without listing storage, and the breadth of its engine-agnostic ecosystem coordinated through the open REST catalog.

That ecosystem is the reason most teams reach for it. Apache Spark is the reference engine, with the most complete DML and the stored procedures used throughout this doc; Flink covers streaming ingest; Trino/Presto and Dremio cover interactive SQL; PyIceberg gives Python/Arrow access with no JVM; and engines like DuckDB and ClickHouse read Iceberg for local or embedded analytics — all over the same tables in the same bucket.

Further reading

• Apache Iceberg — official site & docs. Concepts, the Spark/Flink procedure reference, and the configuration catalog. The place to verify defaults and exact syntax.
• The Iceberg table spec. The authoritative description of metadata, manifests, snapshots, sequence numbers, deletes, and the format versions. Ground truth for everything structural in this doc.
• apache/iceberg on GitHub. The reference Java implementation and the spec source, plus release notes for what shipped when.
• AWS — "What is Apache Iceberg?" The AWS-services view (Athena / EMR / Glue / Redshift) and an open-table-format overview.
• Google Cloud — "What is Apache Iceberg?" The BigQuery / BigLake view.
• Vendor docs: Dremio, Snowflake Iceberg tables, and Databricks Iceberg — the primary sources for the "how companies use Iceberg" section above.