Iceberg never deletes anything on its own. A table that takes a commit every five minutes has 17,280 snapshots after 60 days, and it keeps every one of them, plus every file they point at, until a job you wrote says otherwise.
I like Iceberg. Commits are atomic, so a reader never sees half a write. Snapshots give you time travel and a way to roll back a bad load. Renaming a column doesn’t mean rewriting the table. Those are real gains over a directory of Parquet files with a Hive partition scheme on top.
The cost is that the table format is a specification and a library, and nothing in it runs in the background. A warehouse merges small files, vacuums old versions and cleans up after failed loads where you can’t see it. With Iceberg on object storage, those jobs are yours. When nobody owns them, query planning slows down, the storage bill climbs, and after a couple of months the format takes the blame for a job that was never scheduled.
What piles up when nobody is assigned
There are four kinds of debris, and each has its own procedure.
Small files come first. Every streaming or micro-batch commit writes at least one file per partition it touches. Planning cost follows file count, because the engine reads a manifest entry for every file that could match.
Old snapshots are next. Each one keeps its data files alive. Compaction makes this worse before it makes it better: the rewritten files are new, and the small ones stay on disk because older snapshots still reference them.
Then manifests. Thousands of small commits leave thousands of small manifest files, which is planning overhead even after the data files are tidy.
Last, orphan files. A job that dies between writing data and committing leaves files no snapshot knows about. Expiry can’t remove them, because expiry only walks the metadata.
On format v2 tables with row-level deletes, delete files add a fifth pile. I’m leaving those out here, but the same rule applies: somebody has to rewrite them.
The schedule I’d pin to the table
The order matters more than the cadence. Compaction goes first because it creates the garbage that expiry collects. Orphan cleanup goes last and least often, because it lists the whole table location and is the only one of the four that can delete something a running writer needs.
| Order | Job | Cadence | Run early when |
|---|---|---|---|
| 1 | Compaction (rewrite_data_files) |
Daily, off-peak | Any partition holds over 500 data files under 32 MB |
| 2 | Snapshot expiry (expire_snapshots) |
Daily, after compaction | Snapshot count passes 3,000 |
| 3 | Manifest rewrite (rewrite_manifests) |
Weekly | Planning time on a reference query passes 5 s |
| 4 | Orphan cleanup (remove_orphan_files) |
Weekly, last | Bucket size passes 1.3 times the bytes the metadata accounts for |
Here are the calls, as Spark SQL with the Iceberg extensions enabled. lake is the catalog name. The timestamps are literals because the scheduler should compute them and template them in.
-- 1. Compaction: bin-pack small files up to a 256 MB target
CALL lake.system.rewrite_data_files(
table => 'telemetry.vehicle_events',
strategy => 'binpack',
options => map(
'target-file-size-bytes', '268435456',
'min-input-files', '5',
'partial-progress.enabled', 'true'
)
);
-- 2. Expiry: drop snapshots older than 7 days, always keep the last 10
CALL lake.system.expire_snapshots(
table => 'telemetry.vehicle_events',
older_than => TIMESTAMP '2024-03-16 00:00:00',
retain_last => 10
);
-- 3. Manifests: regroup them so planning reads fewer, larger ones
CALL lake.system.rewrite_manifests(
table => 'telemetry.vehicle_events'
);
-- 4. Orphans: delete unreferenced files older than 3 days
CALL lake.system.remove_orphan_files(
table => 'telemetry.vehicle_events',
older_than => TIMESTAMP '2024-03-20 00:00:00'
);
Two cautions. The older_than on orphan cleanup has to be longer than your longest write job, with margin. A file written by a job that hasn’t committed yet looks exactly like an orphan. Three days is the default and I wouldn’t go below it. Run the procedure with dry_run => true the first time and read the list before you let it delete anything.
The second: seven days of snapshots is a decision about how far back you can time travel or roll back. Pick the number with whoever would ask for the rollback, and write it down next to the job.
One query that tells you whether it’s working
Iceberg exposes its own metadata as tables, so the health check is plain SQL against files.
SELECT
`partition`,
count(*) AS data_files,
sum(CASE WHEN file_size_in_bytes < 33554432 THEN 1 ELSE 0 END) AS files_under_32mb,
round(avg(file_size_in_bytes) / 1048576, 1) AS avg_file_mb
FROM lake.telemetry.vehicle_events.files
WHERE content = 0 -- data files only, not delete files
GROUP BY `partition`
ORDER BY files_under_32mb DESC;
Run it after compaction and store the result. A partition that stays at the top for three days running means compaction is failing or skipping it. Add SELECT count(*) FROM lake.telemetry.vehicle_events.snapshots for the expiry side, and you have the two numbers behind the first two thresholds in the table.
Sixty days at an invented fleet company
To put numbers on it, picture a fleet-tracking company I made up, Harrier Telematics. Its vehicle-event table has 46 partitions and the ingest job commits every five minutes. That’s 288 commits a day, each writing one file per partition: about 13,000 new files daily.
Nobody set up maintenance. After 60 days the table holds 795,000 data files averaging 3 MB. Planning a simple query takes 41 seconds before a single row is read. The bucket is at 6.1 TB even though the live data is about 2.4 TB. The difference is files replaced by late corrections and a few ad hoc rewrites, all still pinned by snapshots that were never expired, plus leftovers from failed runs.
Then the schedule goes in: daily compaction to a 256 MB target, seven-day expiry, weekly orphan cleanup. Once the first week’s snapshots age out, the table sits at about 9,400 files after each nightly run, planning takes 1.8 seconds, and the bucket settles near 2.7 TB. That’s the live data plus a week of superseded small files, which is what seven days of rollback costs.
Same data, same format. The only new thing is that the jobs have an owner.
Before you pick the engine
Some managed offerings take part of this list off your hands. Snowflake’s Iceberg tables, in public preview since December, are one example for tables Snowflake manages. That’s worth weighing, and it’s also still a preview. If you run open-source Iceberg with Spark against your own bucket, assume none of it is done for you.
So my test for whether a team is ready to put Iceberg in production is short. Ask who gets paged when the small-file count for a partition crosses the threshold. If there’s a name, go ahead. If the answer is a shrug, schedule the four jobs first and migrate the table second.