Why are there still many small files after optimizing?

[wangtaohz]

We have encountered situations where there are still many small files after optimizing. There are various reasons for this, and this discussion will help you identify your scenario and provide appropriate solutions.

[wangtaohz]

Case1: these small files are non-expired files

[wangtaohz]

If you are observing a lot of small files in the file system rather than in Iceberg metadata, you should enable the "snapshots expire" feature for the table, which will likely delete these small files when the snapshot expires.

By default, AMS has enabled the "snapshots expire" feature for the table, which retains the latest 12 hours of snapshots and runs every hour.

[wangtaohz]

Case2: these small files are orphan files

[wangtaohz]

If you are confident that the snapshots have expired but the files have not been deleted, they may be orphan files.

You can enable the orphan files cleaning feature to clean up these files. Once enabled, AMS will perform an immediate cleanup and then clean up every seven days. Please note that files modified within the last two days will not be deleted, but you can change this behavior.

[wangtaohz]

Case3: these small files are delete-files

[wangtaohz]

There are some data-files that have not participated in optimizing, and these files have a very small sequence number, which prevents Iceberg from automatically cleaning up delete-files with larger sequence numbers.

Arctic is going to clean these delete-files periodically.

The related issue #1628

[wangtaohz]

Case4: these small files belong to partitions that are not optimized

[wangtaohz]

This is an edge case that some partitions may not have participated in optimizing because Arctic supports optimizing only certain partitions.

You can confirm whether a partition has participated in optimizing with the following SQL query.

select distinct(partition_data) from task_runtime where process_id in (select process_id from table_optimizing_process where 1=1);