Still have many small files in the arctic table after optimizing

[XBaith]

Hi team, I run flink job and sink data into arctic(do a checkpoint every 30 minutes), but I find that still have many small files even though have optimized. How could I reduce small files? For example: one partition has only one base files(with some pos-delete files) that size is smaller than 256M.
I changed the table properties, but it doesn't seem to be working as I expected.

optimize.major.trigger.max-interval	3600000
optimize.minor.trigger.max-interval	180000

[XBaith]

[XBaith]

They both contain small files

[wangtaohz]

It's reasonable that there are only 28 files (of current snapshot) in base store. If you still see many files in your file system, these files most likely belong to old snapshots which have are not expired.

Arctic base store keeps snapshots in latest 12 hours by default, to keep fewer snapshots, you can change the table property snapshot.base.keep.minutes(more info https://arctic.netease.com/ch/meta-service/table-properties/#_5)

[wangtaohz]

In change store, similar to base store，you can change these properties, change.data.ttl.minutes, snapshot.change.keep.minutes.

[XBaith]

That works! Thanks.

[XBaith]

[wangtaohz]

And what is the table type？
If it is upgraded from a hive table, maybe you should set optimize.full.trigger.max-interval, to trigger the full optimize, which will rewrite all base files.

[XBaith]

The table type is arctic