Cache only changed columns?

[Modexus]

Feature request

Cache only the actual changes to the dataset i.e. changed columns.

Motivation

I realized that caching actually saves the complete dataset again.
This is especially problematic for image datasets if one wants to only change another column e.g. some metadata and then has to save 5 TB again.

Your contribution

Is this even viable in the current architecture of the package?
I quickly looked into it and it seems it would require significant changes.

I would spend some time looking into this but maybe somebody could help with the feasibility and some plan to implement before spending too much time on it?

[Modexus]

so I guess a workaround to this is to simply remove all columns except the ones to cache and then add them back with concatenate_datasets(..., axis=1).

[lhoestq]

yes this is the right workaround. We're keeping the cache like this to make it easier for people to delete intermediate cache files