Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support metadata compaction #270

Open
Fokko opened this issue Jan 16, 2024 · 4 comments
Open

Support metadata compaction #270

Fokko opened this issue Jan 16, 2024 · 4 comments
Assignees

Comments

@Fokko
Copy link
Contributor

Fokko commented Jan 16, 2024

Feature Request / Improvement

Add support for compaction. This rewrites the existing manifests into a single one, reducing the number of calls to the object store. This should follow the Java configuration keys:

  • commit.manifest-merge.enabled: Controls whether to automatically merge manifests on writes.
  • commit.manifest.min-count-to-merge: Minimum number of manifests to accumulate before merging.
  • commit.manifest.target-size-bytes: Target size when merging manifest files.
@Fokko Fokko mentioned this issue Jan 18, 2024
@HonahX
Copy link
Contributor

HonahX commented Jan 30, 2024

I am interested in taking this if no one has started working on it.

@Fokko Fokko changed the title Support compaction Support metadata compaction Feb 23, 2024
@HonahX
Copy link
Contributor

HonahX commented Feb 27, 2024

Based on offline discussion with @Fokko, I will first focus on implementing the MergeAppend which supports these keys

  • commit.manifest-merge.enabled
  • commit.manifest.min-count-to-merge
  • commit.manifest.target-size-bytes

The MergeAppend will become the default append method since commit.manifest-merge.enabled is default to True.
The PR for MergeAppend is #363

BTW, it seems rewrite_manifest operations only depends on the commit.manifest.target-size-bytes. Shall we update the description to reflect this?

Copy link

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

Copy link

github-actions bot commented Sep 9, 2024

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Sep 9, 2024
@ndrluis ndrluis reopened this Sep 10, 2024
@github-actions github-actions bot removed the stale label Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants