Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUPPORT] What's the difference between compaction service and log compaction service #12190

Open
bithw1 opened this issue Nov 1, 2024 · 1 comment
Labels
feature-enquiry issue contains feature enquiries/requests or great improvement ideas table-service

Comments

@bithw1
Copy link

bithw1 commented Nov 1, 2024

HI,

I am reading Hudi 0.15.0 source code, find there are two compaction inline options, one is hoodie.compact.inline,the other is hoodie.log.compaction.inline,

Both of them say that the compaction/log compaction service will be triggered after each write, I would ask what't the difference between these two services.

The defintion of the two options are as follows:

  public static final ConfigProperty<String> INLINE_COMPACT = ConfigProperty
      .key("hoodie.compact.inline")
      .defaultValue("false")
      .withDocumentation("When set to true, compaction service is triggered after each write. While being "
          + " simpler operationally, this adds extra latency on the write path.");

and

  public static final ConfigProperty<String> INLINE_LOG_COMPACT = ConfigProperty
      .key("hoodie.log.compaction.inline")
      .defaultValue("false")
      .markAdvanced()
      .sinceVersion("0.13.0")
      .withDocumentation("When set to true, log compaction service is triggered after each write. While being "
          + " simpler operationally, this adds extra latency on the write path.");
@danny0405
Copy link
Contributor

log compaction is used for merging multiple log blocks into a larger one(also a log block); while compaction merges all the log files of the current version into a parquet file, also rolls over the current file slice version into new.

@danny0405 danny0405 added feature-enquiry issue contains feature enquiries/requests or great improvement ideas table-service labels Nov 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-enquiry issue contains feature enquiries/requests or great improvement ideas table-service
Projects
Status: Awaiting Triage
Development

No branches or pull requests

2 participants