etcdserver: compact raft log entries more frequently #18622
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: clement2026 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Hi @clement2026. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files
... and 14 files with indirect coverage changes @@ Coverage Diff @@
## main #18622 +/- ##
==========================================
+ Coverage 68.79% 68.81% +0.01%
==========================================
Files 420 420
Lines 35522 35532 +10
==========================================
+ Hits 24438 24451 +13
+ Misses 9657 9654 -3
Partials 1427 1427 Continue to review full report in Codecov by Sentry.
|
Signed-off-by: Clement <[email protected]>
clement2026
force-pushed
the
compact-raft-log-more-often
branch
from
af9209f
to
f1a76a2
Compare
| // Keep some in memory log entries for slow followers, while keeping the entries up to snapshot index. | ||
| // Only compact raft log once every N applies | ||
| if ep.appliedi <= ep.snapi+s.Cfg.SnapshotCatchUpEntries || ep.appliedi%s.Cfg.CompactRaftLogEveryNApplies != 0 { | ||
| return | ||
| } | ||
|
|
||
| compacti := ep.appliedi - s.Cfg.SnapshotCatchUpEntries |
There was a problem hiding this comment.
| // Keep some in memory log entries for slow followers, while keeping the entries up to snapshot index. | |
| // Only compact raft log once every N applies | |
| if ep.appliedi <= ep.snapi+s.Cfg.SnapshotCatchUpEntries || ep.appliedi%s.Cfg.CompactRaftLogEveryNApplies != 0 { | |
| return | |
| } | |
| compacti := ep.appliedi - s.Cfg.SnapshotCatchUpEntries | |
| // Retain all log entries up to the latest snapshot index to ensure any member can recover from that snapshot. | |
| // Beyond the snapshot index, preserve the most recent s.Cfg.SnapshotCatchUpEntries entries in memory. | |
| // This allows slow followers to catch up by synchronizing entries instead of requiring a full snapshot transfer. | |
| // Only compact raft log once every N applies | |
| if ep.snapi > s.Cfg.SnapshotCatchUpEntries { | |
| return | |
| } | |
| compacti := ep.snapi - s.Cfg.SnapshotCatchUpEntries |
In the first PR, let's keep the same number of entries, just make the compaction independent from snapshot.
|
This change does to many things, there are two changes here that should be independent:
|
|
Sure. I'll break down this PR. |
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Part of #17098
Goal
Compact raft log more frequently to save memory.
Key Changes
maybeCompactRaftLogto handle raft log compaction.CompactRaftLogEveryNAppliesandDefaultCompactRaftLogEveryNAppliesto control how frequently the raft log is compacted.TODO