Increase checksum buffer to 128kb, improving download performance. #295
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reading 2kb at a time to compute the checksum limits network throughput. Bumping up to 128kb seems to give a good balance of memory usage and performance.
Benchmarks done on a m5n.16xlarge EC2 instance doing a reposync on the Amazon Linux 2023 x86-64 repositories showed that this change, when combined with the (smaller) benefits of my avoiding libc IO patch, reduce system CPU time by another half second, and cut a further 3 seconds off total time:
102s (original) -> 99 (no libc buffered io) -> 95s (this patch)