Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write a specialized threaded compressor and special case writing with one thread #162

Merged
merged 17 commits into from
Oct 11, 2023
Merged

Write a specialized threaded compressor and special case writing with one thread #162

merged 17 commits into from
Oct 11, 2023

Conversation

rhpvorderman
Copy link
Collaborator

Checklist

  • Pull request details were added to CHANGELOG.rst
  • Documentation was updated (if needed)

@rhpvorderman
Create a separate function that calculates the decompressed block and…
07584be 
… the crc outside the GIL
@rhpvorderman
Use a separate object with reusable buffers
fed1d1d
@rhpvorderman
Fix memory leak
048cc1d
@rhpvorderman
Calculate crc on the fly
795e79c
@rhpvorderman
Make sure writebuffersize is set
0214e12
@rhpvorderman
Read smaller blocks and store less blocks in queue to reduce memory s…
1278c7b 
…ize for reading
@rhpvorderman
Make block size configurable
49ab675
@rhpvorderman
Use a smaller queue for better characteristics
dff95a3
@rhpvorderman
Combine queues and threads for threads equals 1, for greater efficiency
424a5b9
@rhpvorderman
Parametrize tests for threaded writer so both cases are tested
1509a7a
@rhpvorderman
Build tuple directly rather than using Py_BuildValue
1107f47
@rhpvorderman
Use meth_fastcall for compress_and_crc
550e010
@rhpvorderman
Make sure zdict cannot be excessively large
59620fa
@rhpvorderman
Unblock threads at the earliest possible convenience
0d1cd10
@rhpvorderman
Update changelog with changes for threaded writing
8327101
@codecov
Copy link

codecov bot commented Oct 11, 2023 •
edited
Loading

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (8c5980d) 96.03% compared to head (e8fd360) 94.55%.

Additional details and impacted files 
@@             Coverage Diff             @@
##           develop     #162      +/-   ##
===========================================
- Coverage    96.03%   94.55%   -1.49%     
===========================================
  Files            3        3              
  Lines          404      441      +37     
  Branches        68       72       +4     
===========================================
+ Hits           388      417      +29     
- Misses          13       19       +6     
- Partials         3        5       +2
Files Coverage Δ
src/isal/igzip_threaded.py 94.46% <95.00%> (-2.55%) ⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rhpvorderman rhpvorderman merged commit 71b7d61 into develop Oct 11, 2023
20 checks passed
@rhpvorderman rhpvorderman deleted the specialcaseonethread branch October 11, 2023 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@rhpvorderman