New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: Reduce memory usage of BF in DataNode and QueryNode #38129

Merged

sre-ci-robot merged 6 commits into milvus-io:master from bigsheeper:2412-opt-bf

Jan 14, 2025

Contributor

bigsheeper commented Dec 2, 2024 •

edited

Loading

DataNode: Skip generating BF during the insert phase (BF will be regenerated during the sync phase).
QueryNode: Skip generating or maintaining BF for growing segments; deletion checks will be handled in the segcore.

issue: #37630

sre-ci-robot added the size/M label

sre-ci-robot requested review from aoiasd and godchen0212

December 2, 2024 04:02

mergify bot added dco-passed kind/enhancement labels

Contributor

mergify bot commented Dec 2, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

codecov bot commented Dec 2, 2024 •

edited

Loading

Codecov Report

Attention: Patch coverage is 69.69697% with 10 lines in your changes missing coverage. Please review.

Project coverage is 81.10%. Comparing base (d89768f) to head (5ec2c5a).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
...nternal/flushcommon/writebuffer/l0_write_buffer.go	16.66%	10 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #38129       +/-   ##
===========================================
+ Coverage   69.71%   81.10%   +11.39%     
===========================================
  Files         299     1404     +1105     
  Lines       26768   197767   +170999     
===========================================
+ Hits        18661   160400   +141739     
- Misses       8107    31719    +23612     
- Partials        0     5648     +5648

Components	Coverage Δ
Client	`79.53% <ø> (∅)`
Core	`69.65% <ø> (-0.06%)`	⬇️
Go	`83.03% <69.69%> (∅)`

Files with missing lines	Coverage Δ
internal/querynodev2/segments/segment.go	`67.79% <100.00%> (ø)`
pkg/util/paramtable/component_param.go	`98.40% <100.00%> (ø)`
...nternal/flushcommon/writebuffer/l0_write_buffer.go	`40.47% <16.66%> (ø)`

... and 1109 files with indirect coverage changes


          enhance: Reduce memory usage of BF in DataNode and QueryNode

b9d0a90

Signed-off-by: bigsheeper <[email protected]>

bigsheeper force-pushed the 2412-opt-bf branch from ef5e8e9 to b9d0a90 Compare

December 2, 2024 06:22

Contributor Author

bigsheeper commented Dec 2, 2024

Before:

After:

bigsheeper mentioned this pull request

enhance: [10kcp] Reduce memory usage of BF in DataNode and QueryNode #38133

Merged

czs007 pushed a commit that referenced this pull request


          enhance: [10kcp] Reduce memory usage of BF in DataNode and QueryNode (#…

338ccc9

…38133)

1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.

issue: #37630

pr: #38129

---------

Signed-off-by: bigsheeper <[email protected]>

Contributor

mergify bot commented Dec 2, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Contributor Author

bigsheeper commented Dec 3, 2024

/run-cpu-e2e

Contributor

mergify bot commented Dec 3, 2024

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Contributor Author

bigsheeper commented Dec 3, 2024

/run-cpu-e2e

mergify bot added the ci-passed label

bigsheeper mentioned this pull request

enhance: [2.4] Reduce memory usage of BF in DataNode and QueryNode #38215

Open

czs007 modified the milestone: 2.5.0

bigsheeper mentioned this pull request

[Bug]: Test 10000 collections and 1000 partitions #37630

Open

41 tasks

bigsheeper added the 10kcp label


          Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

a5eeba9

…2-opt-bf

mergify bot removed the ci-passed label

Contributor

mergify bot commented Dec 31, 2024

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

Contributor Author

bigsheeper commented Dec 31, 2024

rerun go-sdk

Contributor

mergify bot commented Dec 31, 2024

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

Contributor Author

bigsheeper commented Jan 1, 2025

rerun go-sdk

Contributor

mergify bot commented Jan 1, 2025

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

Contributor Author

bigsheeper commented Jan 1, 2025

rerun go-sdk

bigsheeper mentioned this pull request

fix: Fix checkGeneralCapacity slowly #37976

Merged

Contributor

mergify bot commented Jan 1, 2025

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

bigsheeper mentioned this pull request

enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode #38913

Merged


          Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

476b309

…2-opt-bf

Contributor

mergify bot commented Jan 9, 2025

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.


          Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

118b060

…2-opt-bf

czs007 self-assigned this

bigsheeper added 2 commits

January 14, 2025 20:28


          Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

fe0da96

…2-opt-bf


          update

5ec2c5a

Signed-off-by: bigsheeper <[email protected]>

Collaborator

czs007 commented Jan 14, 2025

/approve
/lgtm

sre-ci-robot added the lgtm label

Contributor

sre-ci-robot commented Jan 14, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigsheeper, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot added the approved label

mergify bot added the ci-passed label

sre-ci-robot merged commit ec2e77b into milvus-io:master

18 of 20 checks passed

sre-ci-robot pushed a commit that referenced this pull request


          enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode (#…

5b0bb4c

…38913)

1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.

issue: #37630

pr: #38129

---------

Signed-off-by: bigsheeper <[email protected]>

gifi-siby pushed a commit to gifi-siby/milvus that referenced this pull request


          enhance: Reduce memory usage of BF in DataNode and QueryNode (milvus-…

e5400c4

…io#38129)

1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.

issue: milvus-io#37630

---------

Signed-off-by: bigsheeper <[email protected]>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

10kcp approved ci-passed dco-passed kind/enhancement lgtm size/M