Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE-1018] Refine Storage Group Feature #1031

Merged
merged 57 commits into from
Jul 30, 2023

Conversation

CraneShiEMC
Copy link
Collaborator

@CraneShiEMC CraneShiEMC commented Jul 1, 2023

Purpose

Resolves #1018

Refine Storage Group Feature:

  1. Add StorageGroupStatus
  2. If we perform drive removal on the drive selected by some storagegroup with numberDrivesPerNode>0, the storagegroup's reconcilation should be triggered and it can select another drive if applicable.
  3. Block StorageGroupSpec Update
  4. Add StorageGroupController UT
  5. For the drive with existing volumes that matches the driveSelector of some storage group, the drive should be selected into its matching storage group when its exsiting volumes removed.
  6. The blocked storagegroup deletion can continue instantly after all exisitng volumes on drives of this storage group is removed.

PR checklist

  • Add link to the issue
  • Choose Project
  • Choose PR label
  • New unit tests added
  • Modified code has meaningful comments
  • All TODOs are linked with the issues
  • All comments are resolved

Testing

Manual test passed.
Custom CI passed: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-ci/1559/
Custom Acceptance Passed: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-tar_b_ona/36/

@CraneShiEMC CraneShiEMC changed the title Refine Storage Group Feature [ISSUE-1018] Refine Storage Group Feature Jul 1, 2023
@codecov
Copy link

codecov bot commented Jul 1, 2023

Codecov Report

Patch coverage: 77.01% and project coverage change: -0.20% ⚠️

Comparison is base (57095d3) 72.92% compared to head (b57da1f) 72.72%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1031      +/-   ##
==========================================
- Coverage   72.92%   72.72%   -0.20%     
==========================================
  Files          62       63       +1     
  Lines        8266     8796     +530     
==========================================
+ Hits         6028     6397     +369     
- Misses       1980     2111     +131     
- Partials      258      288      +30     
Flag Coverage Δ
unittests 72.72% <77.01%> (-0.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...controllers/storagegroup/storagegroupcontroller.go 68.40% <75.32%> (ø)
pkg/crcontrollers/drive/drivecontroller.go 69.58% <90.00%> (+1.71%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Shi, Crane <[email protected]>
@CraneShiEMC CraneShiEMC force-pushed the feature-refine-storage-group branch from 91816f6 to 61a170a Compare July 2, 2023 06:07
@CraneShiEMC CraneShiEMC force-pushed the feature-refine-storage-group branch from e4ad183 to decc3b6 Compare July 3, 2023 15:01
@CraneShiEMC CraneShiEMC force-pushed the feature-refine-storage-group branch 2 times, most recently from a710249 to c769eb1 Compare July 10, 2023 13:19
Signed-off-by: Shi, Crane <[email protected]>
Signed-off-by: Shi, Crane <[email protected]>
Signed-off-by: Shi, Crane <[email protected]>
Signed-off-by: Shi, Crane <[email protected]>
Signed-off-by: Shi, Crane <[email protected]>
@CraneShiEMC CraneShiEMC requested a review from lordjun July 25, 2023 07:16
@CraneShiEMC CraneShiEMC merged commit 8085a0a into master Jul 30, 2023
6 checks passed
@CraneShiEMC CraneShiEMC deleted the feature-refine-storage-group branch July 30, 2023 20:17
CraneShiEMC added a commit that referenced this pull request Jul 30, 2023
* add StorageGroupStatus

Signed-off-by: Shi, Crane <[email protected]>

* support StorageGroupStatus in current workflows

Signed-off-by: Shi, Crane <[email protected]>

* refine log

Signed-off-by: Shi, Crane <[email protected]>

* refine error handling

Signed-off-by: Shi, Crane <[email protected]>

* trigger storage group resync if applicable in drive removal

Signed-off-by: Shi, Crane <[email protected]>

* A drive whose Usage is REMOVED will not be selected in any storage group and its existing sg label takes no effect

Signed-off-by: Shi, Crane <[email protected]>

* sg feature will not apply to drive physically removed

Signed-off-by: Shi, Crane <[email protected]>

* handle the drive removal case of drive with manual sg label

Signed-off-by: Shi, Crane <[email protected]>

* fix go lint error

Signed-off-by: Shi, Crane <[email protected]>

* add UT case for drive-removal-triggered sg sync

Signed-off-by: Shi, Crane <[email protected]>

* improve UT coverage

Signed-off-by: Shi, Crane <[email protected]>

* refine sg annotation for drive removal

Signed-off-by: Shi, Crane <[email protected]>

* handle case of invalid sg for drive removal

Signed-off-by: Shi, Crane <[email protected]>

* also exclude removing sg for trigger sg resync in drive removal

Signed-off-by: Shi, Crane <[email protected]>

* refine sg removal status handling

Signed-off-by: Shi, Crane <[email protected]>

* Revert "refine error handling"

This reverts commit 06607e7.

Signed-off-by: Shi, Crane <[email protected]>

* refine log and some code logic

Signed-off-by: Shi, Crane <[email protected]>

* try to add immutability validation rule to storagegroup spec

Signed-off-by: Shi, Crane <[email protected]>

* upgrade controller-gen version to v0.9.2

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT initial

Signed-off-by: Shi, Crane <[email protected]>

* Revert "add storagegroupcontroller UT initial"

This reverts commit 1ea8660.

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* refactor and add UT of storagegroupcontroller

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* fix storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* refine the logic of sg deletion

Signed-off-by: Shi, Crane <[email protected]>

* refine

Signed-off-by: Shi, Crane <[email protected]>

* fix bug

Signed-off-by: Shi, Crane <[email protected]>

* fix go-lint err

Signed-off-by: Shi, Crane <[email protected]>

* fix go-lint error

Signed-off-by: Shi, Crane <[email protected]>

* add drive IsClean support, decrease k8s api call, remove manual sg labeling support

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* fix UT

Signed-off-by: Shi, Crane <[email protected]>

* refine corner case handling

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* refine and add UT to storagegroupcontroller

Signed-off-by: Shi, Crane <[email protected]>

* refine storagegroupcontroller and add UT

Signed-off-by: Shi, Crane <[email protected]>

* make controller svc's k8scache also sync sg and lvg objs'

Signed-off-by: Shi, Crane <[email protected]>

* use k8s cache, re-support sg label manual change and refine in sg ctrl

Signed-off-by: Shi, Crane <[email protected]>

* fix lint err

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* storagegroup controller will not reconcile on drive delete event

Signed-off-by: Shi, Crane <[email protected]>

* not support Health, Status, Usage and IsClean as DriveSelector's MatchFields

Signed-off-by: Shi, Crane <[email protected]>

* in storagegroupcontroller's reconcile, only sync drive when reqName is uuid

Signed-off-by: Shi, Crane <[email protected]>

* refine the logic to avoid nil pointer error

Signed-off-by: Shi, Crane <[email protected]>

* revert the usage of k8scache

Signed-off-by: Shi, Crane <[email protected]>

* add custom storage group proposal draft

Signed-off-by: Shi, Crane <[email protected]>

* refine custom storage group proposal

Signed-off-by: Shi, Crane <[email protected]>

---------

Signed-off-by: CraneShiEMC <[email protected]>
CraneShiEMC added a commit that referenced this pull request Jul 31, 2023
* [ISSUE-1040] fix pip3 install PyYAML failed. (#1041)

Signed-off-by: yimingwangdell <[email protected]>

* [ISSUE-1018] Refine Storage Group Feature (#1031)

* add StorageGroupStatus

Signed-off-by: Shi, Crane <[email protected]>

* support StorageGroupStatus in current workflows

Signed-off-by: Shi, Crane <[email protected]>

* refine log

Signed-off-by: Shi, Crane <[email protected]>

* refine error handling

Signed-off-by: Shi, Crane <[email protected]>

* trigger storage group resync if applicable in drive removal

Signed-off-by: Shi, Crane <[email protected]>

* A drive whose Usage is REMOVED will not be selected in any storage group and its existing sg label takes no effect

Signed-off-by: Shi, Crane <[email protected]>

* sg feature will not apply to drive physically removed

Signed-off-by: Shi, Crane <[email protected]>

* handle the drive removal case of drive with manual sg label

Signed-off-by: Shi, Crane <[email protected]>

* fix go lint error

Signed-off-by: Shi, Crane <[email protected]>

* add UT case for drive-removal-triggered sg sync

Signed-off-by: Shi, Crane <[email protected]>

* improve UT coverage

Signed-off-by: Shi, Crane <[email protected]>

* refine sg annotation for drive removal

Signed-off-by: Shi, Crane <[email protected]>

* handle case of invalid sg for drive removal

Signed-off-by: Shi, Crane <[email protected]>

* also exclude removing sg for trigger sg resync in drive removal

Signed-off-by: Shi, Crane <[email protected]>

* refine sg removal status handling

Signed-off-by: Shi, Crane <[email protected]>

* Revert "refine error handling"

This reverts commit 06607e7.

Signed-off-by: Shi, Crane <[email protected]>

* refine log and some code logic

Signed-off-by: Shi, Crane <[email protected]>

* try to add immutability validation rule to storagegroup spec

Signed-off-by: Shi, Crane <[email protected]>

* upgrade controller-gen version to v0.9.2

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT initial

Signed-off-by: Shi, Crane <[email protected]>

* Revert "add storagegroupcontroller UT initial"

This reverts commit 1ea8660.

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* refactor and add UT of storagegroupcontroller

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* fix storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* refine the logic of sg deletion

Signed-off-by: Shi, Crane <[email protected]>

* refine

Signed-off-by: Shi, Crane <[email protected]>

* fix bug

Signed-off-by: Shi, Crane <[email protected]>

* fix go-lint err

Signed-off-by: Shi, Crane <[email protected]>

* fix go-lint error

Signed-off-by: Shi, Crane <[email protected]>

* add drive IsClean support, decrease k8s api call, remove manual sg labeling support

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* fix UT

Signed-off-by: Shi, Crane <[email protected]>

* refine corner case handling

Signed-off-by: Shi, Crane <[email protected]>

* fix

Signed-off-by: Shi, Crane <[email protected]>

* refine and add UT to storagegroupcontroller

Signed-off-by: Shi, Crane <[email protected]>

* refine storagegroupcontroller and add UT

Signed-off-by: Shi, Crane <[email protected]>

* make controller svc's k8scache also sync sg and lvg objs'

Signed-off-by: Shi, Crane <[email protected]>

* use k8s cache, re-support sg label manual change and refine in sg ctrl

Signed-off-by: Shi, Crane <[email protected]>

* fix lint err

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <[email protected]>

* storagegroup controller will not reconcile on drive delete event

Signed-off-by: Shi, Crane <[email protected]>

* not support Health, Status, Usage and IsClean as DriveSelector's MatchFields

Signed-off-by: Shi, Crane <[email protected]>

* in storagegroupcontroller's reconcile, only sync drive when reqName is uuid

Signed-off-by: Shi, Crane <[email protected]>

* refine the logic to avoid nil pointer error

Signed-off-by: Shi, Crane <[email protected]>

* revert the usage of k8scache

Signed-off-by: Shi, Crane <[email protected]>

* add custom storage group proposal draft

Signed-off-by: Shi, Crane <[email protected]>

* refine custom storage group proposal

Signed-off-by: Shi, Crane <[email protected]>

---------

Signed-off-by: CraneShiEMC <[email protected]>

---------

Signed-off-by: yimingwangdell <[email protected]>
Signed-off-by: CraneShiEMC <[email protected]>
Co-authored-by: yimingwangdell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Custom Storage Group Feature
5 participants