Refactor Subnet lock for SubnetPort creation #974
Conversation
yanjunz97
force-pushed
the
refactor-subnet-lock
branch
2 times, most recently
from
ac515ab to
dc0b2c3
Compare
Signed-off-by: Yanjun Zhou <[email protected]>
yanjunz97
force-pushed
the
refactor-subnet-lock
branch
from
dc0b2c3 to
f78aabf
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #974 +/- ##
==========================================
- Coverage 73.37% 73.22% -0.16%
==========================================
Files 118 118
Lines 16394 16440 +46
==========================================
+ Hits 12029 12038 +9
- Misses 3581 3612 +31
- Partials 784 790 +6
|
| @@ -373,3 +404,19 @@ func (service *SubnetPortService) Cleanup(ctx context.Context) error { | |||
| } | |||
| return nil | |||
| } | |||
|
|
|||
| func (service *SubnetPortService) AddPortToSubnet(path string, size int) bool { | |||
There was a problem hiding this comment.
What about rename the functions as AllocatePortFromSubnet and ReleasePortInSubnet?
| } | ||
| } | ||
| // size 0 indicates the Subnet is empty | ||
| if !count.markForDelete && (count.value < size || size == 0) { |
There was a problem hiding this comment.
In what case markForDelete is set as true?
| @@ -128,3 +131,75 @@ func (subnetPortStore *SubnetPortStore) GetByIndex(key string, value string) []* | |||
| } | |||
| return subnetPorts | |||
| } | |||
|
|
|||
| func (subnetPortStore *SubnetPortStore) AddPortToSubnet(path string, size int) bool { | |||
| subnetPortStore.portCountMapLock.Lock() | |||
There was a problem hiding this comment.
The lock scale is lager if we use it on the complete store, can we use lock on the subnet path only?
| type SubnetPortService struct { | ||
| servicecommon.Service | ||
| SubnetPortStore *SubnetPortStore | ||
| } | ||
|
|
||
| type CountWithMark struct { | ||
| value int |
There was a problem hiding this comment.
what about using atomic as the type of value, and use atomic.AddInt(count, 1) or atomic.AddInt(count, -1) to control the value? Or you can use a customized lock in CountWithMark
| }() | ||
| } | ||
|
|
||
| if err := r.SubnetService.DeleteSubnet(*nsxSubnet); err != nil { |
There was a problem hiding this comment.
Is it possible to have a race that when this routine is deleting the subnet, and pod_controller/subnetport_controller is using the subnet to allocate a new port?
| @@ -469,7 +471,17 @@ func (r *SubnetPortReconciler) CheckAndGetSubnetPathForSubnetPort(ctx context.Co | |||
| log.Error(err, "failed to get NSX subnet by subnet CR UID", "subnetList", subnetList) | |||
| return | |||
| } | |||
| subnetPath = *subnetList[0].Path | |||
| nsxSubnet := subnetList[0] | |||
| totalIP := int(*nsxSubnet.Ipv4SubnetSize) | |||
There was a problem hiding this comment.
I would prefer to use totalIP as a field of CountWithMark which is a one-time set value when we create the object for a new subnet, then we can consume it any time we need to allocate a new port with the diff.
Instead of using lock for Subnet, this PR adds a cache in SubnetPort service
to record the number of SubnetPorts on each Subnet.
Read/Write operation for this cache is protected by lock to avoid race condition
and ensure the number of SubnetPort on the Subnet will be controlled below
the Subnet capacity.
Testing done:
delete all Pods and observe the Subnets under the SubnetSet will be deleted by GC.
delete all VMs and observe the Subnet under the SubnetSet will be deleted by GC.
and get IP. The remainings will have
VirtualMachineNetworkReadyasNotReadyandMessage like
network interface "eth0" error: subnetPort is not ready: SubnetPortNotReady - error occurred while processing the SubnetPort CR. Error: no valid IP in Subnet /orgs/default/projects/project-quality/vpcs/ns-1_7fcad11c-da38-4137-96ec-f0f80ae8d96b/subnets/subnet-test-1_60ebfa36-bbe2-4e6e-85df-8228be4e190a