Preferred vDPU index to be standalone when entering into standalone setup. | -##### 1.2.2.2. ENI placement configurations +##### 2.1.2.2. HA scope configurations +* The HA scope configuration table is programmed by SDN controller and contains the HA config for each HA scope (DPU or ENI) that only lands on this specific switch. +* When HA scope is set to `dpu` in HA set, SmartSwitch HA will use the HA set id as the HA scope id, otherwise, HA scope id will be the ENI id. + +| Table | Key | Field | Description | +| --- | --- | --- | --- | +| DASH_HA_SCOPE_CONFIG_TABLE | | | HA scope configuration. | +| | \
(Desired Active) + participant S1D as Switch 1 DPU
(Desired Standby) + participant S1N as Switch 1 NPU + participant SA as All Switches
(Includes Switch 0/1) + participant SDN as SDN Controller + + SDN->>SA: Create HA set and HA scope
on all switches with
desired Active/Standby roles. + SA->>SA: Start BFD probe to all DPUs
but DPUs do not respond yet.
+ SDN->>S0N: Push all DASH configuration objects to DPU0. This includes the HA set and HA scope with
desired Active/Standby roles but Admin state set to disabled. + SDN->>S1N: Push all DASH configuration objects to DPU1.
This includes the HA set and HA scope with
desired Active/Standby roles but
Admin state set to disabled. + S0N->>S0D: Create all DASH objects
including HA set and HA scope
with admin state to Disabled. + S1N->>S1D: Create all DASH objects
including HA set and HA scope
with admin state to Disabled. + + SDN->>S0N: Set DPU0 HA scope
admin state to Enabled + SDN->>S1N: Set DPU1 HA scope
admin state to Enabled + S0N->>S0D: Set HA scope
admin state to Enabled + S1N->>S1D: Set HA scope
admin state to Enabled + + S0D->>S1D: Connect to peer and
start pairing + S1D->>S0D: Connect to peer and
start pairing + Note over S0D,S1D: Bulk sync can happen at
this stage if needed. + + S0D->>S0N: Request role activation + S1D->>S1N: Request role activation + S0N->>SDN: Request role activation for DPU0 + S1N->>SDN: Request role activation for DPU1 + Note over S0D,S1D: The DPUs will wait for activation
from SDN controller before
moving to active/standby state. + Note over S0D,S1D: Inline sync channel can be
established at this stage. + + SDN->>S0N: Approve role activation for DPU0 + SDN->>S1N: Approve role activation for DPU1 + S0N->>S0D: Approve role activation + S1N->>S1D: Approve role activation + + S0D->>S0N: Enter active state + S0N->>S0D: Request to start responding to all BFD probes + S1D->>S1N: Enter standby state + S1N->>S1D: Request to start responding to all BFD probes + S0D->>S0D: Start responding to all BFD probes + S1D->>S1D: Start responding to all BFD probes + + SA->>SA: BFD to both DPUs will be up,
but only DPU0 will be set as next hop + Note over S0N,SA: From now on, traffic for all ENIs in this HA scope will be forwarded to DPU0. +``` + +### 8.2. Planned shutdown + +With `DPU-driven` mode, the shutdown request will be directly forwarded to DPU. `hamgrd` will ***NOT*** work with each other to make sure the shutdown is done in a safe way. + +Let's say, DPUs 0 and 1 are in steady state (Active/Standby) in the HA set and we are trying to shutdown DPU 0. The workflow of planned shutdown will be as follows: + +1. SDN controller programs the DPU 0 HA scope desired HA state to `dead`. +2. `hamgrd` and DPU side `swss` will: + 1. Shutdown the BFD responder on DPU side, which moves all traffic to DPU 1. + 2. Update the DPU 0 HA role to `Dead`. +3. DPU 0 moves itself to `Destroying` state. + 1. At this stage, traffic can initially land on both DPUs and until it fully shifts to DPU 1. + 2. DPU 0 needs to gracefully shutdown and help DPU 1 move to `Standalone` state while continuing to sync flows. + 3. DPU 0 will start a configurable timer to wait for the network to converge and traffic to fully switchover to DPU 1. + 4. Once this timer expires, no traffic should land on DPU 0. DPU 0 stops forwarding traffic and starts shutting down. +4. At the same time that DPU 0 is shutting down, DPU 1 will: + 1. Move to `SwitchingToStandalone` to drain all the in-flight messages between the DPUs and then finally `Standalone` state when DPU 0 is fully shutdown. + 2. If DPU 1 was in `Standby` state, it needs to pause flow resimulation for all connections synced from old Active and send flow reconcile request to SDN controller. + 3. Only after flow reconcile is received from SDN controller, the flow resimulation will be resumed for these connections. This is to ensure the flow resimulation won't accidentally picking up old SDN policy. + +The above DPU state transitions remain same irrespective of which DPU is shutdown. The only difference is if standby DPU 1 was being shutdown instead of DPU 0, then the switch NPUs would continue to forward to DPU 0 without any need to shift traffic. + + +```mermaid +sequenceDiagram + autonumber + + participant S0N as Switch 0 NPU + participant S0D as Switch 0 DPU
(Active->Dead) + participant S1D as Switch 1 DPU
(Standby->Active) + participant S1N as Switch 1 NPU + participant SA as All Switches
(Includes Switch 0/1) + participant SDN as SDN Controller + + Note over SA: Initially, traffic for all ENIs in
this HA scope will be forwarded to DPU0. + + SDN->>S0N: Programs HA scope with dead desired state for DPU0. + S0N->>S0D: Request to stop responding to BFD. + + S0D->>S0D: Stop responding BFD. + SA->>SA: Set DPU1 as next hop for traffic forwarding. + Note over SA: Traffic for all ENIs in this HA scope
will start transitioning if required. + + S0N->>S0D: Update HA scope with dead state. + S0D->>S0N: Enter Destroying state + Note over S0D,S1D: DPU starts to drive internal
HA states to gracefully
remove the peer. + Note over S0D,S1D: DPU waits for
network convergence timer. + + S1D->>S1N: Enter SwitchingToStandalone state + Note over S0D,S1D: Inline sync channel
should be stopped. + + S0D->>S0N: Enter dead state + S1D->>S1N: Enter standalone state + Note over S0D,S1D: DPU1 ignores all flow
resimulation requests
for synced connecctions + + S1D->>S1N: Send flow reconcile needed notification + S1N->>SDN: Send flow reconcile needed notification + SDN->>S1N: Ensure the latest policy is programmed + SDN->>S1N: Request flow reconcile + S1N->>S1D: Request flow reconcile + Note over S0D,S1D: DPU1 resumes handling all flow
resimulation requests +``` + +### 8.3. Planned switchover + +In DPU-driven setup, switchover is done via shutdown one side of the DPU, and DPUs pair need to be able to handle the switchover internally. + +### 8.4. ENI migration / HA re-pair + +To support things like upgrade, we need to update the HA set to pair with another DPU. In this case, the following steps needs to be performed step by step: + +1. Trigger [Planned shutdown](#82-planned-shutdown) on the DPU that needs to be removed from the HA set. +2. Update HA set information on all switches and the corresponding DPUs. + - This will cause the tables and objects related to old HA set to be removed and new HA set to be created. + - The new DPU joining the HA pair will be in Dead state at this point. +3. Program all ENIs on the new DPU. +4. Once all configurations are done, SDN controller updates the HA admin state to Enabled to start the [HA set creation](#81-ha-set-creation) workflow. + +## 9. Unplanned operations + +### 9.1. Unplanned failover + +```mermaid +sequenceDiagram + autonumber + + participant S0N as Switch 0 NPU + participant S0D as Switch 0 DPU
(Active->Dead) + participant S1D as Switch 1 DPU
(Standby->Standalone) + participant S1N as Switch 1 NPU + participant SA as All Switches
(Includes Switch 0/1) + participant SDN as SDN Controller + + Note over S0N,SA: Initially, traffic for all ENIs in
this HA scope will be forwarded to DPU0. + destroy S0D + S0D-XS0D: DPU0 went dead + S0N->>S0N: PMON detects DPU failure
and update DPU state to dead. + + SA->>SA: BFD to DPU0 start to fail. + Note over SA: Traffic for all ENIs in this HA scope
starts to shift to DPU1. + + S1D->>S1D: DPU-to-DPU probe starts to fail. + Note over S1D: DPU starts to drive internal
HA states to failover. + + S1D->>S1N: Enter standalone state + Note over S1D: DPU1 starts to ignore all flow
resimulation requests
for synced connections + + S1D->>S1N: Send flow reconcile needed notification + S1N->>SDN: Send flow reconcile needed notification + SDN->>S1N: Ensure the latest policy is programmed + SDN->>S1N: Request flow reconcile + S1N->>S1D: Request flow reconcile +``` + +## 10. Split-brain and re-pair + +In DPU-driven HA mode, depends on how DPU detects the health state of its pair, it could run into split-brain problem, when the DPU-DPU communication channel breaks while both DPUs are still up. In this case, both DPU could move into `Standalone` state and both starts to make flow decisions. + +When this happens, the DPU could not recover by its own and shall refuse to re-pair automatically. It is the responsibility of the SDN controller to break the split-brain by restarting HA on one of the DPUs (setting HA scope disabled to `true` and then `false`). Unlike the desired HA state, which will drive the state machine gracefully, the disabled state will force the DPU to shutdown. This operation is required, because the DPU HA state machine could stuck and could not never be recovered gracefully. + +## 11. Flow tracking and replication + +`DPU-Scope-DPU-Driven` setup will not change the flow lifetime is managed and how the inline flow replication works, since they are currently managed by DPU under the SAI APIs already. However, it will change how bulk sync works, as DPU will directly do the bulk sync without going through HA control plane sync channel. + +## 12. Detailed design + +Please refer to the [detailed design doc](./smart-switch-ha-detailed-design.md) for DB schema, telemetry, SAI API and CLI design. + +## 13. Test Plan + +Please refer to HA test docs for detailed test bed setup and test case design. diff --git a/doc/smart-switch/high-availability/smart-switch-ha-hld.md b/doc/smart-switch/high-availability/smart-switch-ha-hld.md index 3c0d204e4e..8887360163 100644 --- a/doc/smart-switch/high-availability/smart-switch-ha-hld.md +++ b/doc/smart-switch/high-availability/smart-switch-ha-hld.md @@ -8,6 +8,7 @@ | 0.4 | 08/17/2023 | Riff Jiang | Redesigned HA control plane data channel | | 0.5 | 10/14/2023 | Riff Jiang | Merged resource placement and topology section and moved detailed design out for better readability | | 0.6 | 10/22/2023 | Riff Jiang | Added ENI leak detection | +| 0.7 | 10/13/2024 | Riff Jiang | Update HA control plane components graph to match with latest design update on database and gNMI. | 1. [1. Background](#1-background) 2. [2. Terminology](#2-terminology) @@ -153,6 +154,8 @@ 2. [11.5.2.2. Flow tracking in steady state](#11522-flow-tracking-in-steady-state) 3. [11.5.2.3. Tracking phase](#11523-tracking-phase) 4. [11.5.2.4. Syncing phase](#11524-syncing-phase) + 3. [11.5.3. Multi-channel problem](#1153-multi-channel-problem) + 1. [11.5.3.1. Per-flow version number](#11531-per-flow-version-number) 6. [11.6. Flow re-simulation support](#116-flow-re-simulation-support) 12. [12. Debuggability](#12-debuggability) 1. [12.1. ENI leak detection](#121-eni-leak-detection) @@ -1462,7 +1465,7 @@ Once the HA pair starts to run as standalone setup, the inline sync will stop wo 1. New flows can be created on one side, but not the other. 2. Existing flows can be terminated on one side, but not the other. -3. Existing flows can be aged out on one side, but not the other, depending on how we manage the lifetime of the lows. +3. Existing flows can be aged out on one side, but not the other, depending on how we manage the lifetime of the flows. 4. Due to policy updates, the same flow might get different packet transformations now, e.g., flow resimulation or flow recreation after policy update. And during recovery, we need to merge these 2 sets of flows back to one using "[bulk sync](#115-bulk-sync)". @@ -1880,6 +1883,24 @@ Whenever any flow is created or updated (flow re-simulation), update the flow ve 4. Handle bulk sync done event from ASIC, which will be sent after all flow change events are notified. 5. Call bulk sync completed SAI API, so ASIC can delete all tracked flow deletion records. Also reset `ToSyncFlowVerMin` and `ToSyncFlowVerMax` to 0, because there is nothing to sync anymore. +#### 11.5.3. Multi-channel problem + +During bulk sync, there would be two sync channels now: inline sync and bulk sync. As the 2 channels work independently, if a flow uses both channels to sync states from active to standby, the sync messages received by standby may be out-of-order and thus cause problems. + +The following illustration demonstrates one problematic case: the inline sync first writes a newer state to standby data plane and then bulk sync writes an older state. Finally, the synchronized state in the standby is the older state, rather than the desired newer one. + +
