[Bug]: [T2][Chassis] - TSA/TSB - Intermittent issues seen with neighbors' route verification for traffic-shift tests

[sanjair-git]

Issue Description

Traffic shift test cases such as 'test_traffic_shift.py', 'test_startup_tsa_tsb_service.py', 'test_reliable_tsa.py' fails intermittently either during 'verify_current_routes_announced_to_neighs' or during 'verify_only_loopback_routes_are_announced_to_neighs' mostly on upstream T2 line card with max configuration.

Looks to be a timing issue in both cases.

Results you see

            # Verify that all routes advertised to neighbor at the start of the test
            if not wait_until(300, 3, 0, verify_current_routes_announced_to_neighs, linecard, dut_nbrhosts[linecard],
                              orig_v4_routes[linecard], cur_v4_routes, 4):
                if not check_and_log_routes_diff(linecard, dut_nbrhosts[linecard],
                                                 orig_v4_routes[linecard], cur_v4_routes, 4):
>                   pytest.fail("Not all ipv4 routes are announced to neighbors")
E                   Failed: Not all ipv4 routes are announced to neighbors

 pytest_assert(verify_only_loopback_routes_are_announced_to_neighs(
                    duthosts, linecard, dut_nbrhosts[linecard], traffic_shift_community),
                    "Failed to verify routes on nbr in TSA")
E                   Failed: Failed to verify routes on nbr in TSA

Results you expected to see

All traffic shift test cases such as 'test_traffic_shift.py', 'test_startup_tsa_tsb_service.py', 'test_reliable_tsa.py' should pass without any of these issues.

Is it platform specific

generic

Relevant log output

Output of show version

SONiC Software Version: SONiC.20240532.02
SONiC OS Version: 12
Distribution: Debian 12.6
Kernel: 6.1.0-22-2-amd64
Build commit: b3bd0a2dc2
Build date: Fri Jan 10 11:30:50 UTC 2025
Built by: azureuser@329539e8c000000

Attach files (if any)

No response

[Javier-Tan]

Hi @sanjair-git , we have a PR for test_reliable_tsa #16523 currently, I will have a look across all tests and see if adding wait_until will help there

[sanjair-git]

Hi @sanjair-git , we have a PR for test_reliable_tsa #16523 currently, I will have a look across all tests and see if adding wait_until will help there

Hi @Javier-Tan, thanks for adding wait_until for reliable_tsa. It would be better if we might need to add the same to 'startup-tsa-tsb' and 'traffic-shift' test suite as well. I see similar 'verify_only_loopback_routes_are_announced_to_neighs' failures on these test suites too.

[Javier-Tan]

Hi @sanjair-git , we have a PR for test_reliable_tsa #16523 currently, I will have a look across all tests and see if adding wait_until will help there

Hi @Javier-Tan, thanks for adding wait_until for reliable_tsa. It would be better if we might need to add the same to 'startup-tsa-tsb' and 'traffic-shift' test suite as well. I see similar 'verify_only_loopback_routes_are_announced_to_neighs' failures on these test suites too.

@sanjair-git I have updated #16523 to include wait_until / assert for all tests using verify_only_loopback_routes_are_announced_to_neighs, I will create one more PR for verify_current_routes_announced_to_neighs before closing this issue