[dr-autosync] block read/write more than 40s during change sync to async

[mayjiang0203]

Enhancement Task

[2023/06/25 15:14:18.610 +08:00] [INFO] [workload.go:753] ["[Current] STOCK_LEVEL - Takes(s): 0.7, Count: 1, TPM: 81.6, Sum(ms): 10.3, Avg(ms): 10.2, 50th(ms): 10.5, 90th(ms): 10.5, 95th(ms): 10.5, 99th(ms): 10.5, 99.9th(ms): 10.5, Max(ms): 10.5\r\n"]
[2023/06/25 15:14:49.081 +08:00] [INFO] [pdutil.go:385] ["Current replica status"] [dr-auto-sync.state=async_wait]
[2023/06/25 15:14:49.081 +08:00] [INFO] [primary.go:190] ["monitor status"]
2023-06-25T15:14:49.145+0800    INFO    k8s/client.go:132       it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/06/25 15:15:00.710 +08:00] [INFO] [workload.go:753] ["[Current] DELIVERY - Takes(s): 0.6, Count: 4, TPM: 394.9, Sum(ms): 147.2, Avg(ms): 36.7, 50th(ms): 33.6, 90th(ms): 44.0, 95th(ms): 44.0, 99th(ms): 44.0, 99.9th(ms): 44.0, Max(ms): 44.0\r\n"] 

version: 6.5.3

configuration set as below.

MySQL [(none)]> show config where  name like "%wait-store-timeout%";
+------+---------------+--------------------------------------------------+-------+
| Type | Instance      | Name                                             | Value |
+------+---------------+--------------------------------------------------+-------+
| pd   | pd1-peer:2379 | replication-mode.dr-auto-sync.wait-store-timeout | 10s   |
| pd   | pd2-peer:2379 | replication-mode.dr-auto-sync.wait-store-timeout | 10s   |
| pd   | pd3-peer:2379 | replication-mode.dr-auto-sync.wait-store-timeout | 10s   |
+------+---------------+--------------------------------------------------+-------+
3 rows in set (6.34 sec)
MySQL [(none)]> show config where  name like "%pd-store-heartbeat-tick-interval%";
+------+------------------+--------------------------------------------+-------+
| Type | Instance         | Name                                       | Value |
+------+------------------+--------------------------------------------+-------+
| tikv | tikv2-peer:20160 | raftstore.pd-store-heartbeat-tick-interval | 1s    |
| tikv | tikv1-peer:20160 | raftstore.pd-store-heartbeat-tick-interval | 1s    |
| tikv | tikv4-peer:20160 | raftstore.pd-store-heartbeat-tick-interval | 1s    |
| tikv | tikv6-peer:20160 | raftstore.pd-store-heartbeat-tick-interval | 1s    |
| tikv | tikv5-peer:20160 | raftstore.pd-store-heartbeat-tick-interval | 1s    |
| tikv | tikv3-peer:20160 | raftstore.pd-store-heartbeat-tick-interval | 1s    |
+------+------------------+--------------------------------------------+-------+
6 rows in set (0.06 sec)

[mayjiang0203]

/assign @disksing

[rleungx]

7.1 is fixed by #7789