PD panic after inject PD failed

[fubinzh]

Bug Report

What did you do?

1. TiDB cluster with 1 PD, 3 TiCDC, 3 TiKV, 1 TiDB
2. create changefeed
3. run workload
4. 18:15:17 - 10:15:37 inject PD failure for 20s

What did you expect to see?

PD should be rececovered after PD failure injection

What did you see instead?

PD repeatedly panic

|panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                                                       │
│ [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x21cf4ef]                                                                                                                                                                                      │
│                                                                                                                                                                                                                                                               │
│ goroutine 366 [running]:                                                                                                                                                                                                                                      │
│ github.com/tikv/pd/server.(*GrpcServer).WatchGlobalConfig(0xc0007e2a30, 0xc000876200, {0x3b60b30, 0xc00034d200})                                                                                                                                              │
│     /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/grpc_service.go:2366 +0x18f                                                                                                                                                │
│ github.com/pingcap/kvproto/pkg/pdpb._PD_WatchGlobalConfig_Handler({0x2dbf240?, 0xc0007e2a30}, {0x3b5de68, 0xc001430a20})                                                                                                                                      │
│     /go/pkg/mod/github.com/pingcap/[email protected]/pkg/pdpb/pdpb.pb.go:10264 +0xd3                                                                                                                                                 │
│ github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1({0x2dbf240?, 0xc0007e2a30?}, {0x3b5de68?, 0xc001430a20?})                                                                                                                              │
│     /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:71 +0x89                                                                                                                                           │
│ github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1({0x2dbf240, 0xc0007e2a30}, {0x3b5e9a8?, 0xc0019ecf00}, 0xc000770240?, 0xc0007ca0a0)                                                                               │
│     /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 +0x109                                                                                                                                                              │
│ github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1({0x2dbf240?, 0xc0007e2a30?}, {0x3b5e9a8?, 0xc0019ecf00?})                                                                                                                              │
│     /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:74 +0x6f                                                                                                                                           │
│ go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1({0x2dbf240, 0xc0007e2a30}, {0x3b5e9a8, 0xc0019ecf00}, 0xc0019f3000?, 0xc0007ca0a0)                                                                                                            │
│     /go/pkg/mod/go.etcd.io/[email protected]/etcdserver/api/v3rpc/interceptor.go:237 +0x483                                                                                                                                   │
│ github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1({0x2dbf240, 0xc0007e2a30}, {0x3b5e9a8, 0xc0019ecf00}, 0xc0014309c0, 0x38bf198)                                                                                                           │
│     /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:79 +0x1a3                                                                                                                                          │
│ google.golang.org/grpc.(*Server).processStreamingRPC(0xc001a12f00, {0x3b64f60, 0xc001a13380}, 0xc001a2ff00, 0xc0005a6e70, 0x4df1820, 0x0)                                                                                                                     │
│     /go/pkg/mod/google.golang.org/[email protected]/server.go:1244 +0xdc6                                                                                                                                                                                          │
│ google.golang.org/grpc.(*Server).handleStream(0xc001a12f00, {0x3b64f60, 0xc001a13380}, 0xc001a2ff00, 0x0)                                                                                                                                                     │
│     /go/pkg/mod/google.golang.org/[email protected]/server.go:1317 +0x9de                                                                                                                                                                                          │
│ google.golang.org/grpc.(*Server).serveStreams.func1.1()                                                                                                                                                                                                       │
│     /go/pkg/mod/google.golang.org/[email protected]/server.go:722 +0x98                                                                                                                                                                                            │
│ created by google.golang.org/grpc.(*Server).serveStreams.func1                                                                                                                                                                                                │
│     /go/pkg/mod/google.golang.org/[email protected]/server.go:720 +0xea                                                          

What version of PD are you using (pd-server -V)?

[root@upstream-pd-0 /]# /pd-server -V
Release Version: v7.3.0-alpha
Edition: Community
Git Commit Hash: 4db1735
Git Branch: heads/refs/tags/v7.3.0-alpha
UTC Build Time: 2023-07-27 11:36:08

[fubinzh]

/found automation

[fubinzh]

/severity critical

[HuSharp]

fixed by #6866