[release-21.0] Fix panic in vttablet when closing topo server twice (#17094)

[release-21.0] Fix panic in vttablet when closing topo server twice (#17094) #28304