Implement Timeout Mechanism for Replica Termination

[andylizf]

Currently, replicas can get stuck during shutdown, requiring manual intervention into code logic. This is very unfriendly to users.

We propose implement a timeout mechanism for replica termination. Replicas that do not shut down within the specified timeout should transition to a FAILED_SHUTDOWN state. These failed replicas can then be terminated using the existing terminate_replica command (#4032).

          Instead of adding a `force` terminating option, we decided to implement a timeout mechanism for terminating replicas so that users can tear down those exceed the timeout and end up in `FAILED_SHUTDOWN` status via previously mentioned `terminate_replica` introduced by #4032.

Originally posted by @andylizf in #4059 (comment)