Draft: Add member replace e2e test to repro issue 17052

[ZhouJianMS]

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

[k8s-ci-robot]

Hi @ZhouJianMS. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ahrtr]

I reproduced two issues based on this PR,

• "member count is unequal". It's caused by a known issue /members peer handler is not guarded with linearizable check #16687. When starting the newly added member, it gets members from one of its peer, but the /members doesn't go through raft, so the peer might haven't finish applying the previous conf change.
  • Added more log message in Added some log messages for better diagnosis #17113 to discover this issue.
• "etcdserver: Peer URLs already exists". Still need investigation.

[fuweid]

When I use taskset to use few cores during testing, the test run into the etcdserver: Peer URLs already exists. I think it might be related to txReadBuffer. That cache wasn't be updated. The batchTxBuffer only updates cache for Put. I think we can use tx.Commit during removing member.

[fuweid]

If the client selects that member which is going to be removed, it might return error because that server stops.

client won't retry if the error is code = Unavailable

[ZhouJianMS]

I follow the guide in https://etcd.io/docs/v3.5/tutorials/how-to-deal-with-membership/ to remove member. Which saying is correct?

[fuweid]

I run into this issue in my local one time.

etcd/server/etcdserver/server.go

Line 1115 in dfdffe4

go s.stopWithDelay(10*100*time.Millisecond, fmt.Errorf("the member has been permanently removed from the cluster"))

1 second might good enough to return response. never mind.

[ahrtr]

If the client selects that member which is going to be removed, it might return error because that server stops.

Yes, actually there are two "issues" (one real issue, one improvement) in the test cases:

• [Issue] The client might connect to the member which will be removed, so the client might get an error response.
• [Improvement] We should use client sdk instead of etcdctl to communicate with etcdserver directly. Otherwise when the client failed, it can only see etcdctl 's log instead of etcdserver's log

Actually I already updated the # 1 above. Let me deliver a PR to fix it.

[ahrtr]

FYI. add "EXPECT_DEBUG=true" can get the detailed etcdserver's log,

EXPECT_DEBUG=true go test -run TestMemberReplaceMultiple -v

[ahrtr]

That cache wasn't be updated.

Exactly, thanks for the quick investigation. Let me fix it together with the PR mentioned in #17100 (comment)

[ahrtr]

Resolved in #17119.

Thanks @ZhouJianMS and @fuweid