Data inconsistency caused by TXN with a Range on a compacted revision

[davidhenrygao]

I'm a new user of etcd，and I setup the etcd's cluster by docker on the same machine.
etcd's cluster info as below:

$ etcdctl --endpoints=127.0.0.1:23791,127.0.0.1:23792,127.0.0.1:23793 endpoint status -wtable
+-----------------+------------------+---------+---------+-----------+-----------+------------+
|    ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------+------------------+---------+---------+-----------+-----------+------------+
| 127.0.0.1:23791 | 97f72e8299c69562 |  3.5.15 |   25 kB |     false |         9 |        131 |
| 127.0.0.1:23792 | aea8ea6030d288e8 |  3.5.15 |   25 kB |      true |         9 |        131 |
| 127.0.0.1:23793 | 98b1342fbbf1abeb |  3.5.15 |   25 kB |     false |         9 |        131 |
+-----------------+------------------+---------+---------+-----------+-----------+------------+

when I try a txn operation as below，data inconsistency happen.

$ etcdctl --endpoints=127.0.0.1:23791,127.0.0.1:23792,127.0.0.1:23793 txn --interactive
compares:
value("foo") = "bar"

success requests (get, put, delete):
put fo foo
get foo --rev=45

failure requests (get, put, delete):

Error:  etcdserver: mvcc: required revision has been compacted

$ cluster etcdctl --endpoints=127.0.0.1:23791 get fo
fo
abc
$ cluster etcdctl --endpoints=127.0.0.1:23792 get fo
fo
foo
$ cluster etcdctl --endpoints=127.0.0.1:23793 get fo
fo
foo

$ cluster etcdctl --endpoints=127.0.0.1:23791 get fo -wjson
{"header":{"cluster_id":8027795201070719271,"member_id":10950272157435204962,"revision":71,"raft_term":9},"kvs":[{"key":"Zm8=","create_revision":57,"mod_revision":71,"version":9,"value":"YWJj"}],"count":1}
$ cluster etcdctl --endpoints=127.0.0.1:23792 get fo -wjson
{"header":{"cluster_id":8027795201070719271,"member_id":12585566857543715048,"revision":81,"raft_term":9},"kvs":[{"key":"Zm8=","create_revision":57,"mod_revision":81,"version":18,"value":"Zm9v"}],"count":1}
$ cluster etcdctl --endpoints=127.0.0.1:23793 get fo -wjson
{"header":{"cluster_id":8027795201070719271,"member_id":11002632744264117227,"revision":81,"raft_term":9},"kvs":[{"key":"Zm8=","create_revision":57,"mod_revision":81,"version":18,"value":"Zm9v"}],"count":1}

because revision 45 had been compacted, so txn success operations will fail in the middle, key "fo" original value is "abc", so the txn do nothing on node1, but it put the value "foo" on node2 and node3!!! And "revision" is different too.(I try it many times, so node1's revision is 71, node2's and node3's revision is 81.)

I just test it for learning. Is that test is illegal and not be allowed? Or is it a bug?

Because the txn is failed, my expectation is nothing will be done by the txn. If it is impossible, at least "put fo foo" should apply to all nodes to keep consistently. But it isn't.

On the other hand, I need some help to repair the inconsistent data. I restart the etcd cluster, but it doesn't work.

Thanks!

[ahrtr]

Thanks @davidhenrygao , and I reproduced this issue!

How to reproduce this issue

• Step 1: start a brand new 3 member cluster
• Step 2: Execute for i in {1..20}; do ./etcdctl put k$i v$i; done
• Step 3: Execute ./etcdctl compact 21
• Step 4: Execute ./etcdctl txn --interactive

compares:
value("k1") = "v1"

success requests (get, put, del):
put k2 foo
get k1 --rev=10

failure requests (get, put, del):

The client will get a **etcdserver: mvcc: required revision has been compacted** error.

• Step 5: Execute ./etcdctl get k2 against different endpoints, you will get different values,

$ ./etcdctl --endpoints=127.0.0.1:2379 get k2
k2
v2
$ ./etcdctl --endpoints=127.0.0.1:22379 get k2
k2
foo
$ ./etcdctl --endpoints=127.0.0.1:32379 get k2
k2
foo

Root cause

etcd server removes range requests from the TXN for endpoints the client isn’t connected to.

etcd/server/etcdserver/server.go

Lines 1967 to 1971 in 2c97110

	needResult := s.w.IsRegistered(id)
	if needResult || !noSideEffect(&raftReq) {
	if !needResult && raftReq.Txn != nil {
	removeNeedlessRangeReqs(raftReq.Txn)
	}

For example, if the client connects to member 1, then etcdserver removes the range request (get k1 --rev=10 in this example) from the TXN in member 2 and 3. Accordingly, member 1 applies failed due to checkRange's failures, but member 2 & 3 apply the TXN successfully because the range request was removed. Eventually it leads to the situation that different members have different data.

etcd/server/etcdserver/txn/txn.go

Lines 431 to 441 in 2c97110

	func checkRange(rv mvcc.ReadView, req *pb.RangeRequest) error {
	switch {
	case req.Revision == 0:
	return nil
	case req.Revision > rv.Rev():
	return mvcc.ErrFutureRev
	case req.Revision < rv.FirstRev():
	return mvcc.ErrCompacted
	}
	return nil
	}

Solution

The simplest solution is we don't remove range requests from TXN for any member. The side effect is that other endpoints (the client isn't connected to) will execute the unnecessary range operations.

To resolve the side effect above, we don't execute the range requests on other endpoints that the client isn't connected to; instead etcdservers only verify them to ensure all endpoints always execute consistent validation.

Impact

• Just searched the Kubernetes repo, and confirmed that kubernetes doesn't specify revision for range request in TXN. So Kubernetes isn't affected
• For non-Kubernetes usage... hopefully it's unlikely to be used this way in practice. @davidhenrygao Did you test this usage in your environment, or does your product actually use TXN this way?

[ahrtr]

And How can I repair the data inconsistency now? Is there any docs or tutorial?

If there is only one inconsistent member, then you just need to replace the member. Please see https://etcd.io/docs/v3.5/op-guide/runtime-configuration/#replace-a-failed-machine . You need to remove the data after removing a member.

If all members are inconsistent, then it's a little complicated. In that case, you have to use one member's data as the source of truth, and forcely create an one-member cluster, afterwards, add other members into the cluster (remember to remove other members' data before adding them into the cluster).

[serathius]

This is really awesome investigation. I think we should move this discussion to an issue. @ahrtr can you create issue with your findings?

[ahrtr]

@ahrtr can you create issue with your findings?

Sure.

[ahrtr]

Raised #18667

[davidhenrygao]

And How can I repair the data inconsistency now? Is there any docs or tutorial?

If there is only one inconsistent member, then you just need to replace the member. Please see https://etcd.io/docs/v3.5/op-guide/runtime-configuration/#replace-a-failed-machine . You need to remove the data after removing a member.

If all members are inconsistent, then it's a little complicated. In that case, you have to use one member's data as the source of truth, and forcely create an one-member cluster, afterwards, add other members into the cluster (remember to remove other members' data before adding them into the cluster).

@ahrtr Thanks! I try it later.