Multi-master replication - implement or validate

[nlindq-maei]

In 2020--prior to the release of 3.4.0--Bron mentioned that multi-master replication was close:

2020-04-07 in response to a question:

No, not really. It'll mostly be fine, but it doesn't (yet) handle folder create/rename/delete safely.
[ .... ]
The plan in 3.4+ is to use the mailbox tombstone records to get the create/rename/delete to the same level of split-brain safety as the UIDs inside the mailbox have.

2020-12-14 in response to a question:

It's almost all good, the main problem is split brain recovery when you delete or rename folders - it could wind up reverting the change if a 'sync user' gets triggered by the other end.

The last piece of the puzzle (yeah, 10 years later!) is going to be having proper tombstone records in the mailboxes.db including name history for each mailbox, so that we know whether a mailbox has been added or deleted. The mailboxes-by-uuid work, which should be landing on master early next year, is going to add that.

So yeah, I wouldn't do it just yet. Our load balancing involves shutting down the master slot and running a sync_client to pick up any remaining events before switching configs and bringing the other one up.

After the 3.4.0 release (2021-04-20), Ellie replied to someone asking about the status:

If it were clear to me that master-master replication was now safe, I would have listed it as a feature in the release notes. 😉

I'm not directly certain what is or isn't remaining to make this work, but, it's worth observing that even if the right code is now in place, afaik nobody has used it in this way, so I'm not going to tell people they should start relying on it.

If you're willing to experiment on the side, it might work with bugs (which can be fixed once found), or it might not be complete (and such limitations can be addressed once identified). Either way, feedback from actual usage would be very useful!

But for production purposes, I would assume that 3.4 does not support master-master replication, and stick with the traditional replication schemes.

So, given the changes in storage (organizing by UUID, etc.) has there been any evaluation of the state of multi-master replication? Is it theoretically possible in any or all of 3.4.x, 3.6.x, 3.8.x, 3.10.x but untested? Are there any outstanding roadblocks? What kind of testing plan would be required to validate it?