In addition to classic replication using createHistoryStream, some Scuttlebutt clients implement a more efficient form of replication known as Epidemic broadcast tree replication. This is often referred to by the abbreviation EBT. The implementation of EBT used in Scuttlebutt is loosely based on the push-lazy-push multicast tree protocol, more commonly known as the Plumtree protocol [1].
+
Session Initiation
+
An EBT session may be initiated once two peers have completed the secret handshake and have established their respective box streams. The peer who acted as the client during the secret handshake takes on the role of the requester, sending an ["ebt", "replicate"] request to the connected peer.
The peer who acted as the server during the secret handshake takes on the role of the responder. After having received the replicate request, the responder first validates the arguments to ensure that the version is 3 and the format is "classic". If either of those values are incorrect, the responder terminates the stream with an error.
+
Vector Clocks
+
The responder then sends a vector clock (also known as a "note" or "control message") to the requester. The vector clock takes the form of a JSON object with one or more key-value pairs. The key of each pair specifies a Scuttlebutt feed identified by the @-prefixed public key of the author. The value of each pair is a signed integer encoding a replicate flag, a receive flag and a feed sequence number.
+
+
+
The requester terminates the stream with an error if any of the received feed identifiers or encoded values are malformed. If the received vector clock is valid, the requester can proceed with decoding the values.
+
The value in each key-value pair of a vector clock encodes a maximum of three data points: a replicate flag, a receive flag and a sequence number. A negative value (usually -1) signals that the responder does not wish to replicate the associated feed, neither sending nor receiving messages. In this scenario, the replicate flag is set to false and both the receive flag and sequence number are irrelevant.
+
A positive value signals that the responder wishes to replicate the associated feed. If the value is positive it should be decoded as follows. First, the JSON number is parsed and converted to a signed integer. Then, the rightmost (lowest order) bit of the number is interpreted as a binary flag with 0 equal to true and 1 equal to false. This flag is referred to as the receive flag. Next, a sign-extending right shift (also called arithmetic right shift) by 1 bit is performed on the binary number, therefore discarding the rightmost (lowest order) bit. The remaining number is then interpreted as a sequence number for the associated feed.
+
If the receive flag is set to true, the peer who sent the vector clock wishes to receive messages for the associated feed. The decoded sequence number defines the latest message held by the peer for that feed.
+
Encoding of a vector clock value involves reversing the steps outlined above. If the peer does not wish to replicate a feed, the value is simply set to -1. Otherwise, the latest sequence number of the associated feed is stored as a signed integer and an arithmetic left shift is performed. The rightmost (lowest order) bit is then set according to the replicate flag as described previously.
+
+
+
+
Encoded
+
Decoded
+
+
+
Replicate flag
+
Receive flag
+
Sequence
+
+
+
+
+
-1
+
False
+
Irrelevant
+
Irrelevant
+
+
+
0
+
True
+
True
+
0
+
+
+
1
+
True
+
False
+
0
+
+
+
2
+
True
+
True
+
1
+
+
+
3
+
True
+
False
+
1
+
+
+
12
+
True
+
True
+
6
+
+
+
450
+
True
+
True
+
225
+
+
+
+
The requester then sends their own vector clock to the responder. At this point, the initial exchange of vector clocks is complete and both peers may begin sending messages at will. Updated vector clocks may continue to be sent by both peers at any point during the session. These updated clocks may reference a subset of the feeds represented in the initial vector clock, or they may reference different feeds entirely. This provides a means for responding to state changes in the local database and follow graph.
An EBT session may be terminated by either peer at any point, either by sending an error response or by closing the stream. If no error has occurred, the stream is closed when a peer wishes to conclude the session (as described in the Source example of the RPC protocol section above).
+
Request Skipping
+
EBT implementations rely on a mechanism known as request skipping to lower bandwidth overhead and increase replication efficiency. Each peer stores the vector clocks they receive from remote peers; these may be held in memory and persisted to disk to allow later retrieval. When a subsequent EBT session is initiated between peers, each peer first checks the stored vector clock of their remote peer before calculating an updated vector clock to be sent. If the latest locally-available sequence of a feed from the remote peer's vector clock is the same as the sequence in the saved vector clock for that peer, that feed is left out of the new vector clock in the outgoing request (hence the name request skipping). This provides a mechanism for limiting the total number of bytes to be sent over the wire.
+
+
The stored vector clock for the remote peer may differ from their current vector clock. In that case, the remote peer will include the updated feed in their request and the local peer will respond by sending an additional partial vector clock including their sequence for that feed. Once both sides have exchanged their sequence for a particular feed, replication of messages in that feed may occur.
+
Vector Clock Partitioning
+
In order to further increase efficiency when connecting to multiple peers, feeds for which the local peer would like to receive updates are only sent to one peer at a time (in the outbound vector clock). A timeout may be used to request the feed from an alternate peer if no updates are available from the initial peer. In this way, the total set of requested feeds is spread across multiple peers.
+
Fallback Mechanisms
+
EBT is the preferred means for peers to exchange messages. However, not all Scuttlebutt clients support EBT replication. In the case that only one of two connected peers support EBT, both peers may instead fallback to using createHistoryStream to exchange messages. There are several scenarios which may trigger initiation of createHistoryStream replication:
+
+
If the requester attempts to initiate an EBT session but the session is terminated with an error by the responder
+
If a createHistoryStream request is immediately sent by the client upon successful connection
+
If the client doesn’t attempt to initiate an EBT session for a certain amount of time
+
+
[1] Joao Leitao, Jose Pereira and Luis Rodrigues. 2007. Epidemic Broadcast Trees. In 2007 26th IEEE International Symposium on Reliable Distributed Systems (SRDS 2007), 301-310. https://doi.org/10.1109/SRDS.2007.27
+