Experiment to try to use forgetful blooms to dedup gossip #280
Conversation
There was a problem hiding this comment.
It looks like you are applying the filters on received data and avoiding having to handle it and thus if we have already handled it we avoid gossiping to our peers a subsequent time.
But our peer list will change, so I would not have thought it ideal to assume because my peer has previously seen it, we dont need to gossip it to any of our current peers.
I had thought, originally, your intent was to apply the filter to outgoing data on a per peer basis. In that model you would always attempt to gossip data that you receive but will avoid sending it to peers you previously gossiped that same data too. This could be achieved by using the filter on at the group_worker:send level.....
| {_, Pids} = lists:unzip(connections(all, State)), | ||
| %% track we've seen this now | ||
| forgetful_bloom:set(Bloom, {Key, Data}), |
There was a problem hiding this comment.
At this point ( in the gossip server ) we do not know whether the send will be successful. For example a stream may not yet be associated with the worker
No description provided.