Handling truncation events within coprocessors

[graphcareful]

I am thinking of a way to handle truncation events within coprocessing. The reason they must be explicitly addressed is due to the fact that v/coproc does not depend on v/cluster and therefore cannot choose to follow source topics via an offset that is guaranteed to be committed cluster-wide.

Therefore we can expect that truncations may occur within offset ranges that coproc has already operated on. This means stateful effects have been produced on data that is now going to be erased and these stateful effects must also be un-done. Which means the materialized topics must also be truncated at the point the source topic was when the truncation occurred (or earlier, since duplicates are tolerated in materialized logs).

The non-trivial portion of the issue is to discover where to truncate materialized logs. In the simplest case if a materialized log has a 1:1 relationship with the source topic, we can expect all offsets to be the same, therefore we can truncate exactly where the source topic has been truncated. But this is not the case, as coprocessors may have any arbitrary logic within them, have any type of relationship with the source topic (2:1, 15:1, etc) and may even filter out records if desired.

The most naive solution would require keeping track of every effect made by reading the source topic and observing its effect. For example, if we read offsets 100-1000 and write 500-2000, if we truncate the source topic to 100 we must truncate the materialized topic to 500. However all of these ranges must be kept for each action since the beginning of time because we don't know our lower-bound i.e. everything before the raft committed offset that could be thrown away but won't because coproc is unaware of it.

However having coproc depend on cluster might involve a big refactor. This proposal may require moving a big portion of code and other options should be evaluated and weighed against this.

[dotnwat]

Thanks for putting this together! I think it makes sense to revise the post above as part of addressing feedback.

I am thinking of a way to handle truncation events within coprocessing. The reason they must be explicitly addressed is due to the fact that v/coproc does not depend on v/cluster and therefore

Can you separate out the implementation detail from the fundamentals? For example, the reason co-proc must explicitly address truncations (I'm assuming) is independent from particular code dependencies between coproc/cluster. The salient question is then: why is truncation important for co-proc, and how do things work (or break) today? Is this prompted by a new feature, or an existing issue that is being address?

cannot choose to follow source topics via an offset that is guaranteed to be committed cluster-wide.

This sounds like a solution, but I'm not sure what the problem is. Above it's stated that truncations in co-proc are needed. But this quoted statement "follow[ing] source topics via an offset that is guaranteed to be committed cluster-wide" seems unrelated to truncation. What is the connection?

Therefore we can expect that truncations may occur within offset ranges that coproc has already operated on. This means stateful effects have been produced on data that is now going to be erased and these stateful effects must also be un-done. Which means the materialized topics must also be truncated at the point the source topic was when the truncation occurred (or earlier, since duplicates are tolerated in materialized logs).

Can you provide an example of the scenarios of interest related to this description? Walking through the details of this paragraph slowly seems to be key to this proposal.

The non-trivial portion of the issue is to discover where to truncate materialized logs. In the simplest case if a materialized log has a 1:1 relationship with the source topic, we can expect all offsets to be the same, therefore we can truncate exactly where the source topic has been truncated. But this is not the case, as coprocessors may have any arbitrary logic within them, have any type of relationship with the source topic (2:1, 15:1, etc) and may even filter out records if desired.

This paragraph would be useful at the beginning of the design document where it's unclear what the problem is.

The most naive solution would require keeping track of every effect made by reading the source topic and observing its effect. For example, if we read offsets 100-1000 and write 500-2000, if we truncate the source topic to 100 we must truncate the materialized topic to 500. However all of these ranges must be kept for each action since the beginning of time because we don't know our lower-bound i.e. everything before the raft committed offset that could be thrown away but won't because coproc is unaware of it.

This makes sense. Like the paragraph before, it'd be useful at the beginning of the document.

However having coproc depend on cluster might involve a big refactor. This proposal may require moving a big portion of code and other options should be evaluated and weighed against this.

I don't think that this discussion has stated what the proposed solution is. It's clear there is an implementation challenge related to coproc/cluster dependencies, but the important bit seems to be "cannot choose to follow source topics via an offset that is guaranteed to be committed cluster-wide." and it's not clear to me from reading this how being able to follow a topic via a committed offset solves the problem of handling truncation events in co-proc. I think the 2nd paragraph attempts to do that, but it needs an example or needs to be expanded.

[emaxerrno]

@graphcareful it's probably useful to actually use the rfc format which is designed to extract out motivations for the person reading the proposal