go/worker/compute/executor/committee: Support backup proposers

[peternose]

All committee members can now propose batches, but each has its own priority which dictates the time after which a worker can propose a new batch. The consensus and every (fair) node take this priority into account when processing a proposal or scheduling a batch.

Consensus:

• When consensus accepts a new commitment, it checks the priorities of all commitments in the pool. If the new commitment has a lower priority, it is rejected, if it is the same, it is added to the pool, and if the priority is higher, the pool is reset.
• If the consensus detects a discrepancy, immediate resolution only occurs for the primary proposer (highest priority). Others need to wait for a consensus timeout. This prevents nodes from submit two conflicting commitments that would trigger immediate resolution of non-primary proposal. Once all commitments for the currently highest priority are submitted,
  resolution or discrepancy resolution is initiated.

Workers:

• Each worker, when generating a commitment, considers proposal's priority. This means that a node will publish a commitment from a proposer with priority N only after N amounts of time has passed. The same applies to submissions.

Changes:

• State transitions were simplified to support processing multiple proposals with different priorities inside the same round.
• The executor worker doesn't block the common node anymore.
• Proposals with too many transactions are not discarded on the P2P layer but instead treated as failures as this is a protocol
  violation from the transaction scheduler.

TODO:

• Improve pool tests.
• Improve executor committee tester.
• Check if we still need timeouts.

[codecov]

Codecov Report

Merging #5354 (88ff0dc) into master (a7ee9ce) will increase coverage by 0.33%.
Report is 1 commits behind head on master.
The diff coverage is 81.91%.

❗ Current head 88ff0dc differs from pull request most recent head 2a53cf8. Consider uploading reports for the commit 2a53cf8 to get more accurate results

@@            Coverage Diff             @@
##           master    #5354      +/-   ##
==========================================
+ Coverage   66.23%   66.57%   +0.33%     
==========================================
  Files         528      534       +6     
  Lines       56237    56288      +51     
==========================================
+ Hits        37251    37471     +220     
+ Misses      14526    14380     -146     
+ Partials     4460     4437      -23     

Files	Coverage Δ
go/consensus/cometbft/apps/roothash/api/block.go	100.00% <100.00%> (ø)
go/consensus/cometbft/apps/roothash/genesis.go	73.33% <100.00%> (ø)
go/consensus/cometbft/apps/roothash/query.go	86.20% <100.00%> (-6.90%)	⬇️
go/consensus/cometbft/apps/roothash/state/state.go	73.28% <100.00%> (-1.45%)	⬇️
go/consensus/cometbft/apps/scheduler/shuffle.go	69.71% <ø> (ø)
go/consensus/cometbft/roothash/roothash.go	67.47% <100.00%> (-1.83%)	⬇️
go/p2p/api/convert.go	73.33% <100.00%> (ø)
go/registry/tests/tester.go	92.48% <100.00%> (ø)
go/roothash/api/commitment/executor.go	69.23% <ø> (-2.20%)	⬇️
go/roothash/api/commitment/votes.go	100.00% <100.00%> (ø)
... and 34 more

... and 47 files with indirect coverage changes

[pro-wh]

I'm mostly reading through the design at this point.

"priority" is a little tricky to understand just by the name of it. the lowest priority number is least preferred schedule when a node considers multiple schedules, and the highest priority number is the most preferred. meanwhile, the lowest priority number is meant to be the one that gets produced first. thus the node with the highest priority number is also the one that's least urgently doing the scheduling.

the system of waiting to publish a schedule based on the priority number is noted not to be enforced. it feels like following this schedule is not rational though. I'm guessing rational would be either always schedule right away to have control over tx ordering and/or get rewards; or never schedule to save compute+bandwidth.

what's the amount of time per 1 priority? I'm seeing a 2 second constant in the code? can a committee routinely finish in that amount of time? the design says the pool resets when it sees a higher priority commitment. we wouldn't want this repeatedly to have a few fast nodes commit, then slower nodes not finish before the next time interval and a new schedule comes out.

do nodes get these commitments through mechanisms other than reading them from the roothash app in the consensus layer? they communicate over p2p right? that feels like it would lead to the possibility of nodes coming to different conclusions as messages propagated throughout the committee. e.g. if the last needed commit for priority i and the proposed schedule for priority i+1 are being propagated at the same time, some nodes can see priority i finalize while some nodes can see it reset and start on priority i+1

[peternose]

"priority" is a little tricky to understand just by the name of it

I can rename to token one something similar. Will think about that.

I'm guessing rational would be either always schedule right away to have control over tx ordering and/or get rewards; or never schedule to save compute+bandwidth.

That is not true.

• A node should not schedule right away if it's not its turn, as time and energy would be invested in creating a proposal that will most likely be superseded by a primary transaction scheduler (or backup schedulers with higher pripority).
• A node can decide never to schedule, i.e., never be a backup scheduler. However, when it is its turn to propose, the node will be penalized if it decides to never schedule.

what's the amount of time per 1 priority? I'm seeing a 2 second constant in the code?

Yes, currently hard-coded (could changed to a parameter).

can a committee routinely finish in that amount of time?

The committee doesn't need to finish in that time. The proposer which turn it is should finish in that time and publish a proposal via P2P. And once others receive the proposal, they have time until the consensus timeout fires to finish and accept the proposal.
Actually, even the proposer doesn't need to finish in that time, as we want that the next in line starts preparing a batch before the previous finishes.

the design says the pool resets when it sees a higher priority commitment. we wouldn't want this repeatedly to have a few fast nodes commit, then slower nodes not finish before the next time interval and a new schedule comes out.

Fast node commits don't bother other committee nodes as they will start executing proposals only when the time is right, So we can have 100 fast commits in the first second, but the committee nodes will do nothing if the primary proposal hasn't committed yet.
From the primary proposal's point of view, nothing should change. Once other nodes see his proposal, they will cancel everything and start working on that.

do nodes get these commitments through mechanisms other than reading them from the roothash app in the consensus
layer? they communicate over p2p right?

Commitments are over consensus and P2P layer (just for discrepancy detection), proposals are over P2P.

that feels like it would lead to the possibility of nodes coming to different conclusions as messages propagated throughout the committee. e.g. if the last needed commit for priority i and the proposed schedule for priority i+1 are being propagated at the same time, some nodes can see priority i finalize while some nodes can see it reset and start on priority i+1

Yes, nodes could have different conclusions, depending on which proposals they have received. But they always build on proposals with higher priority until the round is finalized or discrepancy detected. Those events come from the consensus, so it is absolute truth and every node should see that.

[pro-wh]

could you describe the protocol in more detail? it sounds like I'm missing some functionality to discourage dishonesty in the priority based waiting scheme.

time and energy would be invested in creating a proposal that will most likely be superseded by a primary transaction scheduler (or backup schedulers with higher priority).

can the primary transaction scheduler supersede it? it sounded like "primary" refers to the node with priority 1, is that the case? a node that doesn't follow the prescribed timing scheme could win by sending its schedule right away, as any node other than the primary has higher priority. and if other nodes are honest, it'll be a while before they get around to scheduling, so our dishonest node's proposal would reasonably get finalized

when it is its turn to propose, the node will be penalized if it decides to never schedule.

what does it have to do to avoid this penalty? does it have to get its schedule out in the right order? I'm guessing no, as there's no proof of who received what in what order. maybe that's fine, as stealing a higher priority node's work is complicated by the signature being done in a TEE

The committee doesn't need to finish in that time [the per-priority time]. The proposer which turn it is should finish in that time and publish a proposal via P2P. And once others receive the proposal, they have time until the consensus timeout fires to finish and accept the proposal.
Actually, even the proposer doesn't need to finish in that time, as we want that the next in line starts preparing a batch before the previous finishes.

ah and honest nodes are supposed to cancel their scheduling when they see a proposal come out?

Commitments are over consensus and P2P layer (just for discrepancy detection), proposals are over P2P.

nodes could have different conclusions, depending on which proposals they have received. But they always build on proposals with higher priority until the round is finalized or discrepancy detected. Those events come from the consensus

could a proposal cause a node to reset a lower priority pool, then consensus tell the node that actually that lower priority schedule was finalized?

[peternose]

Could you describe the protocol in more detail?

Original Protocol:

Committee:

• Every epoch, the scheduler elects a committee. Elected nodes can be either workers, backup workers, or both.
• In every round, one worker is selected to be a proposer. In round N, the selected worker is the one at position N % len(workers).
• When the committee changes, an empty epoch transition block is produced.

Commitment Collection:

• The pool is reset every time a new block is produced.
• The pool stores all valid commitments.
  • A commitment is considered valid if the round matches, the node is a committee member, and the signatures match, among other criteria.
  • Commitments from backup workers are stored even if the backup workers have not been activated yet (the pool is still in the discrepancy detection mode).

Block Finalization:

• The Roothash app attempts to finalize a block every time a new executor commitment is added to the pool (unforced), or if the round times out (forced).
  • If a discrepancy is detected, the discrepancy resolution mode is activated, and finalization is retried.
  • If there are not enough commitments, finalization fails.
  • Otherwise, the round fails, and an empty round failed block is produced.

Consensus Round Timeout:

• If too many consensus blocks have passed since the last block was produced, a committee node that is not a proposer can publish a transaction which will fail the round if the pool is empty.

Commitment Collection Timeout:

• Is set to infinity when the pool is reset.
• Is increased every time block finalization fails because the pool is still waiting for commitments, or if a discrepancy has been detected (the backup workers should get some time to perform the computation).
• Is increased only after a valid commitment is added to the pool.

Pool:

• The pool processes commitments in two modes.
  • In the discrepancy detection mode, the pool checks if all commitments from the workers have been received and if the commitments match.
    • If that is true, the pool is in agreement, and the block can be finalized.
    • If two commitments differ or if there are too many failures, the discrepancy resolution mode is activated.
    • If the round times out and the proposer's commitment hasn't been received, an error is returned.
    • If the round times out, a few stragglers are allowed, and it is checked if the majority has been reached.
    • Otherwise, it waits for more commitments.
  • In the discrepancy resolution mode, the pool takes all commitments from the backup workers.
    • If the majority (total/2+1) of the commitments are the same as the proposer's commitment, the pool is in agreement, and the block can be finalized (and some nodes punished).
    • If the round times out and the proposer's commitment hasn't been received, an error is returned.
    • If the round times out and the majority hasn't been reached, an error is returned (but no one is punished).
    • Otherwise, it waits for more commitments.

New Protocol:

Terminology:

• Priority can range from 0 to infinity, with 0 being the best priority.
• Pool priority is the priority of the worker who prepared a proposal to which all commitments in the pool belong.
• Commitment priority is the priority of the worker who prepared a proposal to which the commitment belongs.

Committee:

• All workers act as proposers, but they have different priorities.
  • In round N, the worker closer to position N % len(workers) has better priority.

Commitment Collection:

• The pool prioritizes commitments with better priority and stores only commitments with the same priority.
• When a new commitment is received, its priority is computed.
  • If the pool is empty, the new commitment is added to the pool.
  • If the pool is not empty, priorities are checked:
    • If the pool priority is lower, the new commitment is discarded.
    • If the pool priority is higher, the pool is reset, and the new commitment is added to the pool.
    • If they match, the new commitment is added to the pool.

Pool:

• The pool processes commitments in two modes.
  • In the discrepancy detection mode:
    • If two commitments differ, the pool transitions to the discrepancy resolution mode only if the pool has the best priority. Other wise, it waits for the round timeout.
      • This ensures that the backup workers (those with priority > 0) don't skip the queue by triggering the discrepancy resolution immediately.
  • In the discrepancy resolution mode:
    • The pool's priority is now locked and cannot be changed anymore.

Nodes:

• Nodes store proposals from all workers received via the P2P network.
• Workers respect the priority queue and publish a commitment for a proposal with priority N only after N amount of time has passed since the last block.
• It can happen that a node computes multiple commitments in the same round. However, every new commitment will have a better priority.

it sounded like "primary" refers to the node with priority 1, is that the case?

Primary refers to the node with priority 0, where 0 is the best priority.

can the primary transaction scheduler supersede it?

Commitment belonging to a proposal from a worker with better priority can supersede worse commitments, but only in the discrepancy detection mode. So the primary proposer can publish a commitment that will supersede anyone.

a node that doesn't follow the prescribed timing scheme could win by sending its schedule right away as any node other than the primary has higher priority. and if other nodes are honest, it'll be a while before they get around to scheduling, so our dishonest node's proposal would reasonably get finalized

Anyone can ignore the timing scheme and share its proposal via P2P network. But the problem arises at honest workers, as they will figure out that they have received the proposal too soon and will wait until they publish a commitment for that proposal to the consensus. Meanwhile, they will probably receive the proposal from the primary proposer, submit a commitment for that one and supersede the dishonest commitment (if there is one in the pool and waiting for the majority).

ah and honest nodes are supposed to cancel their scheduling when they see a proposal come out?

Honest workers don't propose if they see a proposal with better priority.

could a proposal cause a node to reset a lower priority pool, then consensus tell the node that actually that lower priority schedule was finalized?

It could happen that a node would receive a proposal with better priority, create a commitment and publish it to the consensus only to be rejected because a block was already finalized from a proposal with worse priority. That is ok, as that would mean that the node with better priority needed too long to publish a proposal, so the backup proposers stepped in.

[pro-wh]

0 being the best priority

ah that clears things up a lot, thanks

[peternose]

I'm fixing backup proposers, and I've come across the following questions:

• Backup proposers will never receive all the votes if the primary proposer is offline. This means they always have to wait for a timeout. On mainnet, the settings are RoundTimeout = 5 and SchedulerTimeout = 2. The new solution is therefore slower than the original. Options:
  • Shorten timeouts (not sure about that).
  • Allow stragglers before the timeout.
  • Use different committee for every backup proposer, which does not include workers with better priorities (e.g. extend the worker set, and use m out of n workers only).
• The RAK signature is a signature of the ComputeResultHeader, which includes the round but does not include the proposal data. We probably should also sign the BatchHash. This way the inputs, current state, and the output of the computation are all signed by the enclave. Not trivial, as we need to update runtimes also.
• Currently, a worker can submit a commitment even though the proposer is offline. Therefore, a proof would be needed. Options:
  • Add signed proposal to the commitment (round and scheduler id are already included, the signature and batch hash not).
  • Support multiple pools.
  • Queue commitments until the scheduler submits its own commitment.
  • Let proposers collect commitments off-chain (not sure about that).

Any thoughts?

[kostko]

On mainnet, the settings are RoundTimeout = 5 and SchedulerTimeout = 2.

Which runtime are you looking at? It should both be 2 for Sapphire.

Allow stragglers before the timeout.

This roughly seems to make sense, the downside being that one could DoS the straggler so it is slow to prove the discrepancy. But having a timeout just gives the straggler slightly more time. The other implication is that liveness evaluation could cause such stragglers to be penalized more.

The RAK signature is a signature of the ComputeResultHeader, which includes the round but does not include the proposal data. We probably should also sign the BatchHash. This way the inputs, current state, and the output of the computation are all signed by the enclave. Not trivial, as we need to update runtimes also.

The IO root includes both inputs (transactions) and outputs (results and events), so this should already be the case?

Currently, a worker can submit a commitment even though the proposer is offline. Therefore, a proof would be needed.

What are the tradeoffs of these different options?

[peternose]

Which runtime are you looking at? It should both be 2 for Sapphire.

Was looking at the genesis file. Not sure which runtime that was.

What are the tradeoffs of these different options?

• Larger commitment (add signed proposal to the commitment).
• Complex code, multiple timeouts, ... (support multiple pools).
• Easy (queue commitments until the scheduler submits its own commitment).
• Additional code, not sure of other downsides. I guess this was implemented before, when one could post multiple commitments (let proposers collect commitments off-chain).

[kostko]

Let's go with queuing commitments until the scheduler submits its own commitment. The round cannot be finalized until we have the proposer commitment anyway.

[peternose]

Still have to fix some byzantine tests, but the latest changes are:

• Finalize runtime rounds/blocks during the EndBlock phase.
• Store votes in the pool instead of commitments to avoid unnecessary hashing.
• Remove round and runtime from the pool to eliminate data duplication.
• Move the committee to the runtime state since it is shared between liveness statistics and the pool.
• Remove the transaction scheduler timeout transaction as it is no longer needed.
• Adjust round timeouts. Schedulers should have unlimited time to submit a commitment, workers RoundTimeout blocks, and backup workers 1.5 * RoundTimeout blocks. When a higher-ranked scheduler commits, the timeout for workers is reset.
• Prevent workers from submitting commitments during discrepancy resolution to avoid potential manipulation of liveness statistics.
• Change SchedulerTimeout from int64 to time.Duration (should not break runtimes).
• Rename CurrentBlock and CurrentBlockHeight in runtime state to LastBlock and LastBlockHeight.
• Prohibit schedulers from submitting executor commitment failures.
• Suspend runtimes if there is no committee, even if the DebugDoNotSuspendRuntimes flag is set.
• Update tests.

[ptrus]

Would be nice to have a short ADR with a formal description and pros & cons. But since it's already implemented, no need to add it now retrospectively IMO.

[peternose]

Non-formal description of pros/cons (comparing to the previous solution and settings):

Settings:

• allowed_stragglers: 1
• round_timeout: 2 blocks
• propose_batch_timeout: 2 blocks (before), 2 seconds (now)

Pros:

• If the primary scheduler is offline, the round will finalize with 2 seconds delay. The penalty will be 0 or 1 consensus block (0 or 6 seconds). Currently, the penalty is 3 consensus blocks (18 seconds), i.e. 2 blocks for proposer timeout and 1 block for proposer timeout transaction.
• If the primary scheduler and the first/one backup scheduler are offline, the penalty will be 2 or 3 consensus blocks (18 seconds), as the second backup scheduler will fire after 4 seconds and discrepancy resolution will start after 2 consensus blocks.
• No need for proposer timeout transaction.

Cons:

• A scheduler with higher rank could intentionally propose after schedulers with lower rank to force them to do extra work and to delay rounds. Not sure, if there are any gains here, as the rounds will be delayed and less fees collected.
  • If this becomes a problem we can fight back with statistics and slashing, by lowering the timeout for overriding the current scheduler, with limits how many overrides can be done, ...
• If worker's clock is not synchronized, he could do extra work by submitting commitments for proposals with lower rank which turn hasn't come yet.
• Workers need to exchange and store more proposals via P2P network. However, some can be dropped when a scheduler with higher rank commits.
• Pool needs to store more commitments/votes. However, some can be dropped when a scheduler with higher rank commits.

[peternose]

Cons:

• A worker can compute multiple commitments for the same round, if there are multiple overrides. If schedulers collide, the inputs and transactions can be the same, forcing the worker to run its enclave multiple times on the same data (sounds like a replay attack with possibility of side channel attacks). But I guess this is possible even now, as failed rounds have the same roots. Don't see any danger here.

[kostko]

forcing the worker to run its enclave multiple times on the same data (sounds like a replay attack with possibility of side channel attacks)

Yeah use cases that care about this should perform a two-step process, e.g. in the first step only commit something to storage and then only after it is finalized, perform the second step based on the data in storage.

Also to limit this, we could only allow one run per round/scheduler (until restart).

[ptrus]

This is because the "Early discrepancy detection" would already catch the discrepancy if p.HighestRank=0 right, so we know that this means that the primary scheduler has not yet committed?

Maybe include this in a comment (all the facts already assumed here due to the early discrepancy checks and not rechecked), as this was not immediately clear to me.

[peternose]

This is because the "Early discrepancy detection" would already catch the discrepancy if p.HighestRank=0 right.

Yes. If the highest rank is 0, this means that the primary scheduler already committed and the early discrepancy detection handled this case. Therefore, this if cannot happen.

so we know that this means that the primary scheduler has not yet committed

Yes. If this if is true, we know that the primary scheduler hasn't committed but some other backup scheduler has. And backup schedulers need to wait for round timeout, as mentioned above.

Will update the comment.

[pro-wh]

reviewed, focusing on changes to the central part in the pool system. looks to be implemented as described