From b3ae8e5743ca63808b99e7d4300098eee37fed56 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Tue, 19 Sep 2023 15:01:57 -0700 Subject: [PATCH 01/28] Create nep-0509.md initial draft --- neps/nep-0509.md | 121 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 neps/nep-0509.md diff --git a/neps/nep-0509.md b/neps/nep-0509.md new file mode 100644 index 000000000..f7851bcb9 --- /dev/null +++ b/neps/nep-0509.md @@ -0,0 +1,121 @@ +--- +NEP: 509 +Title: Stateless validation stage 0 +Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong +Status: Draft +DiscussionsTo: [Stateless validation - stage 0](https://docs.google.com/document/d/1C-w4FNeXl8ZMd_Z_YxOf30XA1JM6eMDp5Nf3N-zzNWU/edit?usp=sharing), [Stateless validation roadmap](https://docs.google.com/document/d/1TzMENFGYjwc2g5A3Yf4zilvBwuYJufsUQJwRjXGb9Xc/edit?usp=sharing) +Type: Protocol +Version: 1.0.0 +Created: 2023-09-19 +LastUpdated: 2023-09-19 +--- + +## Summary + +The NEP proposes an solution to achieve phase 2 of sharding (where none of the validators needs to track all shards), with stateless validation while not relying of the traditional approach of fraud proof and state rollback. + +The fundamental idea is that validators do not need to have state locally to validate chunks. +* Under stateless validation, responsibility of a chunk producer extends to package transactions and receipts and annotate them with state witnesses (we will call them with the new role name, chunk proposers). +* State witness of a chunk is defined to be the state alongside proof of inclusion that is needed to execute a chunk. State witness allows anyone to execute a chunk without having the state of a shard locally. +* Then, validators will not store state locally and be randomly assigned to a shard at a given block height. Once a validator receives a chunk, along with its state witnesses, it verifies the state transition of the chunk, signs an approval, and sends it to the next block producer, similar to how it works today. + +## Motivation + +As phase 1 of sharding requires block producers to track all shards due to underlying security concerns, the team explored potential ways to achieve phase 2 of sharding, where none of the validators has to track all shards. + +Initial design of phase 2 relied on the security assumption that as long as there is one honest validator or fisherman tracking a shard, the shard is secure; by doing so, it naturally relied on protocol's ability to challenge (when an honest validator or fisherman detects a malicious behavior), rollback (when validators agree that the submitted challenge is valid), slashing (to punish the malicious validator), and rewarding (for chllenger). While it sounds straigtforward and simple on paper, the detailed implication and implementation of these features turned out to be extremely complicated. + +As a result, the team sought alternative approach and concluded that stateless validation is the most realistic and promising approach among candidates; the stateless validation approach does not assume the existence of fishermen and assumes that a shard is secure if every single chunk in that shard is validated by a randomly sampled subset of all validators. + +## Specification + +### Assumptions + +### High level requirements + +### Out of scope + +### High level flow + +TBD. [Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] + +## Reference Implementation + +[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. +- For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] + +## Security Implications + +[Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] + +## Alternatives + +[Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] + +## Future possibilities + +[Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] + +## Consequences + +[This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] + +### Positive + +- p1 + +### Neutral + +- n1 + +### Negative + +- n1 + +### Backwards Compatibility + +[All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.] + +## Unresolved Issues (Optional) + +[Explain any issues that warrant further discussion. Considerations + +- What parts of the design do you expect to resolve through the NEP process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] + +## Changelog + +[The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.] + +### 1.0.0 - Initial Version + +> Placeholder for the context about when and who approved this NEP version. + +#### Benefits + +> List of benefits filled by the Subject Matter Experts while reviewing this version: + +- Benefit 1 +- Benefit 2 + +#### Concerns + +> Template for Subject Matter Experts review for this version: +> Status: New | Ongoing | Resolved + +| # | Concern | Resolution | Status | +| --: | :------ | :--------- | -----: | +| 1 | | | | +| 2 | | | | + +## Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). From 867d414b44ba927ec95280bcd15b37df71b26df1 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Wed, 20 Sep 2023 07:37:16 -0700 Subject: [PATCH 02/28] Update nep-0509.md From a1c36b9dd2ab61bf7b4a06bb3be78bd41dcc922f Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Wed, 20 Sep 2023 07:37:38 -0700 Subject: [PATCH 03/28] Update nep-0509.md --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index f7851bcb9..a11802e65 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -3,7 +3,7 @@ NEP: 509 Title: Stateless validation stage 0 Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong Status: Draft -DiscussionsTo: [Stateless validation - stage 0](https://docs.google.com/document/d/1C-w4FNeXl8ZMd_Z_YxOf30XA1JM6eMDp5Nf3N-zzNWU/edit?usp=sharing), [Stateless validation roadmap](https://docs.google.com/document/d/1TzMENFGYjwc2g5A3Yf4zilvBwuYJufsUQJwRjXGb9Xc/edit?usp=sharing) +DiscussionsTo: https://docs.google.com/document/d/1C-w4FNeXl8ZMd_Z_YxOf30XA1JM6eMDp5Nf3N-zzNWU/edit?usp=sharing, https://docs.google.com/document/d/1TzMENFGYjwc2g5A3Yf4zilvBwuYJufsUQJwRjXGb9Xc/edit?usp=sharing Type: Protocol Version: 1.0.0 Created: 2023-09-19 From abbf6f097f1fe0cbb03129c9dbdd644249d6dba5 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Wed, 20 Sep 2023 08:41:24 -0700 Subject: [PATCH 04/28] Update nep-0509.md Add specifications --- neps/nep-0509.md | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index a11802e65..ea4225ac4 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -25,15 +25,38 @@ As phase 1 of sharding requires block producers to track all shards due to under Initial design of phase 2 relied on the security assumption that as long as there is one honest validator or fisherman tracking a shard, the shard is secure; by doing so, it naturally relied on protocol's ability to challenge (when an honest validator or fisherman detects a malicious behavior), rollback (when validators agree that the submitted challenge is valid), slashing (to punish the malicious validator), and rewarding (for chllenger). While it sounds straigtforward and simple on paper, the detailed implication and implementation of these features turned out to be extremely complicated. -As a result, the team sought alternative approach and concluded that stateless validation is the most realistic and promising approach among candidates; the stateless validation approach does not assume the existence of fishermen and assumes that a shard is secure if every single chunk in that shard is validated by a randomly sampled subset of all validators. +As a result, the team sought alternative approach and concluded that stateless validation is the most realistic and promising one; stateless validation approach does not assume the existence of fishermen and assumes that a shard is secure if every single chunk in that shard is validated by a randomly sampled subset of all validators. ## Specification ### Assumptions +* Post state root is enabled - [NEP-507](https://github.com/near/NEPs/pull/507) +* In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) +* State sync is enabled +* Merkle Patricia Trie is underlying data structure +* TBD ### High level requirements +* Block processing time should not take more than what it takes today. +* Additional load on network and node should not affect other functionalities. +* Security of protocol must not degrade. + * Validator assignment for both chunk validation and block validation should not create any security vulnerability. +* Validators should be rewarded with the same amount as they are now. +* No validator needs to track all shards. +* Only chunk producers need to maintain state locally. +* State witness size should be small enough to delivered over network. +* Majority of undercharging issues should be mitigated. +* [TBD] Resharding should work as expected after stateless validation in place. +* TBD ### Out of scope +* Data compression of both chunk data and state witness +* Separation of consensus and execution: we can run consensus independently from execution and let chunk producers to provide state witnesses to validators once every X blocks. This helps reduce both the amortized size of state witness and pressure on the p2p network. Validators send attestations to the state witness they receive and attestations are aggregated onchain as a finality gadget for state. +* More shards +* ZK integration +* Underlying data structure change (e.g. verkle tree) +* Validator structure change (e.g. block only producer, Infertile validator (validator which does not produce any chunk or block)) +* TBD ### High level flow From 229d1e298dfafd03025ce832b22cfcdd9a4d6231 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 23 Oct 2023 08:15:53 -0400 Subject: [PATCH 05/28] Update nep-0509.md Add validator role change section --- neps/nep-0509.md | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index ea4225ac4..37f952a68 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -30,7 +30,6 @@ As a result, the team sought alternative approach and concluded that stateless v ## Specification ### Assumptions -* Post state root is enabled - [NEP-507](https://github.com/near/NEPs/pull/507) * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) * State sync is enabled * Merkle Patricia Trie is underlying data structure @@ -46,6 +45,7 @@ As a result, the team sought alternative approach and concluded that stateless v * Only chunk producers need to maintain state locally. * State witness size should be small enough to delivered over network. * Majority of undercharging issues should be mitigated. +* Validator roles are updated accordingly to reflect the necessary changes with stateless validation. * [TBD] Resharding should work as expected after stateless validation in place. * TBD @@ -55,10 +55,10 @@ As a result, the team sought alternative approach and concluded that stateless v * More shards * ZK integration * Underlying data structure change (e.g. verkle tree) -* Validator structure change (e.g. block only producer, Infertile validator (validator which does not produce any chunk or block)) +* Validator reward change * TBD -### High level flow +## High level flow TBD. [Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] @@ -74,6 +74,39 @@ TBD. [Explain the proposal as if you were teaching it to another developer. This The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] +## Validatior role change +Currently, there are two different types of validators and their responsibilities are as follows: +| | Top ~50% validators | Remaining validatiors (Chunk only producers) | +|-----|:-----:|:----:| +| block production | Y | N | +| chunk production | Y | Y | +| block validation | Y | N | + +With stateless validation, this structure does not make sense anymore for several reasons: +* Chunk production is the most resource consuming activity. +* (Only) chunk production needs state in memory while other responsibilities can be completed via acquiring state witness +* Chunk production does not have to be performed by all validators. + +Hence, the most simple proposal is to change Chunk-only producers to Chunk-only validators as follows: +| | Top ~50% validators | Remaining validatiors (Chunk-only validators) | +|-----|:-----:|:----:| +| block production | Y | N | +| chunk production | Y | N | +| block validation | Y | N | +| chunk validation | Y | Y | + +Block production and validation remain as responsibility of validators with more stake to maintain the same level of security. + +This approach is the most straight forward as it maintains the same grouping as we have today. + +Potential improvement to which can lower hardware requirement for more validators is limiting the responsibility of chunk production to top N validators, who are often equipped with powerful machines already. +| | Top N validatiors (Chunk proposers) | Top ~50% - N validators | Remaining validators (Chunk-only validators) | +|-----|:-----:|:----:|:----:| +| block production | Y | Y | N | +| chunk production | Y | N | N | +| block validation | Y | Y | N | +| chunk validation | Y | Y | N | + ## Security Implications [Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] From 5627cede868a9d55a20fe0716a9c0d5aabe0e6ed Mon Sep 17 00:00:00 2001 From: robin-near <111538878+robin-near@users.noreply.github.com> Date: Fri, 26 Jan 2024 07:55:45 -0800 Subject: [PATCH 06/28] Revise the summary and motivation, add high level flow. (#527) --- neps/nep-0509.md | 103 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 76 insertions(+), 27 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 37f952a68..f72f608be 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -1,6 +1,6 @@ --- NEP: 509 -Title: Stateless validation stage 0 +Title: Stateless validation Stage 0 Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong Status: Draft DiscussionsTo: https://docs.google.com/document/d/1C-w4FNeXl8ZMd_Z_YxOf30XA1JM6eMDp5Nf3N-zzNWU/edit?usp=sharing, https://docs.google.com/document/d/1TzMENFGYjwc2g5A3Yf4zilvBwuYJufsUQJwRjXGb9Xc/edit?usp=sharing @@ -12,58 +12,107 @@ LastUpdated: 2023-09-19 ## Summary -The NEP proposes an solution to achieve phase 2 of sharding (where none of the validators needs to track all shards), with stateless validation while not relying of the traditional approach of fraud proof and state rollback. +The NEP proposes an solution to achieve phase 2 of sharding (where none of the validators needs to track all shards), with stateless validation, instead of the traditionally proposed approach of fraud proof and state rollback. The fundamental idea is that validators do not need to have state locally to validate chunks. -* Under stateless validation, responsibility of a chunk producer extends to package transactions and receipts and annotate them with state witnesses (we will call them with the new role name, chunk proposers). -* State witness of a chunk is defined to be the state alongside proof of inclusion that is needed to execute a chunk. State witness allows anyone to execute a chunk without having the state of a shard locally. -* Then, validators will not store state locally and be randomly assigned to a shard at a given block height. Once a validator receives a chunk, along with its state witnesses, it verifies the state transition of the chunk, signs an approval, and sends it to the next block producer, similar to how it works today. +* Under stateless validation, the responsibility of a chunk producer extends to packaging transactions and receipts and annotating them with state witnesses. This extended role will be called "chunk proposers". +* The state witness of a chunk is defined to be a subset of the trie state, alongside its proof of inclusion in the trie, that is needed to execute a chunk. A state witness allows anyone to execute the chunk without having the state of its shard locally. +* * Then, at each block height, validators will be randomly assigned to a shard, to validate the state witness for that shard. Once a validator receives both a chunk and its state witness, it verifies the state transition of the chunk, signs a chunk endorsement and sends it to the block producer. This is similar to, but separate from, block approvals and consensus. +* The block producer waits for sufficient chunk endorsements before including a chunk into the block it produces, or omits the chunk if not enough endorsements arrive in time. ## Motivation As phase 1 of sharding requires block producers to track all shards due to underlying security concerns, the team explored potential ways to achieve phase 2 of sharding, where none of the validators has to track all shards. -Initial design of phase 2 relied on the security assumption that as long as there is one honest validator or fisherman tracking a shard, the shard is secure; by doing so, it naturally relied on protocol's ability to challenge (when an honest validator or fisherman detects a malicious behavior), rollback (when validators agree that the submitted challenge is valid), slashing (to punish the malicious validator), and rewarding (for chllenger). While it sounds straigtforward and simple on paper, the detailed implication and implementation of these features turned out to be extremely complicated. +The early design of phase 2 relied on the security assumption that as long as there is one honest validator or fisherman tracking a shard, the shard is secure; by doing so, it naturally relied on protocol's ability to handle challenges (when an honest validator or fisherman detects a malicious behavior and submits a proof of such), state rollbacks (when validators agree that the submitted challenge is valid), and slashing (to punish the malicious validator). While it sounds straightforward and simple on paper, the complex interactions between these abilities and the rest of the protocol led to concrete designs that were extremely complicated, involving several specific problems we still don't know how to solve. -As a result, the team sought alternative approach and concluded that stateless validation is the most realistic and promising one; stateless validation approach does not assume the existence of fishermen and assumes that a shard is secure if every single chunk in that shard is validated by a randomly sampled subset of all validators. +As a result, the team sought alternative approaches and concluded that stateless validation is the most realistic and promising one; the stateless validation approach does not assume the existence of a fishermen, does not rely on challenges, and never rolls back state. Instead, it relies on the assumption that a shard is secure if every single chunk in that shard is validated by a randomly sampled subset of all validators, to always produce valid chunks in the first place. ## Specification ### Assumptions * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) -* State sync is enabled -* Merkle Patricia Trie is underlying data structure +* State sync is enabled (so that nodes can track different shards across epochs) +* Merkle Patricia Trie continues to be the state trie implementation * TBD ### High level requirements -* Block processing time should not take more than what it takes today. -* Additional load on network and node should not affect other functionalities. -* Security of protocol must not degrade. - * Validator assignment for both chunk validation and block validation should not create any security vulnerability. -* Validators should be rewarded with the same amount as they are now. * No validator needs to track all shards. -* Only chunk producers need to maintain state locally. -* State witness size should be small enough to delivered over network. -* Majority of undercharging issues should be mitigated. -* Validator roles are updated accordingly to reflect the necessary changes with stateless validation. -* [TBD] Resharding should work as expected after stateless validation in place. +* Security of protocol must not degrade. + * Validator assignment for both chunk validation and block validation should not create any security vulnerabilities. +* Block processing time should not take significantly more than what it takes today. +* Any additional load on network and compute should not negatively affect existing functionalities of any node in the blockchain. + * The cost of additional network and compute should be acceptable. +* Validator rewards should not be reduced. +* Resharding should still be possible after stateless validation is in place. * TBD ### Out of scope -* Data compression of both chunk data and state witness -* Separation of consensus and execution: we can run consensus independently from execution and let chunk producers to provide state witnesses to validators once every X blocks. This helps reduce both the amortized size of state witness and pressure on the p2p network. Validators send attestations to the state witness they receive and attestations are aggregated onchain as a finality gadget for state. -* More shards -* ZK integration -* Underlying data structure change (e.g. verkle tree) -* Validator reward change +* Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. +* Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. +* More shards - this is covered in the resharding project. +* ZK integration. +* Underlying data structure change (e.g. verkle tree). +* Change to validator rewards. * TBD ## High level flow -TBD. [Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] +We propose a change to the following parts of the chunk and block production flow: + +* When a chunk producer produces a chunk, in addition to collecting transactions and receipts for the chunk, it will also produce a `ChunkStateWitness`. + * The `ChunkStateWitness` contains whatever data necessary to prove that this chunk's header should indeed be what is being produced: + * As it is today, all fields of the `ShardChunkHeaderInnerV2`, except `tx_root`, are uniquely determined by the blockchain's history based on where the chunk is located (i.e. its parent block and shard ID). + * The `tx_root` is based on the list of transactions proposed, which is at the discretion of the chunk producer. However, these transactions must be valid (i.e. the sender accounts have enough balance and the correct nonce, etc.). + * This `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct, meaning that the uniquely determined fields are exactly what should be expected, and the discretionary `tx_root` field corresponds to a valid set of transactions. + * The `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. +* The chunk producer then distributes the `ChunkStateWitness` to a subset of *Chunk Validators* assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. + * Chunk Validator is a new role described in the "Validator role change" section. + * The subset of chunk validators assigned to a shard is determined by a random shuffle, once per block. See the "Validator Shuffling" section. +* A chunk validator, upon receiving a `ChunkStateWitness`, validates the state witness and determines if the chunk header is indeed correctly produced. If so, it sends a `ChunkEndorsement` to the current block producer. + * A `ChunkEndorsement` contains the chunk hash along with a signature proving the endorsement by the chunk validator. It implicitly carries a weight equal to the amount of the chunk validator's stake that is assigned to this shard for this block. (See Chunk Validator Shuffling). +* As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by weight. + * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need challenges**. + * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. See Chunk Validator Shuffling. +* The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements for each chunk included in the block. + * This necessitates a new block format. + * If a block fails validation because of not having the required chunk endorsements, it is considered a block validation failure for the purpose of Doomslug consensus, just like any other block validation failure. In other words, nodes will not apply the block on top of their blockchain, and (block) validators will not endorse the block. + +We also propose a change to the validator roles and responsibilities. This is the list of roles after the proposal, with same and new behavior clearly labelled: + +* Block producers: + * (Same as today) Produce blocks, (new) including waiting for chunk endorsements + * (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding) + * (Same as today) Do not require tracking any shard + * (Same as today) Should have a higher barrier of entry for security reasons (e.g. to make block double signing harder) +* Chunk producers: + * (Same as today) Produce chunks, (new) including producing chunk state witnesses + * (New) Distributes state witnesses to chunk validators + * (Same as today) Must track the shard it produces the chunk for + * (Same as today) Rotate shards across epoch boundaries, (new) but at a lower rate (e.g. 1 week) +* Block validators: + * (Same as today) Validate blocks, (new) including verifying chunk endorsements + * (Same as today) Vote for blocks with endorsement or skip messages + * (New) No longer require tracking any shard + * (Same as today) Must collectively have a majority of all the validator stake, for security reasons. +* (New) Chunk validators: + * Validate state witnesses, and sends chunk endorsements to block producers + * Do not require tracking any shard + * Must collectively have a majority of all the validator stake, to ensure the security of chunk validation. + +See the Validator Role Change section for more details. + +## Chunk Validator Shuffling +Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. + +Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The amount of stake for a full mandate is a fixed parameter determined by the stake distribution of all validators, and any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. + +We have done research to show that the security of this algorithm is sufficient with a reasonable number of chunk validators and a reasonable number of shards, assuming a reasonable bound for the total stake of malicious nodes. TODO: Include or link to that research here. ## Reference Implementation +TODO: This is essentially going to be describing the exact structure of `ChunkStateWitness`, `ChunkEndorsement`, and describing the exact algorithm to be used for the chunk validator shuffling. + [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: - Its interaction with other features is clear. @@ -74,7 +123,7 @@ TBD. [Explain the proposal as if you were teaching it to another developer. This The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] -## Validatior role change +## Validator Role Change Currently, there are two different types of validators and their responsibilities are as follows: | | Top ~50% validators | Remaining validatiors (Chunk only producers) | |-----|:-----:|:----:| From c05952660a164a349d3c86c127fc844918f7fa67 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Fri, 26 Jan 2024 11:38:24 -0800 Subject: [PATCH 07/28] Update nep-0509.md fix lint errors --- neps/nep-0509.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index f72f608be..ec41cb7b9 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -15,9 +15,10 @@ LastUpdated: 2023-09-19 The NEP proposes an solution to achieve phase 2 of sharding (where none of the validators needs to track all shards), with stateless validation, instead of the traditionally proposed approach of fraud proof and state rollback. The fundamental idea is that validators do not need to have state locally to validate chunks. + * Under stateless validation, the responsibility of a chunk producer extends to packaging transactions and receipts and annotating them with state witnesses. This extended role will be called "chunk proposers". * The state witness of a chunk is defined to be a subset of the trie state, alongside its proof of inclusion in the trie, that is needed to execute a chunk. A state witness allows anyone to execute the chunk without having the state of its shard locally. -* * Then, at each block height, validators will be randomly assigned to a shard, to validate the state witness for that shard. Once a validator receives both a chunk and its state witness, it verifies the state transition of the chunk, signs a chunk endorsement and sends it to the block producer. This is similar to, but separate from, block approvals and consensus. +* Then, at each block height, validators will be randomly assigned to a shard, to validate the state witness for that shard. Once a validator receives both a chunk and its state witness, it verifies the state transition of the chunk, signs a chunk endorsement and sends it to the block producer. This is similar to, but separate from, block approvals and consensus. * The block producer waits for sufficient chunk endorsements before including a chunk into the block it produces, or omits the chunk if not enough endorsements arrive in time. ## Motivation @@ -31,12 +32,14 @@ As a result, the team sought alternative approaches and concluded that stateless ## Specification ### Assumptions + * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) * State sync is enabled (so that nodes can track different shards across epochs) * Merkle Patricia Trie continues to be the state trie implementation * TBD ### High level requirements + * No validator needs to track all shards. * Security of protocol must not degrade. * Validator assignment for both chunk validation and block validation should not create any security vulnerabilities. @@ -48,6 +51,7 @@ As a result, the team sought alternative approaches and concluded that stateless * TBD ### Out of scope + * Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. * Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. * More shards - this is covered in the resharding project. @@ -103,6 +107,7 @@ We also propose a change to the validator roles and responsibilities. This is th See the Validator Role Change section for more details. ## Chunk Validator Shuffling + Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The amount of stake for a full mandate is a fixed parameter determined by the stake distribution of all validators, and any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. @@ -115,11 +120,11 @@ TODO: This is essentially going to be describing the exact structure of `ChunkSt [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: -- Its interaction with other features is clear. -- Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. -- For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. +* Its interaction with other features is clear. +* Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) +* It is reasonably clear how the feature would be implemented. +* Corner cases are dissected by example. +* For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] From 6c7c1d9f8b6407396b493b768ebedb6e3954c702 Mon Sep 17 00:00:00 2001 From: wacban Date: Wed, 12 Jun 2024 16:54:53 +0100 Subject: [PATCH 08/28] test test --- neps/nep-0509.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index ec41cb7b9..ce83f1d3f 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -179,7 +179,8 @@ Potential improvement to which can lower hardware requirement for more validator ### Positive -- p1 +- The validator nodes will only need to track one shard. +- ### Neutral From a999b1df8c50eb04d5dec010a19860953f860897 Mon Sep 17 00:00:00 2001 From: Aleksandr Logunov Date: Fri, 14 Jun 2024 00:24:19 +0400 Subject: [PATCH 09/28] doc: validator structure change, state witness reference impl (#546) Documenting changes to validators and describing basics of reference implementation. --- neps/nep-0509.md | 392 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 313 insertions(+), 79 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index ce83f1d3f..ff87bc8c6 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -33,12 +33,13 @@ As a result, the team sought alternative approaches and concluded that stateless ### Assumptions +* Not more than 1/3 of validators is corrupted. * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) * State sync is enabled (so that nodes can track different shards across epochs) * Merkle Patricia Trie continues to be the state trie implementation -* TBD +* Congestion Control is enabled - [NEP-539](https://github.com/near/NEPs/pull/539) -### High level requirements +### Design requirements * No validator needs to track all shards. * Security of protocol must not degrade. @@ -47,135 +48,368 @@ As a result, the team sought alternative approaches and concluded that stateless * Any additional load on network and compute should not negatively affect existing functionalities of any node in the blockchain. * The cost of additional network and compute should be acceptable. * Validator rewards should not be reduced. -* Resharding should still be possible after stateless validation is in place. -* TBD -### Out of scope +### Current design -* Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. -* Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. -* More shards - this is covered in the resharding project. -* ZK integration. -* Underlying data structure change (e.g. verkle tree). -* Change to validator rewards. -* TBD - -## High level flow - -We propose a change to the following parts of the chunk and block production flow: - -* When a chunk producer produces a chunk, in addition to collecting transactions and receipts for the chunk, it will also produce a `ChunkStateWitness`. - * The `ChunkStateWitness` contains whatever data necessary to prove that this chunk's header should indeed be what is being produced: - * As it is today, all fields of the `ShardChunkHeaderInnerV2`, except `tx_root`, are uniquely determined by the blockchain's history based on where the chunk is located (i.e. its parent block and shard ID). - * The `tx_root` is based on the list of transactions proposed, which is at the discretion of the chunk producer. However, these transactions must be valid (i.e. the sender accounts have enough balance and the correct nonce, etc.). - * This `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct, meaning that the uniquely determined fields are exactly what should be expected, and the discretionary `tx_root` field corresponds to a valid set of transactions. - * The `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. -* The chunk producer then distributes the `ChunkStateWitness` to a subset of *Chunk Validators* assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. - * Chunk Validator is a new role described in the "Validator role change" section. - * The subset of chunk validators assigned to a shard is determined by a random shuffle, once per block. See the "Validator Shuffling" section. +The current high-level chunk production flow, excluding details and edge cases, is as follows: +* Block producer at height `H` `BP(H)` produces block `B(H)` with chunks accessible to it and distributes it. +* Chunk producer for shard `S` at height `H+1` `CP(S, H+1)` produces chunk `C(S, H+1)` based on `B(H)` and distributes it. +* `BP(H+1)` collects all chunks at height `H+1` until certain timeout is reached. +* `BP(H+1)` produces block `B(H+1)` with chunks `C(*, H+1)` accessible to it and distributes it, etc. + +The "induction base" is at genesis height, where genesis block with default chunks is accessible to everyone, so chunk producers can start right away from genesis height + 1. + +One can observe that there is no "chunk validation" step here. In fact, validity of chunks is implicitly guaranteed by **requirement for all block producers to track all shards**. +To achieve phase 2 of sharding, we want to drop this requirement. For that, we propose the following changes to the flow: + +### New design + +* Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header that is being produced. + * `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct. + * `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. +* The chunk producer distributes the `ChunkStateWitness` to a subset of **chunk validators** which are assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. + * Chunk Validator selection and assignment are described below. * A chunk validator, upon receiving a `ChunkStateWitness`, validates the state witness and determines if the chunk header is indeed correctly produced. If so, it sends a `ChunkEndorsement` to the current block producer. - * A `ChunkEndorsement` contains the chunk hash along with a signature proving the endorsement by the chunk validator. It implicitly carries a weight equal to the amount of the chunk validator's stake that is assigned to this shard for this block. (See Chunk Validator Shuffling). -* As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by weight. - * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need challenges**. - * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. See Chunk Validator Shuffling. -* The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements for each chunk included in the block. - * This necessitates a new block format. +* As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by stake. + * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need fraud proofs / tracking all shards**. + * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. +* The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements by stake for each chunk included in the block. * If a block fails validation because of not having the required chunk endorsements, it is considered a block validation failure for the purpose of Doomslug consensus, just like any other block validation failure. In other words, nodes will not apply the block on top of their blockchain, and (block) validators will not endorse the block. -We also propose a change to the validator roles and responsibilities. This is the list of roles after the proposal, with same and new behavior clearly labelled: +So the high-level specification can be described as the list of changes in the validator roles and responsibilities: * Block producers: * (Same as today) Produce blocks, (new) including waiting for chunk endorsements * (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding) - * (Same as today) Do not require tracking any shard - * (Same as today) Should have a higher barrier of entry for security reasons (e.g. to make block double signing harder) + * (New) No longer require tracking any shard + * (Same as today) Should have high barrier of entry for security reasons, to make block double signing harder. * Chunk producers: - * (Same as today) Produce chunks, (new) including producing chunk state witnesses - * (New) Distributes state witnesses to chunk validators + * (Same as today) Produce chunks + * (New) Produces and distributes state witnesses to chunk validators * (Same as today) Must track the shard it produces the chunk for - * (Same as today) Rotate shards across epoch boundaries, (new) but at a lower rate (e.g. 1 week) * Block validators: * (Same as today) Validate blocks, (new) including verifying chunk endorsements * (Same as today) Vote for blocks with endorsement or skip messages * (New) No longer require tracking any shard * (Same as today) Must collectively have a majority of all the validator stake, for security reasons. + * (Same as today) Should have high barrier of entry to keep `BlockHeader` size low, because it is proportional to the total byte size of block validator signatures; * (New) Chunk validators: - * Validate state witnesses, and sends chunk endorsements to block producers + * Validate state witnesses and sends chunk endorsements to block producers * Do not require tracking any shard * Must collectively have a majority of all the validator stake, to ensure the security of chunk validation. -See the Validator Role Change section for more details. - -## Chunk Validator Shuffling +See the Validator Structure Change section below for more details. -Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. - -Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The amount of stake for a full mandate is a fixed parameter determined by the stake distribution of all validators, and any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. +### Out of scope -We have done research to show that the security of this algorithm is sufficient with a reasonable number of chunk validators and a reasonable number of shards, assuming a reasonable bound for the total stake of malicious nodes. TODO: Include or link to that research here. +* Resharding support. +* Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. +* Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. +* ZK integration. +* Underlying data structure change (e.g. verkle tree). ## Reference Implementation -TODO: This is essentially going to be describing the exact structure of `ChunkStateWitness`, `ChunkEndorsement`, and describing the exact algorithm to be used for the chunk validator shuffling. +Here we carefully describe new structures and logic introduced, without going into too much technical details. -[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: +### Validator Structure Change -* Its interaction with other features is clear. -* Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) -* It is reasonably clear how the feature would be implemented. -* Corner cases are dissected by example. -* For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. +#### Roles +Currently, there are two different types of validators. Their responsibilities are defined as in the following pseudocode: -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] +```python +if index(validator) < 100: + roles(validator).append("block producer") +roles(validator).append("chunk producer") +``` -## Validator Role Change -Currently, there are two different types of validators and their responsibilities are as follows: -| | Top ~50% validators | Remaining validatiors (Chunk only producers) | -|-----|:-----:|:----:| -| block production | Y | N | -| chunk production | Y | Y | -| block validation | Y | N | +The validators are ordered by non-increasing stake in the considered epoch. Here and below by "block production" we mean both production and validation. -With stateless validation, this structure does not make sense anymore for several reasons: +With stateless validation, this structure must change for several reasons: * Chunk production is the most resource consuming activity. * (Only) chunk production needs state in memory while other responsibilities can be completed via acquiring state witness * Chunk production does not have to be performed by all validators. -Hence, the most simple proposal is to change Chunk-only producers to Chunk-only validators as follows: -| | Top ~50% validators | Remaining validatiors (Chunk-only validators) | -|-----|:-----:|:----:| -| block production | Y | N | -| chunk production | Y | N | -| block validation | Y | N | -| chunk validation | Y | Y | +Hence, to make transition seamless, we change the role of nodes out of top 100 to only validate chunks: -Block production and validation remain as responsibility of validators with more stake to maintain the same level of security. +```python +if index(validator) < 100: + roles(validator).append("chunk producer") + roles(validator).append("block producer") +roles(validator).append("chunk validator") +``` -This approach is the most straight forward as it maintains the same grouping as we have today. +The more stake validator has, the more **heavy** work it will get assigned, because we assume that validators with higher stakes have more powerful hardware. +With stateless validation, relative heaviness of the work changes. Comparing to the current order "block production" > "chunk production", the new order is "chunk production" > "block production" > "chunk validation". + +Shards are equally split among chunk producers: as in Mainnet on 12 Jun 2024 we have 6 shards, each shard would have ~16 chunk producers assigned. + +In the future, with increase in number of shards, we can generalise the assignment by saying that each shard should have `X` chunk producers assigned, if we have at least `X * S` validators. In such case, pseudocode for the role assignment would look as follows: + +```python +if index(validator) < X * S: + roles(validator).append("chunk producer") +if index(validator) < 100: + roles(validator).append("block producer") +roles(validator).append("chunk validator") +``` + +#### Rewards + +Reward for each validator is defined as `total_epoch_reward * validator_relative_stake * work_quality_ratio`, where: +* `total_epoch_reward` is selected so that total inflation of the token is 5% per annum; +* `validator_relative_stake = validator_stake / total_epoch_stake`; +* `work_quality_ratio` is the measure of the work quality from 0 to 1. + +So, the actual reward never exceeds total reward, and when everyone does perfect work, they are equal. +For the context of the NEP, it is enough to assume that `work_quality_ratio = avg_{role}({role}_quality_ratio)`. +So, if node is both a block and chunk producer, we compute quality for each role separately and then take average of them. + +When epoch is finalized, all block headers in it uniquely determine who was expected to produce each block and chunk. +Thus, if we define quality ratio for block producer as `produced_blocks/expected_blocks`, everyone is able to compute it. +Similarly, `produced_chunks/expected_chunks` is a quality for chunk producer. +It is more accurate to say `included_chunks/expected_chunks`, because inclusion of chunk in block is a final decision of a block producer which defines success here. + +Ideally, we could compute quality for chunk validator as `produced_endorsements/expected_endorsements`. Unfortunately, we won't do it in Stage 0 because: +* Mask of endorsements is not part of the block header, and it would be a significant change; +* Block producer doesn't have to wait for all endorsements to be collected, so it could be unfair to say that endorsement was not produced if block producer just went ahead. + +So for now we decided to compute quality for chunk validator as ratio of `included_chunks/expected_chunks`, where we iterate over chunks which node was expected to validate. +The obvious drawback here is that if chunks are not produced at all, chunk validators will also be impacted. We plan to address it in the future releases. + +#### Kickouts + +In addition to that, if node performance is too poor, we want a mechanism to kick it out of the validator list, to ensure healthy protocol performance and validator rotation. +Currently, we have a threshold for each role, and if for some role the same `{role}_quality_ratio` is lower than threshold, the node is kicked out. + +If we write this in pseudocode, + +```python +if validator is block producer and block_producer_quality_ratio < 0.8: + kick out validator +if validator is chunk producer and chunk_producer_quality_ratio < 0.8: + kick out validator +``` + +For chunk validator, we apply absolutely the same formula. However, because: +* the formula doesn't count endorsements explicitly +* for chunk producers it kind of just makes chunk production condition stronger without adding value + +we apply it to nodes which **only validate chunks**. So, we add this line: + +```python +if validator is only chunk validator and chunk_validator_quality_ratio < 0.8: + kick out validator +``` + +As we pointed out above, current formula `chunk_validator_quality_ratio` is problematic. +Here it brings even a bigger issue: if chunk producers don't produce chunks, chunk validators will be kicked out as well, which impacts network stability. +This is another reason to come up with the better formula. + +#### Shard assignment + +As chunk producer becomes the most important role, we need to ensure that every epoch has significant amount of healthy chunk producers. +This is a **significant difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. + +The most challenging part of becoming a chunk producer for a shard is to download most recent shard state within previous epoch. This is called "state sync". +Unfortunately, as of now, state sync is centralised on published snapshots, which is a major point of failure, until we don't have decentralised state sync. + +Because of that, we make additional change: if node was a chunk producer for some shard in the previous epoch, and it is a chunk producer for current epoch, it will be assigned to the same shard. +This way, we minimise number of required state syncs at each epoch. + +The exact algorithm needs a thorough description to satisfy different edge cases, so we will just leave a link to full explanation: https://github.com/near/nearcore/issues/11213#issuecomment-2111234940. + +### ChunkStateWitness + +The full structure is described [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/core/primitives/src/stateless_validation.rs#L247). +Let's construct it sequentially together with explaining why every field is needed. Start from simple data: +```rust +pub struct ChunkStateWitness { + pub chunk_producer: AccountId, + pub epoch_id: EpochId, + /// The chunk header which this witness is proving. + pub chunk_header: ShardChunkHeader, +} +``` + +What is needed to prove `ShardChunkHeader`? + +The key function we have in codebase is [validate_chunk_with_chunk_extra_and_receipts_root](https://github.com/near/nearcore/blob/c2d80742187d9b8fc1bb672f16e3d5c144722742/chain/chain/src/validate.rs#L141). +The main arguments there are `prev_chunk_extra: &ChunkExtra` which stands for execution result of previous chunk, and `chunk_header`. +The most important field for `ShardChunkHeader` is `prev_state_root` - consider latest implementation `ShardChunkHeaderInnerV3`. It stands for state root resulted from updating shard for the previous block, which means applying previous chunk if there is no missing chunks. +So, chunk validator needs some way to run transactions and receipts from the previous chunk. Let's call it a "main state transition" and add two more fields to state witness: + +```rust + /// The base state and post-state-root of the main transition where we + /// apply transactions and receipts. Corresponds to the state transition + /// that takes us from the pre-state-root of the last new chunk of this + /// shard to the post-state-root of that same chunk. + pub main_state_transition: ChunkStateTransition, + /// The transactions to apply. These must be in the correct order in which + /// they are to be applied. + pub transactions: Vec, +``` + +where +```rust +/// Represents the base state and the expected post-state-root of a chunk's state +/// transition. The actual state transition itself is not included here. +pub struct ChunkStateTransition { + /// The block that contains the chunk; this identifies which part of the + /// state transition we're talking about. + pub block_hash: CryptoHash, + /// The partial state before the state transition. This includes whatever + /// initial state that is necessary to compute the state transition for this + /// chunk. It is a list of Merkle tree nodes. + pub base_state: PartialState, + /// The expected final state root after applying the state transition. + pub post_state_root: CryptoHash, +} +``` + +Fine, but where do we take the receipts? + +Receipts are internal messages, resulting from transaction execution, sent between shards, and **by default** they are not signed by anyone. + +However, each receipt is an execution outcome of some transaction or other parent receipt, executed in some previous chunk. +For every chunk, we conveniently store `prev_outgoing_receipts_root` which is a Merkle hash of all receipts sent to other shards resulting by execution of this chunk. So, for every receipt, there is a proof of its generation in some parent chunk. If there are no missing chunk, then it's enough to consider chunks from previous block. + +So we add another field: + +```rust + /// Non-strict superset of the receipts that must be applied, along with + /// information that allows these receipts to be verifiable against the + /// blockchain history. + pub source_receipt_proofs: HashMap, +``` + +What about missing chunks though? + +Unfortunately, production and inclusion of any chunk **cannot be guaranteed**: +* chunk producer may go offline; +* chunk validators may not generate 2/3 endorsements; +* block producer may not receive enough information to include chunk. + +Let's handle this case as well. +First, each chunk producer needs not just to prove main state transition, but also all state transitions for latest missing chunks: +```rust + /// For each missing chunk after the last new chunk of the shard, we need + /// to carry out an implicit state transition. This is technically needed + /// to handle validator rewards distribution. This list contains one for each + /// such chunk, in forward chronological order. + /// + /// After these are applied as well, we should arrive at the pre-state-root + /// of the chunk that this witness is for. + pub implicit_transitions: Vec, +``` + +Then, while our shard was missing chunks, other shards could still produce chunks, which could generate receipts targeting our shards. So, we need to extend `source_receipt_proofs`. +Field structure doesn't change, but we need to carefully pick range of set of source chunks, so different subsets will cover all source receipts without intersection. + +Let's say B2 is the block that contains the last new chunk of shard S before chunk which state transition we execute, and B1 is the block that contains the last new chunk of shard S before B2. +Then, we will define set of blocks B as the contiguous subsequence of blocks B1 (EXCLUSIVE) to B2 (inclusive) in this chunk's chain (i.e. the linear chain that this chunk's parent block is on). Lastly, source chunks are all chunks included in blocks from B. + +The last caveat is **new** transactions introduced by chunk with `chunk_header`. As chunk header introduces `tx_root` for them, we need to check validity of this field as well. +If we don't do it, malicious chunk producer can include invalid transaction, and if it gets its chunk endorsed, nodes which track the shard must either accept invalid transaction or refuse to process chunk, but the latter means that shard will get stuck. + +To validate new `tx_root`, we also need Merkle partial state to validate sender' balances, access keys, nonces, etc., which leads to two last fields to be added: + +```rust + pub new_transactions: Vec, + pub new_transactions_validation_state: PartialState, +``` + +The logic to produce `ChunkStateWitness` is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/state_witness_producer.rs#L79). +Itself, it requires some minor changes to the logic of applying chunks, related to generating `ChunkStateTransition::base_state`. +It is controlled by [this line](https://github.com/near/nearcore/blob/dc03a34101f77a17210873c4b5be28ef23443864/chain/chain/src/runtime/mod.rs#L977), which causes all nodes read during applying chunk to be put inside `TrieRecorder`. +After applying chunk, its contents are saved to `StateTransitionData`. + +The validation logic is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/chunk_validator/mod.rs#L85). +First, it performs all validation steps for which access to `ChainStore` is required, `pre_validate_chunk_state_witness` is responsible for this. It is done separately because `ChainStore` is owned by a single thread. +Then, it spawns a thread which runs computation-heavy `validate_chunk_state_witness` which main purpose is to apply chunk based on received state transitions and verify that execution results in chunk header are correct. +If validation is successful, `ChunkEndorsement` is sent. + +### ChunkEndorsement + +It is basically a triple of `(ChunkHash, AccountId, Signature)`. +Receiving this message means that specific chunk validator account endorsed chunk with specific chunk hash. +Ideally chunk validator would send chunk endorsement to just the next block producer at the same height for which chunk was produced. +However, block at that height can be skipped and block producers at heights h+1, h+2, ... will have to pick up the chunk. +To address that, we send `ChunkEndorsement` to all block producers at heights from h to `h+d-1`. We pick `d=5` as more than 5 skipped blocks in a row are very unlikely to occur. + +On block producer side, chunk endorsements are collected and stored in `ChunkEndorsementTracker`. +Small **caveat** is that *sometimes* chunk endorsement may be received before chunk header which is required to understand that sender is indeed a validator of the chunk. +Such endorsements are stored as *pending*. +When chunk header is received, all pending endorsements are checked for validity and marked as *validated*. +All endorsements received after that are validated right away. + +Finally, when block producer attempts to produce a block, in addition to checking chunk existence, it also checks that it has 2/3 endorsement stake for that chunk hash. +To make chunk inclusion verifiable, we introduce [another version](https://github.com/near/nearcore/blob/cf2caa3513f58da8be758d1c93b0900ffd5d51d2/core/primitives/src/block_body.rs#L30) of block body `BlockBodyV2` which has new field `chunk_endorsements`. +It is basically a `Vec>>` where element with indices `(s, i)` contains signature of i-th chunk validator for shard s if it was included and None otherwise. +Lastly, we add condition to block validation, such that if chunk `s` was included in the block, then block body must contain 2/3 endorsements for that shard. + +This logic is triggered in `ChunkInclusionTracker` by methods [get_chunk_headers_ready_for_inclusion](https://github.com/near/nearcore/blob/6184e5dac45afb10a920cfa5532ce6b3c088deee/chain/client/src/chunk_inclusion_tracker.rs#L146) and couple similar ones. Number of ready chunks is returned by [num_chunk_headers_ready_for_inclusion](https://github.com/near/nearcore/blob/6184e5dac45afb10a920cfa5532ce6b3c088deee/chain/client/src/chunk_inclusion_tracker.rs#L178). + +### Chunk validators selection -Potential improvement to which can lower hardware requirement for more validators is limiting the responsibility of chunk production to top N validators, who are often equipped with powerful machines already. -| | Top N validatiors (Chunk proposers) | Top ~50% - N validators | Remaining validators (Chunk-only validators) | -|-----|:-----:|:----:|:----:| -| block production | Y | Y | N | -| chunk production | Y | N | N | -| block validation | Y | Y | N | -| chunk validation | Y | Y | N | +Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. + +Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The amount of stake for a full mandate is a fixed parameter determined by the stake distribution of all validators, and any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. + +For Stage 0, we select **target amount of mandates per shard** to 68, which was a [result of the latest research](https://near.zulipchat.com/#narrow/stream/407237-core.2Fstateless-validation/topic/validator.20seat.20assignment/near/435252304). +With this number of mandates per shard and 6 shards, we predict the protocol to be secure for 40 years at 90% confidence. +Based on target number of mandates and total chunk validators stake, [here](https://github.com/near/nearcore/blob/696190b150dd2347f9f042fa99b844b67c8001d8/core/primitives/src/validator_mandates/mod.rs#L76) we compute price of a single full mandate for each new epoch using binary search. +All the mandates are stored in new version of `EpochInfo` `EpochInfoV4` in [validator_mandates](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L775) field. + +After that, for each height in the epoch, [EpochInfo::sample_chunk_validators](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L1224) is called to return `ChunkValidatorStakeAssignment`. It is `Vec>` where s-th element corresponds to s-th shard in the epoch, contains ids of all chunk validator for that height and shard, alongside with its total mandate stake assigned to that shard. +`sample_chunk_validators` basically just shuffles `validator_mandates` among shards using height-specific seed. If there are no more than 1/3 malicious validators, then by Chernoff bound the probability that at least one shard is corrupted is small enough. **This is a reason why we can split validators among shards and still rely on basic consensus assumption**. + +This way, everyone tracking block headers can compute chunk validator assignment for each height and shard. + +### Limits + +`ChunkStateWitness` is relatively large message. Given large number of receivers as well, its size must be strictly limited. +If `ChunkStateWitness` for some state transition gets so uncontrollably large that it never can be handled by majority of validators, then its shard gets stuck. + +All the limits are described [here](https://github.com/near/nearcore/blob/b34db1e2281fbfe1d99a36b4a90df3fc7f5d00cb/docs/misc/state_witness_size_limits.md). +Additionally, we have limit on currently stored chunk endorsements, because malicious chunk validators can spam these as well. + +### Partial state witness distribution + +TODO + +### Protocol upgrade + +The good property of the approach taken is that protocol upgrade happens almost seamlessly. + +If (main transition, implicit transitions) fully belong to the protocol version before upgrade to stateless validation, chunk validator endorsements are not distributed, chunk validators are not sampled, but the protocol is safe because of all-shards tracking, as we described in "High-level flow". + +If at least some transition belongs to the protocol version after upgrade, chunk header height also belongs to epoch after upgrade, so it has chunk validators assigned and requirement of 2/3 endorsements is enabled. + +The minor accuracy needed is that generating and saving of state transition proofs have to be saved one epoch in advance, so we won't have to re-apply chunks to generate proofs once stateless validation is enabled. But new epoch protocol version is defined by finalization of **previous previous epoch**, so this is fine. + +It also assumes that each epoch has at least two chunks, but if this is not the case, the chain is having a major disruption which never happened before. ## Security Implications [Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] + + ## Alternatives [Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] +TODO ## Future possibilities -[Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] +* Integration with ZK allowing to get rid of large state witness distribution. If we treat state witness as a proof and ZK-ify it, anyone can validate that state witness indeed proves the new chunk header with much lower effort. Complexity of actual proof generation and computation indeed increases, but it can be distributed among chunk producers, and we can have separate concept of finality while allowing generic users to query optimistic chunks. +* TODO [Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] ## Consequences [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] +TODO ### Positive From 28b5fb29444526d161010d54b9ee839a1d219cbd Mon Sep 17 00:00:00 2001 From: Longarithm Date: Fri, 14 Jun 2024 00:27:02 +0400 Subject: [PATCH 10/28] linter linebreaks --- neps/nep-0509.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index ff87bc8c6..56bb9cacb 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -52,6 +52,7 @@ As a result, the team sought alternative approaches and concluded that stateless ### Current design The current high-level chunk production flow, excluding details and edge cases, is as follows: + * Block producer at height `H` `BP(H)` produces block `B(H)` with chunks accessible to it and distributes it. * Chunk producer for shard `S` at height `H+1` `CP(S, H+1)` produces chunk `C(S, H+1)` based on `B(H)` and distributes it. * `BP(H+1)` collects all chunks at height `H+1` until certain timeout is reached. @@ -115,6 +116,7 @@ Here we carefully describe new structures and logic introduced, without going in ### Validator Structure Change #### Roles + Currently, there are two different types of validators. Their responsibilities are defined as in the following pseudocode: ```python @@ -126,6 +128,7 @@ roles(validator).append("chunk producer") The validators are ordered by non-increasing stake in the considered epoch. Here and below by "block production" we mean both production and validation. With stateless validation, this structure must change for several reasons: + * Chunk production is the most resource consuming activity. * (Only) chunk production needs state in memory while other responsibilities can be completed via acquiring state witness * Chunk production does not have to be performed by all validators. @@ -157,6 +160,7 @@ roles(validator).append("chunk validator") #### Rewards Reward for each validator is defined as `total_epoch_reward * validator_relative_stake * work_quality_ratio`, where: + * `total_epoch_reward` is selected so that total inflation of the token is 5% per annum; * `validator_relative_stake = validator_stake / total_epoch_stake`; * `work_quality_ratio` is the measure of the work quality from 0 to 1. @@ -171,6 +175,7 @@ Similarly, `produced_chunks/expected_chunks` is a quality for chunk producer. It is more accurate to say `included_chunks/expected_chunks`, because inclusion of chunk in block is a final decision of a block producer which defines success here. Ideally, we could compute quality for chunk validator as `produced_endorsements/expected_endorsements`. Unfortunately, we won't do it in Stage 0 because: + * Mask of endorsements is not part of the block header, and it would be a significant change; * Block producer doesn't have to wait for all endorsements to be collected, so it could be unfair to say that endorsement was not produced if block producer just went ahead. @@ -192,6 +197,7 @@ if validator is chunk producer and chunk_producer_quality_ratio < 0.8: ``` For chunk validator, we apply absolutely the same formula. However, because: + * the formula doesn't count endorsements explicitly * for chunk producers it kind of just makes chunk production condition stronger without adding value @@ -222,7 +228,8 @@ The exact algorithm needs a thorough description to satisfy different edge cases ### ChunkStateWitness The full structure is described [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/core/primitives/src/stateless_validation.rs#L247). -Let's construct it sequentially together with explaining why every field is needed. Start from simple data: +Let's construct it sequentially together with explaining why every field is needed. Start from simple data: + ```rust pub struct ChunkStateWitness { pub chunk_producer: AccountId, @@ -251,6 +258,7 @@ So, chunk validator needs some way to run transactions and receipts from the pre ``` where + ```rust /// Represents the base state and the expected post-state-root of a chunk's state /// transition. The actual state transition itself is not included here. @@ -286,12 +294,14 @@ So we add another field: What about missing chunks though? Unfortunately, production and inclusion of any chunk **cannot be guaranteed**: + * chunk producer may go offline; * chunk validators may not generate 2/3 endorsements; * block producer may not receive enough information to include chunk. Let's handle this case as well. First, each chunk producer needs not just to prove main state transition, but also all state transitions for latest missing chunks: + ```rust /// For each missing chunk after the last new chunk of the shard, we need /// to carry out an implicit state transition. This is technically needed From dd9ba0901d72473d17b0897170f4789bf8d4ffae Mon Sep 17 00:00:00 2001 From: Anton Puhach Date: Thu, 13 Jun 2024 23:43:55 +0200 Subject: [PATCH 11/28] add security implications --- neps/nep-0509.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 56bb9cacb..036d46fd8 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -402,7 +402,10 @@ It also assumes that each epoch has at least two chunks, but if this is not the ## Security Implications -[Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] +Block validators no longer required to track any shard which means they don't have to validate state transitions proposed by the chunks in the block. Instead they trust chunk endorsements included in the block to certify the validity of the state transitions. +This makes chunk validator selection algorithm correctness critical for the security of the whole protocol, which is probabilistic by nature unlike the current more strict 2/3 of non-malicious validators requirement. + +It is also worth mentioning that large state witness size makes witness distribution slow which could result in a missing chunk because the block producer won't get chunk endorsements in time. This design tries to address that by meticulously limiting max witness size (see [this doc](https://github.com/near/nearcore/blob/master/docs/misc/state_witness_size_limits.md)). From ca02c290f10b9da54a67d89b480b21b6464f1912 Mon Sep 17 00:00:00 2001 From: Jan Ciolek <149345204+jancionear@users.noreply.github.com> Date: Fri, 14 Jun 2024 17:11:04 +0200 Subject: [PATCH 12/28] Add information about state witness size limits to NEP 509 (#547) --- neps/nep-0509.md | 65 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 036d46fd8..73d5ad08b 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -384,6 +384,71 @@ If `ChunkStateWitness` for some state transition gets so uncontrollably large th All the limits are described [here](https://github.com/near/nearcore/blob/b34db1e2281fbfe1d99a36b4a90df3fc7f5d00cb/docs/misc/state_witness_size_limits.md). Additionally, we have limit on currently stored chunk endorsements, because malicious chunk validators can spam these as well. +## State witness size limits + +A number of new limits will be introduced in order to keep the size of `ChunkStateWitness` reasonable. +`ChunkStateWitness` contains all the incoming transactions and receipts that will be processed during chunk application and in theory a single receipt could be tens of megabatytes in size. Distributing a `ChunkStateWitness` this large would be troublesome, so we limit the size and number of transactions, receipts, etc. The limits aim to keep the total uncompressed size of `ChunkStateWitness` under 16MiB. + +There are two types of size limits: +* Hard limit - the size must be below this limit, anything else is considered invalid +* Soft limit - things are added until the limit is exceeded, after that things stop being added. The last added thing is allowed to slightly exceed the limit. + +The limits are: +* `max_transaction_size = 1.5 MiB` + * All transactions must be below 1.5 MiB, otherwise they'll be considered invalid and rejected. + * Previously was 4MiB, now reduced to 1.5MiB +* `max_receipt_size - 4 MiB`: + * All receipts must be below 4 MiB, otherwise they'll be considered invalid and rejected. + * Previously there was no limit on receipt size. Set to 4MiB, might be reduced to 1.5MiB in the future to match the transaction limit. +* `combined_transactions_size_limit - 2 MiB` + * Hard limit on total size of transactions from this and previous chunk. `ChunkStateWitness` contains transactions from two chunks, this limit applies to the sum of their sizes. +* `new_transactions_validation_state_size_soft_limit - 500 KiB` + * Validating new transactions generates storage proof (recorded trie nodes), which has to be limited. Once transaction validation generates more storage proof than this limit, the chunk producer stops adding new transactions to the chunk. +* `per_receipt_storage_proof_size_limit - 4 MB` + * Executing a receipt generates storage proof. A single receipt is allowed to generate at most 4MB of storage proof. This is a hard limit, receipts which generate more than that will fail. +* `main_storage_proof_size_soft_limit - 3 MB` + * This is a limit on the total size of storage proof generated by receipts in one chunk. Once receipts generate more storage proof than this limit, the chunk producer stops processing receipts and moves the rest to the delayed queue. + * It's a soft limit, which means that the total size of storage proof could reach 7 MB (2.99MB + one receipt which generates 4MB of storage proof) + * Due to implementation details it's hard to find the exact amount of storage proof generated by a receipt, so an upper bound estimation is used instead. This upper bound assumes that every removal generates additional 2000 bytes of storage proof, so receipts which perform a lot of trie removals might be limited more than theoretically applicable. +* `outgoing_receipts_usual_size_limit - 100 KiB` + * Limit on the size of outgoing receipts to another shard. Needed to keep the size of `source_receipt_proofs` small. + * On most block heights a shard isn't allowed to send receipts larger than 100 KiB to another shard. +* `outgoing_receipts_big_size_limit - 4.5 MiB` + * On every block height there's one special "allowed shard" which is allowed to send larger receipts, up to 4.5 MiB in total. + * A receiving shard will receive receipts from `num_shards - 1` shards using the usual limit and one shard using the big limit. + * The "allowed shard" is the same shard as in cross-shard congestion control. It's chosen in a round-robin fashion, at height 1 the special shard is 0, at height 2 it's 1 and so on. + +In total that gives 2 MiB + 500 KiB + 7MB + 5*100 KiB + 4.5 MiB ~= 14 MiB of maximum witness size + +### New limits breaking contracts + +The new limits will break some existing contracts (for example all transctions larger than 1.5 MiB). This is sad, but it's necessary. Stateless validation uses much more network bandwitch than the previous approach, as it has to send over all state on each chunk application. Because network bandwitch is limited, stateless validation +can't support some operations that were allowed in the previous design. + +In the past year (31,536,000 blocks) there were only 679 transactions bigger than 1.5MiB, sent between 164 unique (sender -> receiver) pairs. +Only 0.002% of blocks contain such transactions, so the hope is that the breakage will be minimal. Contracts generally shouldn't require more than 1.5MiB of WASM. + +The full list of transactions from the past year which would fail with the new limit is available here: https://gist.github.com/jancionear/4cf373aff5301a5905a5f685ff24ed6f +Contract developers can take a look at this list and see if their contract will be affected. + +### Validating the limits + +Chunk validators have to verify that chunk producer respected all of the limits while producing the chunk. This means that validators also have to keep track of recorded storage proof by recording all trie accesses and they have to enforce the limits. +If it turns out that some limits weren't respected, the validators will generate a different result of chunk application and they won't endorse the chunk. + +### Missing chunks + +When a shard is missing some chunks, the following chunk on that shard will receive receipts from multiple blocks. This could lead to large `source_receipt_proofs` so a mechanism is added to reduce the impact. If there are two or more missing chunks in a row, +the shard is considered fully congested and no new receipts will be sent to it (unless it's the `allowed_shard` to avoid deadlocks). + +## Validator Role Change +Currently, there are two different types of validators and their responsibilities are as follows: +| | Top ~50% validators | Remaining validatiors (Chunk only producers) | +|-----|:-----:|:----:| +| block production | Y | N | +| chunk production | Y | Y | +| block validation | Y | N | + ### Partial state witness distribution TODO From 763427545457933ad31f1da90308aaaab81486e5 Mon Sep 17 00:00:00 2001 From: Jan Ciolek <149345204+jancionear@users.noreply.github.com> Date: Fri, 14 Jun 2024 20:40:46 +0200 Subject: [PATCH 13/28] Update documentation about combined_transactions_size_limit (#549) In https://github.com/near/nearcore/pull/11582 we're increasing `combined_transactions_size_limit`, so let's update the NEP to match the implementation. --- neps/nep-0509.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 73d5ad08b..3699d2163 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -400,7 +400,7 @@ The limits are: * `max_receipt_size - 4 MiB`: * All receipts must be below 4 MiB, otherwise they'll be considered invalid and rejected. * Previously there was no limit on receipt size. Set to 4MiB, might be reduced to 1.5MiB in the future to match the transaction limit. -* `combined_transactions_size_limit - 2 MiB` +* `combined_transactions_size_limit - 4 MiB` * Hard limit on total size of transactions from this and previous chunk. `ChunkStateWitness` contains transactions from two chunks, this limit applies to the sum of their sizes. * `new_transactions_validation_state_size_soft_limit - 500 KiB` * Validating new transactions generates storage proof (recorded trie nodes), which has to be limited. Once transaction validation generates more storage proof than this limit, the chunk producer stops adding new transactions to the chunk. @@ -418,7 +418,7 @@ The limits are: * A receiving shard will receive receipts from `num_shards - 1` shards using the usual limit and one shard using the big limit. * The "allowed shard" is the same shard as in cross-shard congestion control. It's chosen in a round-robin fashion, at height 1 the special shard is 0, at height 2 it's 1 and so on. -In total that gives 2 MiB + 500 KiB + 7MB + 5*100 KiB + 4.5 MiB ~= 14 MiB of maximum witness size +In total that gives 4 MiB + 500 KiB + 7MB + 5*100 KiB + 4.5 MiB ~= 16 MiB of maximum witness size. Possibly a little more on missing chunks. ### New limits breaking contracts From 60698056fceb1451b29befe27b00f6a1a125f9b8 Mon Sep 17 00:00:00 2001 From: wacban Date: Mon, 17 Jun 2024 10:54:28 +0100 Subject: [PATCH 14/28] future possibilities and consequences (#548) --- neps/nep-0509.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 3699d2163..97830ea67 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -473,34 +473,37 @@ This makes chunk validator selection algorithm correctness critical for the secu It is also worth mentioning that large state witness size makes witness distribution slow which could result in a missing chunk because the block producer won't get chunk endorsements in time. This design tries to address that by meticulously limiting max witness size (see [this doc](https://github.com/near/nearcore/blob/master/docs/misc/state_witness_size_limits.md)). - ## Alternatives -[Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] -TODO +The only real alternative that was considered is the original nightshade proposal. The full overview of the differences can be found in the revised nightshade whitepaper at https://near.org/papers/nightshade. ## Future possibilities * Integration with ZK allowing to get rid of large state witness distribution. If we treat state witness as a proof and ZK-ify it, anyone can validate that state witness indeed proves the new chunk header with much lower effort. Complexity of actual proof generation and computation indeed increases, but it can be distributed among chunk producers, and we can have separate concept of finality while allowing generic users to query optimistic chunks. -* TODO [Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] +* Integration with resharding to further increase the number of shards and the total throughput. +* The sharding of non-validating nodes and services. There are a number of services that may benefit from tracking only a subset of shards. Some examples include the RPC, archival and read-RPC nodes. ## Consequences -[This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] -TODO - ### Positive -- The validator nodes will only need to track one shard. -- +- The validator nodes will need to track at most one shard. +- The state will be held in memory making the chunk application much faster. +- The disk space hardware requirement will decrease. The top 100 nodes will need to store at most 2 shards at a time and the remaining nodes will not need to store any shards. +- Thanks to the above, in the future, it will be possible to reduce the gas costs and by doing so increase the throughput of the system. ### Neutral -- n1 +- The current approach to resharding will need to be revised to support generating state witness. +- The security assumptions will change. The responsibility will be moved from block producers to chunk validators and the security will become probabilistic. ### Negative -- n1 +- The network bandwidth and memory hardware requirements will increase. + - The top 100 validators will need to store up to 2 shards in memory and participate in state witness distribution. + - The remaining validators will need to participate in state witness distribution. +- Additional limits will be put on the size of transactions, receipts and, more generally, cross shard communication. +- The dependency on cloud state sync will increase the centralization of the blockchain. This will be resolved separately by the decentralized state sync. ### Backwards Compatibility From c211773c938af8539d3afac66c5ccb1a60407e85 Mon Sep 17 00:00:00 2001 From: Shreyan Gupta Date: Tue, 18 Jun 2024 14:14:04 -0700 Subject: [PATCH 15/28] Add section about partial witness distribution (#550) --- neps/nep-0509.md | 73 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 58 insertions(+), 15 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 97830ea67..4c3359103 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -345,7 +345,7 @@ It is basically a triple of `(ChunkHash, AccountId, Signature)`. Receiving this message means that specific chunk validator account endorsed chunk with specific chunk hash. Ideally chunk validator would send chunk endorsement to just the next block producer at the same height for which chunk was produced. However, block at that height can be skipped and block producers at heights h+1, h+2, ... will have to pick up the chunk. -To address that, we send `ChunkEndorsement` to all block producers at heights from h to `h+d-1`. We pick `d=5` as more than 5 skipped blocks in a row are very unlikely to occur. +To address that, we send `ChunkEndorsement` to all block producers at heights from `h` to `h+d-1`. We pick `d=5` as more than 5 skipped blocks in a row are very unlikely to occur. On block producer side, chunk endorsements are collected and stored in `ChunkEndorsementTracker`. Small **caveat** is that *sometimes* chunk endorsement may be received before chunk header which is required to understand that sender is indeed a validator of the chunk. @@ -376,36 +376,36 @@ After that, for each height in the epoch, [EpochInfo::sample_chunk_validators](h This way, everyone tracking block headers can compute chunk validator assignment for each height and shard. -### Limits +### Size limits `ChunkStateWitness` is relatively large message. Given large number of receivers as well, its size must be strictly limited. If `ChunkStateWitness` for some state transition gets so uncontrollably large that it never can be handled by majority of validators, then its shard gets stuck. -All the limits are described [here](https://github.com/near/nearcore/blob/b34db1e2281fbfe1d99a36b4a90df3fc7f5d00cb/docs/misc/state_witness_size_limits.md). -Additionally, we have limit on currently stored chunk endorsements, because malicious chunk validators can spam these as well. +We try to limit the size of the `ChunkStateWitness` to 16 MiB. All the limits are described [in this section](https://github.com/near/nearcore/blob/b34db1e2281fbfe1d99a36b4a90df3fc7f5d00cb/docs/misc/state_witness_size_limits.md). +Additionally, we have limit on currently stored partial state witnesses and chunk endorsements, because malicious chunk validators can spam these as well. ## State witness size limits A number of new limits will be introduced in order to keep the size of `ChunkStateWitness` reasonable. -`ChunkStateWitness` contains all the incoming transactions and receipts that will be processed during chunk application and in theory a single receipt could be tens of megabatytes in size. Distributing a `ChunkStateWitness` this large would be troublesome, so we limit the size and number of transactions, receipts, etc. The limits aim to keep the total uncompressed size of `ChunkStateWitness` under 16MiB. +`ChunkStateWitness` contains all the incoming transactions and receipts that will be processed during chunk application and in theory a single receipt could be tens of megabatytes in size. Distributing a `ChunkStateWitness` this large to all chunk validators would be troublesome, so we limit the size and number of transactions, receipts, etc. The limits aim to keep the total uncompressed size of `ChunkStateWitness` under 16 MiB. There are two types of size limits: -* Hard limit - the size must be below this limit, anything else is considered invalid -* Soft limit - things are added until the limit is exceeded, after that things stop being added. The last added thing is allowed to slightly exceed the limit. +* Hard limit - The size must be below this limit, anything else is considered invalid. This is usually used in the context of having limits for a single item. +* Soft limit - Things are added until the limit is exceeded, after that things stop being added. The last added thing is allowed to slightly exceed the limit. This is used in the context of having limits for a list of items. The limits are: -* `max_transaction_size = 1.5 MiB` +* `max_transaction_size - 1.5 MiB` * All transactions must be below 1.5 MiB, otherwise they'll be considered invalid and rejected. - * Previously was 4MiB, now reduced to 1.5MiB + * Previously was 4 MiB, now reduced to 1.5 MiB * `max_receipt_size - 4 MiB`: * All receipts must be below 4 MiB, otherwise they'll be considered invalid and rejected. - * Previously there was no limit on receipt size. Set to 4MiB, might be reduced to 1.5MiB in the future to match the transaction limit. + * Previously there was no limit on receipt size. Set to 4 MiB, might be reduced to 1.5 MiB in the future to match the transaction limit. * `combined_transactions_size_limit - 4 MiB` * Hard limit on total size of transactions from this and previous chunk. `ChunkStateWitness` contains transactions from two chunks, this limit applies to the sum of their sizes. * `new_transactions_validation_state_size_soft_limit - 500 KiB` * Validating new transactions generates storage proof (recorded trie nodes), which has to be limited. Once transaction validation generates more storage proof than this limit, the chunk producer stops adding new transactions to the chunk. * `per_receipt_storage_proof_size_limit - 4 MB` - * Executing a receipt generates storage proof. A single receipt is allowed to generate at most 4MB of storage proof. This is a hard limit, receipts which generate more than that will fail. + * Executing a receipt generates storage proof. A single receipt is allowed to generate at most 4 MB of storage proof. This is a hard limit, receipts which generate more than that will fail. * `main_storage_proof_size_soft_limit - 3 MB` * This is a limit on the total size of storage proof generated by receipts in one chunk. Once receipts generate more storage proof than this limit, the chunk producer stops processing receipts and moves the rest to the delayed queue. * It's a soft limit, which means that the total size of storage proof could reach 7 MB (2.99MB + one receipt which generates 4MB of storage proof) @@ -441,6 +441,53 @@ If it turns out that some limits weren't respected, the validators will generate When a shard is missing some chunks, the following chunk on that shard will receive receipts from multiple blocks. This could lead to large `source_receipt_proofs` so a mechanism is added to reduce the impact. If there are two or more missing chunks in a row, the shard is considered fully congested and no new receipts will be sent to it (unless it's the `allowed_shard` to avoid deadlocks). +## ChunkStateWitness distribution + +For chunk production, the chunk producer is required to distribute the chunk state witness to all the chunk validators. The chunk validators then validate the chunk and send the chunk endorsement to the block producer. Chunk state witness distribution is on a latency critical path. + +As we saw in the section above, the maximum size of the state witness can be ~16 MiB. If the chunk producer were to send the chunk state witness to all the chunk validators it would add a massive bandwidth requirement for the chunk producer. To ease and distribute the network requirements across all the chunk producers, we have a distribution mechanism similar to what we have for chunks in the shards manager. We divide the chunk state witness into a number of parts, and let the chunk validators distribute the parts among themselves, and later reconstruct the chunk state witness. + +### Distribution mechanism + +A chunk producer divides the state witness into a set of `N` parts where `N` is the number of chunk validators. The parts or partial witnesses are represented as [PartialEncodedStateWitness](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/core/primitives/src/stateless_validation.rs#L40). Each chunk validator is the owner of one part. The chunk producer uses the [PartialEncodedStateWitnessMessage](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/chain/network/src/state_witness.rs#L11) to send each part to their respective owners. The chunk validator part owners, on receiving the `PartialEncodedStateWitnessMessage`, forward this part to all other chunk validators via the [PartialEncodedStateWitnessForwardMessage](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/chain/network/src/state_witness.rs#L15). Each validator then uses the partial witnesses received to reconstruct the full chunk state witness. + +We have a separate [PartialWitnessActor](https://github.com/near/nearcore/blob/66d3b134343d9f35f6e0b437ebbdbef3e4aa1de3/chain/client/src/stateless_validation/partial_witness/partial_witness_actor.rs#L32) actor/module that is responsible for dividing the state witness into parts, distributing the parts, handling both partial encoded state witness message and the forward message, validating and storing the parts, and reconstructing the state witness from the parts and sending is to the chunk validation module. + +### Building redundancy using Reed Solomon Erasure encoding + +During the distribution mechanism, it's possible that some of the chunk validators are malicious, offline, or have a high network latency. Since chunk witness distribution is on the critical path for block production, we safeguard the distribution mechanism by building in redundancy using the Reed Solomon Erasure encoding. + +With Reed Solomon Erasure encoding, we can divde the chunk state witness into `N` total parts with `D` number of data parts. We can reconstruct the whole state witness as long as we have `D` of the `N` parts. The ratio of data parts `r = D/N` is something we can play around with. + +While reducing `r`, i.e. reducing the number of data parts required to reconstruct the state witness does allow for a more robust distribution mechanism, it comes with the cost of bloating the overall size of parts we need to distribute. If `S` is the size of the state witness, after reed solomon encoding, the total size `S'` of all parts becomes `S' = S/r` or `S' = S * N / D`. + +For the first release of stateless validation, we've kept the ratio as `0.6` representing that ~2/3rd of all chunk validators need to be online for chunk state witness distribution mechanism to work smoothly. + +One thing to note here is that the redundancy and upkeep requirement of 2/3rd is the _number_ of chunk validators and not the _stake_ of chunk validators. + +### PartialEncodedStateWitness structure + +The partial encoded state witness has the following fields: +- `(epoch_id, shard_id, height_created)` : These are the three fields that together uniquely determine the chunk associated with the partial witness. Since the chunk and chunk header distribution mechanism is independent of the partial witness, we rely on this triplet to uniquely identify which chunk is a part associated with. +- `part_ord` : The index or id of the part in the array of partial witnesses. +- `part` : The data associated with the part +- `encoded_length` : The total length of the state witness. This is required in the reed solomon decoding process to reconstruct the state witness. +- `signature` : Each part is signed by the chunk producer. This way the validity of the partial witness can be verified by the chunk validators receiving the parts. + +The `PartialEncodedStateWitnessTracker` module that is responsible for the storage and decoding of partial witnesses. This module has a LRU cache to store all the partial witnesses with `(epoch_id, shard_id, height_created)` triplet as the key. We reconstruct the state witness as soon as we have `D` of the `N` parts as forward the state witness to the validation module. + +### Network tradeoffs + +To get a sense of network requirements for validators with an without partial state witness distribution mechanism, we can do some quick back of the envelop calculations. Let `N` but the number of chunk validators, `S` be the size of the chunk state witness, `r` be the ratio of data parts to total parts for Reed Solomon Erasure encoding. + +Without the partial state witness distribution, each chunk producer would have to send the state witness to all chunk validators, which would require a bandwidth `B` of `B = N * S`. For the worst case of ~16 validators and ~16 MiB of state witness size, this can be a burst requirement of 2 Gbps. + +Partial state witness distribution takes this load off the chunk producer and distributes it evenly among all the chunk validators. However, we get an additional factor of `1/r` of extra data being transferred for redundancy. Each partial witness has a size of `P = S' / N` or `P = S / r / N`. The chunk producer and validators needs a bandwidth `B` of `B = P * N` or `B = S / r` to forward its owned part to all `N` chunk validators. For worst case of ~16 MiB of state witness size and encoding ratio of `0.6`, this works out to be ~214 Mbps, which is much more reasonable. + +### Future work + +In the Reed Solomon Erasure encoding section we discussed that the chunk state distribution mechanism relies on 2/3rd of the _number_ of chunk validators being available/non-malicious and not 2/3rd of the _total stake_ of the chunk validators. This can cause a potential issue where it's possible for more than 1/3rd of the chunk validators with small enough stake to be unavailable and cause the chunk production to stall. In the future we would like to address this problem. + ## Validator Role Change Currently, there are two different types of validators and their responsibilities are as follows: | | Top ~50% validators | Remaining validatiors (Chunk only producers) | @@ -449,10 +496,6 @@ Currently, there are two different types of validators and their responsibilitie | chunk production | Y | Y | | block validation | Y | N | -### Partial state witness distribution - -TODO - ### Protocol upgrade The good property of the approach taken is that protocol upgrade happens almost seamlessly. From 1d1e74b0b17d263dea31ed3e5dde0f30ea8d7845 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 24 Jun 2024 10:41:31 -0700 Subject: [PATCH 16/28] Update nep-0509.md fix lint warning and apply Michael's suggestion --- neps/nep-0509.md | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 4c3359103..842fd796c 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -33,7 +33,7 @@ As a result, the team sought alternative approaches and concluded that stateless ### Assumptions -* Not more than 1/3 of validators is corrupted. +* Not more than 1/3 of validators (by stake) is corrupted. * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) * State sync is enabled (so that nodes can track different shards across epochs) * Merkle Patricia Trie continues to be the state trie implementation @@ -390,33 +390,35 @@ A number of new limits will be introduced in order to keep the size of `ChunkSta `ChunkStateWitness` contains all the incoming transactions and receipts that will be processed during chunk application and in theory a single receipt could be tens of megabatytes in size. Distributing a `ChunkStateWitness` this large to all chunk validators would be troublesome, so we limit the size and number of transactions, receipts, etc. The limits aim to keep the total uncompressed size of `ChunkStateWitness` under 16 MiB. There are two types of size limits: + * Hard limit - The size must be below this limit, anything else is considered invalid. This is usually used in the context of having limits for a single item. * Soft limit - Things are added until the limit is exceeded, after that things stop being added. The last added thing is allowed to slightly exceed the limit. This is used in the context of having limits for a list of items. The limits are: + * `max_transaction_size - 1.5 MiB` - * All transactions must be below 1.5 MiB, otherwise they'll be considered invalid and rejected. - * Previously was 4 MiB, now reduced to 1.5 MiB + * All transactions must be below 1.5 MiB, otherwise they'll be considered invalid and rejected. +* Previously was 4 MiB, now reduced to 1.5 MiB * `max_receipt_size - 4 MiB`: - * All receipts must be below 4 MiB, otherwise they'll be considered invalid and rejected. - * Previously there was no limit on receipt size. Set to 4 MiB, might be reduced to 1.5 MiB in the future to match the transaction limit. + * All receipts must be below 4 MiB, otherwise they'll be considered invalid and rejected. + * Previously there was no limit on receipt size. Set to 4 MiB, might be reduced to 1.5 MiB in the future to match the transaction limit. * `combined_transactions_size_limit - 4 MiB` - * Hard limit on total size of transactions from this and previous chunk. `ChunkStateWitness` contains transactions from two chunks, this limit applies to the sum of their sizes. + * Hard limit on total size of transactions from this and previous chunk. `ChunkStateWitness` contains transactions from two chunks, this limit applies to the sum of their sizes. * `new_transactions_validation_state_size_soft_limit - 500 KiB` - * Validating new transactions generates storage proof (recorded trie nodes), which has to be limited. Once transaction validation generates more storage proof than this limit, the chunk producer stops adding new transactions to the chunk. + * Validating new transactions generates storage proof (recorded trie nodes), which has to be limited. Once transaction validation generates more storage proof than this limit, the chunk producer stops adding new transactions to the chunk. * `per_receipt_storage_proof_size_limit - 4 MB` - * Executing a receipt generates storage proof. A single receipt is allowed to generate at most 4 MB of storage proof. This is a hard limit, receipts which generate more than that will fail. + * Executing a receipt generates storage proof. A single receipt is allowed to generate at most 4 MB of storage proof. This is a hard limit, receipts which generate more than that will fail. * `main_storage_proof_size_soft_limit - 3 MB` - * This is a limit on the total size of storage proof generated by receipts in one chunk. Once receipts generate more storage proof than this limit, the chunk producer stops processing receipts and moves the rest to the delayed queue. - * It's a soft limit, which means that the total size of storage proof could reach 7 MB (2.99MB + one receipt which generates 4MB of storage proof) - * Due to implementation details it's hard to find the exact amount of storage proof generated by a receipt, so an upper bound estimation is used instead. This upper bound assumes that every removal generates additional 2000 bytes of storage proof, so receipts which perform a lot of trie removals might be limited more than theoretically applicable. + * This is a limit on the total size of storage proof generated by receipts in one chunk. Once receipts generate more storage proof than this limit, the chunk producer stops processing receipts and moves the rest to the delayed queue. + * It's a soft limit, which means that the total size of storage proof could reach 7 MB (2.99MB + one receipt which generates 4MB of storage proof) + * Due to implementation details it's hard to find the exact amount of storage proof generated by a receipt, so an upper bound estimation is used instead. This upper bound assumes that every removal generates additional 2000 bytes of storage proof, so receipts which perform a lot of trie removals might be limited more than theoretically applicable. * `outgoing_receipts_usual_size_limit - 100 KiB` - * Limit on the size of outgoing receipts to another shard. Needed to keep the size of `source_receipt_proofs` small. - * On most block heights a shard isn't allowed to send receipts larger than 100 KiB to another shard. + * Limit on the size of outgoing receipts to another shard. Needed to keep the size of `source_receipt_proofs` small. + * On most block heights a shard isn't allowed to send receipts larger than 100 KiB to another shard. * `outgoing_receipts_big_size_limit - 4.5 MiB` - * On every block height there's one special "allowed shard" which is allowed to send larger receipts, up to 4.5 MiB in total. - * A receiving shard will receive receipts from `num_shards - 1` shards using the usual limit and one shard using the big limit. - * The "allowed shard" is the same shard as in cross-shard congestion control. It's chosen in a round-robin fashion, at height 1 the special shard is 0, at height 2 it's 1 and so on. + * On every block height there's one special "allowed shard" which is allowed to send larger receipts, up to 4.5 MiB in total. + * A receiving shard will receive receipts from `num_shards - 1` shards using the usual limit and one shard using the big limit. + * The "allowed shard" is the same shard as in cross-shard congestion control. It's chosen in a round-robin fashion, at height 1 the special shard is 0, at height 2 it's 1 and so on. In total that gives 4 MiB + 500 KiB + 7MB + 5*100 KiB + 4.5 MiB ~= 16 MiB of maximum witness size. Possibly a little more on missing chunks. From e498ab7f00ceb268a5ca01b11183aff328e8406b Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 24 Jun 2024 10:54:17 -0700 Subject: [PATCH 17/28] Update nep-0509.md apply Michael's suggestion and fix more lint warnings --- neps/nep-0509.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 842fd796c..20ed14b26 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -364,7 +364,7 @@ This logic is triggered in `ChunkInclusionTracker` by methods [get_chunk_headers Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. -Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The amount of stake for a full mandate is a fixed parameter determined by the stake distribution of all validators, and any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. +Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The number of mandates per shard is a fixed parameter and the amount of stake per mandate is dynamically computed based on this parameter and the actual stake distribution; any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. For Stage 0, we select **target amount of mandates per shard** to 68, which was a [result of the latest research](https://near.zulipchat.com/#narrow/stream/407237-core.2Fstateless-validation/topic/validator.20seat.20assignment/near/435252304). With this number of mandates per shard and 6 shards, we predict the protocol to be secure for 40 years at 90% confidence. @@ -465,16 +465,16 @@ While reducing `r`, i.e. reducing the number of data parts required to reconstru For the first release of stateless validation, we've kept the ratio as `0.6` representing that ~2/3rd of all chunk validators need to be online for chunk state witness distribution mechanism to work smoothly. -One thing to note here is that the redundancy and upkeep requirement of 2/3rd is the _number_ of chunk validators and not the _stake_ of chunk validators. +One thing to note here is that the redundancy and upkeep requirement of 2/3rd is the *number* of chunk validators and not the *stake* of chunk validators. ### PartialEncodedStateWitness structure The partial encoded state witness has the following fields: -- `(epoch_id, shard_id, height_created)` : These are the three fields that together uniquely determine the chunk associated with the partial witness. Since the chunk and chunk header distribution mechanism is independent of the partial witness, we rely on this triplet to uniquely identify which chunk is a part associated with. -- `part_ord` : The index or id of the part in the array of partial witnesses. -- `part` : The data associated with the part -- `encoded_length` : The total length of the state witness. This is required in the reed solomon decoding process to reconstruct the state witness. -- `signature` : Each part is signed by the chunk producer. This way the validity of the partial witness can be verified by the chunk validators receiving the parts. +* `(epoch_id, shard_id, height_created)` : These are the three fields that together uniquely determine the chunk associated with the partial witness. Since the chunk and chunk header distribution mechanism is independent of the partial witness, we rely on this triplet to uniquely identify which chunk is a part associated with. +* `part_ord` : The index or id of the part in the array of partial witnesses. +* `part` : The data associated with the part +* `encoded_length` : The total length of the state witness. This is required in the reed solomon decoding process to reconstruct the state witness. +* `signature` : Each part is signed by the chunk producer. This way the validity of the partial witness can be verified by the chunk validators receiving the parts. The `PartialEncodedStateWitnessTracker` module that is responsible for the storage and decoding of partial witnesses. This module has a LRU cache to store all the partial witnesses with `(epoch_id, shard_id, height_created)` triplet as the key. We reconstruct the state witness as soon as we have `D` of the `N` parts as forward the state witness to the validation module. @@ -488,7 +488,7 @@ Partial state witness distribution takes this load off the chunk producer and di ### Future work -In the Reed Solomon Erasure encoding section we discussed that the chunk state distribution mechanism relies on 2/3rd of the _number_ of chunk validators being available/non-malicious and not 2/3rd of the _total stake_ of the chunk validators. This can cause a potential issue where it's possible for more than 1/3rd of the chunk validators with small enough stake to be unavailable and cause the chunk production to stall. In the future we would like to address this problem. +In the Reed Solomon Erasure encoding section we discussed that the chunk state distribution mechanism relies on 2/3rd of the *number* of chunk validators being available/non-malicious and not 2/3rd of the *total stake* of the chunk validators. This can cause a potential issue where it's possible for more than 1/3rd of the chunk validators with small enough stake to be unavailable and cause the chunk production to stall. In the future we would like to address this problem. ## Validator Role Change Currently, there are two different types of validators and their responsibilities are as follows: From cf26420ff0aebcae796180bd5548d5de78d55297 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 24 Jun 2024 12:37:06 -0700 Subject: [PATCH 18/28] Update nep-0509.md fix more lint warnings --- neps/nep-0509.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 20ed14b26..fb6e1e0bb 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -532,23 +532,23 @@ The only real alternative that was considered is the original nightshade proposa ### Positive -- The validator nodes will need to track at most one shard. -- The state will be held in memory making the chunk application much faster. -- The disk space hardware requirement will decrease. The top 100 nodes will need to store at most 2 shards at a time and the remaining nodes will not need to store any shards. -- Thanks to the above, in the future, it will be possible to reduce the gas costs and by doing so increase the throughput of the system. +* The validator nodes will need to track at most one shard. +* The state will be held in memory making the chunk application much faster. +* The disk space hardware requirement will decrease. The top 100 nodes will need to store at most 2 shards at a time and the remaining nodes will not need to store any shards. +* Thanks to the above, in the future, it will be possible to reduce the gas costs and by doing so increase the throughput of the system. ### Neutral -- The current approach to resharding will need to be revised to support generating state witness. -- The security assumptions will change. The responsibility will be moved from block producers to chunk validators and the security will become probabilistic. +* The current approach to resharding will need to be revised to support generating state witness. +* The security assumptions will change. The responsibility will be moved from block producers to chunk validators and the security will become probabilistic. ### Negative -- The network bandwidth and memory hardware requirements will increase. - - The top 100 validators will need to store up to 2 shards in memory and participate in state witness distribution. - - The remaining validators will need to participate in state witness distribution. -- Additional limits will be put on the size of transactions, receipts and, more generally, cross shard communication. -- The dependency on cloud state sync will increase the centralization of the blockchain. This will be resolved separately by the decentralized state sync. +* The network bandwidth and memory hardware requirements will increase. + * The top 100 validators will need to store up to 2 shards in memory and participate in state witness distribution. + * The remaining validators will need to participate in state witness distribution. +* Additional limits will be put on the size of transactions, receipts and, more generally, cross shard communication. +* The dependency on cloud state sync will increase the centralization of the blockchain. This will be resolved separately by the decentralized state sync. ### Backwards Compatibility @@ -558,9 +558,9 @@ The only real alternative that was considered is the original nightshade proposa [Explain any issues that warrant further discussion. Considerations -- What parts of the design do you expect to resolve through the NEP process before this gets merged? -- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? -- What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] +* What parts of the design do you expect to resolve through the NEP process before this gets merged? +* What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +* What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?] ## Changelog From 0a6ced4b579917a40427ac22211c92d662cb9314 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 8 Jul 2024 08:13:29 -0700 Subject: [PATCH 19/28] Update neps/nep-0509.md Co-authored-by: Marcelo Fornet --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index fb6e1e0bb..acc9215d9 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -63,7 +63,7 @@ The "induction base" is at genesis height, where genesis block with default chun One can observe that there is no "chunk validation" step here. In fact, validity of chunks is implicitly guaranteed by **requirement for all block producers to track all shards**. To achieve phase 2 of sharding, we want to drop this requirement. For that, we propose the following changes to the flow: -### New design +### Design after NEP-509 * Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header that is being produced. * `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct. From 6bfb3c679341b2a82af58615f47ce54ed383f8c9 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 8 Jul 2024 08:13:35 -0700 Subject: [PATCH 20/28] Update neps/nep-0509.md Co-authored-by: Marcelo Fornet --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index acc9215d9..62731e68f 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -49,7 +49,7 @@ As a result, the team sought alternative approaches and concluded that stateless * The cost of additional network and compute should be acceptable. * Validator rewards should not be reduced. -### Current design +### Design before NEP-509 The current high-level chunk production flow, excluding details and edge cases, is as follows: From faccd3fedbd0af775c82bb19791ce66b31f328ba Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 8 Jul 2024 08:13:46 -0700 Subject: [PATCH 21/28] Update neps/nep-0509.md Co-authored-by: Marcelo Fornet --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 62731e68f..98d34b20a 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -53,7 +53,7 @@ As a result, the team sought alternative approaches and concluded that stateless The current high-level chunk production flow, excluding details and edge cases, is as follows: -* Block producer at height `H` `BP(H)` produces block `B(H)` with chunks accessible to it and distributes it. +* Block producer at height `H`, `BP(H)`, produces block `B(H)` with chunks accessible to it and distributes it. * Chunk producer for shard `S` at height `H+1` `CP(S, H+1)` produces chunk `C(S, H+1)` based on `B(H)` and distributes it. * `BP(H+1)` collects all chunks at height `H+1` until certain timeout is reached. * `BP(H+1)` produces block `B(H+1)` with chunks `C(*, H+1)` accessible to it and distributes it, etc. From 880be3411e4f9c740547144e3278d58c92e7ccc0 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 8 Jul 2024 08:13:55 -0700 Subject: [PATCH 22/28] Update neps/nep-0509.md Co-authored-by: Marcelo Fornet --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 98d34b20a..b4b385123 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -54,7 +54,7 @@ As a result, the team sought alternative approaches and concluded that stateless The current high-level chunk production flow, excluding details and edge cases, is as follows: * Block producer at height `H`, `BP(H)`, produces block `B(H)` with chunks accessible to it and distributes it. -* Chunk producer for shard `S` at height `H+1` `CP(S, H+1)` produces chunk `C(S, H+1)` based on `B(H)` and distributes it. +* Chunk producer for shard `S` at height `H+1`, `CP(S, H+1)`, produces chunk `C(S, H+1)` based on `B(H)` and distributes it. * `BP(H+1)` collects all chunks at height `H+1` until certain timeout is reached. * `BP(H+1)` produces block `B(H+1)` with chunks `C(*, H+1)` accessible to it and distributes it, etc. From 39698149687f6c4c8e9c2e7750385658af0cb0fe Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 8 Jul 2024 08:14:06 -0700 Subject: [PATCH 23/28] Update neps/nep-0509.md Co-authored-by: Marcelo Fornet --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index b4b385123..4be2d67ec 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -142,7 +142,7 @@ if index(validator) < 100: roles(validator).append("chunk validator") ``` -The more stake validator has, the more **heavy** work it will get assigned, because we assume that validators with higher stakes have more powerful hardware. +The more stake validator has, the more **heavy** work it will get assigned. We expect that validators with higher stakes have more powerful hardware. With stateless validation, relative heaviness of the work changes. Comparing to the current order "block production" > "chunk production", the new order is "chunk production" > "block production" > "chunk validation". Shards are equally split among chunk producers: as in Mainnet on 12 Jun 2024 we have 6 shards, each shard would have ~16 chunk producers assigned. From 46ada3f18aeba877d6757d9ba9ecca2d5937c338 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 8 Jul 2024 08:14:15 -0700 Subject: [PATCH 24/28] Update neps/nep-0509.md Co-authored-by: Marcelo Fornet --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 4be2d67ec..6953f837c 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -130,7 +130,7 @@ The validators are ordered by non-increasing stake in the considered epoch. Here With stateless validation, this structure must change for several reasons: * Chunk production is the most resource consuming activity. -* (Only) chunk production needs state in memory while other responsibilities can be completed via acquiring state witness +* _Only_ chunk production needs state in memory while other responsibilities can be completed via acquiring state witness * Chunk production does not have to be performed by all validators. Hence, to make transition seamless, we change the role of nodes out of top 100 to only validate chunks: From 684b962dfa090daf14d63dca2ddfda9f616cb7a8 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Mon, 8 Jul 2024 08:14:27 -0700 Subject: [PATCH 25/28] Update neps/nep-0509.md Co-authored-by: Marcelo Fornet --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 6953f837c..8835d4183 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -424,7 +424,7 @@ In total that gives 4 MiB + 500 KiB + 7MB + 5*100 KiB + 4.5 MiB ~= 16 MiB of max ### New limits breaking contracts -The new limits will break some existing contracts (for example all transctions larger than 1.5 MiB). This is sad, but it's necessary. Stateless validation uses much more network bandwitch than the previous approach, as it has to send over all state on each chunk application. Because network bandwitch is limited, stateless validation +The new limits will break some existing contracts (for example, all transactions larger than 1.5 MiB). This is sad, but it's necessary. Stateless validation uses much more network bandwidth than the previous approach, as it has to send over all states on each chunk application. Because network bandwidth is limited, stateless validation can't support some operations that were allowed in the previous design. In the past year (31,536,000 blocks) there were only 679 transactions bigger than 1.5MiB, sent between 164 unique (sender -> receiver) pairs. From da9939880566a6932399366e0f9aaae38be2de17 Mon Sep 17 00:00:00 2001 From: Longarithm Date: Mon, 8 Jul 2024 23:11:27 +0400 Subject: [PATCH 26/28] feedback --- neps/nep-0509.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 8835d4183..e70918b3c 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -56,9 +56,9 @@ The current high-level chunk production flow, excluding details and edge cases, * Block producer at height `H`, `BP(H)`, produces block `B(H)` with chunks accessible to it and distributes it. * Chunk producer for shard `S` at height `H+1`, `CP(S, H+1)`, produces chunk `C(S, H+1)` based on `B(H)` and distributes it. * `BP(H+1)` collects all chunks at height `H+1` until certain timeout is reached. -* `BP(H+1)` produces block `B(H+1)` with chunks `C(*, H+1)` accessible to it and distributes it, etc. +* `BP(H+1)` produces block `B(H+1)` with chunks `C(*, H+1)` accessible to it and distributes it. -The "induction base" is at genesis height, where genesis block with default chunks is accessible to everyone, so chunk producers can start right away from genesis height + 1. +And the flow goes on for heights H+1, H+2, etc. The "induction base" is at genesis height, where genesis block with default chunks is accessible to everyone, so chunk producers can start right away from genesis height + 1. One can observe that there is no "chunk validation" step here. In fact, validity of chunks is implicitly guaranteed by **requirement for all block producers to track all shards**. To achieve phase 2 of sharding, we want to drop this requirement. For that, we propose the following changes to the flow: @@ -83,7 +83,7 @@ So the high-level specification can be described as the list of changes in the v * (Same as today) Produce blocks, (new) including waiting for chunk endorsements * (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding) * (New) No longer require tracking any shard - * (Same as today) Should have high barrier of entry for security reasons, to make block double signing harder. + * (Same as today) Should have high barrier of entry (required stake) for security reasons, to make block double signing harder. * Chunk producers: * (Same as today) Produce chunks * (New) Produces and distributes state witnesses to chunk validators @@ -180,7 +180,11 @@ Ideally, we could compute quality for chunk validator as `produced_endorsements/ * Block producer doesn't have to wait for all endorsements to be collected, so it could be unfair to say that endorsement was not produced if block producer just went ahead. So for now we decided to compute quality for chunk validator as ratio of `included_chunks/expected_chunks`, where we iterate over chunks which node was expected to validate. -The obvious drawback here is that if chunks are not produced at all, chunk validators will also be impacted. We plan to address it in the future releases. +It has clear drawbacks though: +* chunk validators are not incentivized to validate the chunks, given they will be rewarded the same in either case; +* if chunks are not produced at all, chunk validators will also be impacted. + +We plan to address them in the future releases. #### Kickouts From b711506cfb1e819667da4e40ec877e2946bb6860 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Fri, 19 Jul 2024 10:30:44 -0700 Subject: [PATCH 27/28] Update nep-0509.md lint error --- neps/nep-0509.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index e70918b3c..95c4a89c6 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -181,6 +181,7 @@ Ideally, we could compute quality for chunk validator as `produced_endorsements/ So for now we decided to compute quality for chunk validator as ratio of `included_chunks/expected_chunks`, where we iterate over chunks which node was expected to validate. It has clear drawbacks though: + * chunk validators are not incentivized to validate the chunks, given they will be rewarded the same in either case; * if chunks are not produced at all, chunk validators will also be impacted. @@ -474,6 +475,7 @@ One thing to note here is that the redundancy and upkeep requirement of 2/3rd is ### PartialEncodedStateWitness structure The partial encoded state witness has the following fields: + * `(epoch_id, shard_id, height_created)` : These are the three fields that together uniquely determine the chunk associated with the partial witness. Since the chunk and chunk header distribution mechanism is independent of the partial witness, we rely on this triplet to uniquely identify which chunk is a part associated with. * `part_ord` : The index or id of the part in the array of partial witnesses. * `part` : The data associated with the part @@ -495,6 +497,7 @@ Partial state witness distribution takes this load off the chunk producer and di In the Reed Solomon Erasure encoding section we discussed that the chunk state distribution mechanism relies on 2/3rd of the *number* of chunk validators being available/non-malicious and not 2/3rd of the *total stake* of the chunk validators. This can cause a potential issue where it's possible for more than 1/3rd of the chunk validators with small enough stake to be unavailable and cause the chunk production to stall. In the future we would like to address this problem. ## Validator Role Change + Currently, there are two different types of validators and their responsibilities are as follows: | | Top ~50% validators | Remaining validatiors (Chunk only producers) | |-----|:-----:|:----:| @@ -578,8 +581,8 @@ The only real alternative that was considered is the original nightshade proposa > List of benefits filled by the Subject Matter Experts while reviewing this version: -- Benefit 1 -- Benefit 2 +* Benefit 1 +* Benefit 2 #### Concerns From 425f4652dccf2a5dc33206408aba181a921fb459 Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Fri, 19 Jul 2024 10:32:00 -0700 Subject: [PATCH 28/28] Update nep-0509.md lint error --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 95c4a89c6..4b10254cc 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -130,7 +130,7 @@ The validators are ordered by non-increasing stake in the considered epoch. Here With stateless validation, this structure must change for several reasons: * Chunk production is the most resource consuming activity. -* _Only_ chunk production needs state in memory while other responsibilities can be completed via acquiring state witness +* *Only* chunk production needs state in memory while other responsibilities can be completed via acquiring state witness * Chunk production does not have to be performed by all validators. Hence, to make transition seamless, we change the role of nodes out of top 100 to only validate chunks: