NEP-568: Resharding v3

neps/nep-0568.md

@@ -0,0 +1,224 @@
+---
+NEP: 568
+Title: Resharding V3
+Authors: Adam Chudas, Aleksandr Logunov, Andrea Spurio, Marcelo Diop-Gonzalez, Shreyan Gupta, Waclaw Banasik
+Status: New
+DiscussionsTo: https://github.com/near/nearcore/issues/11881
+Type: Protocol
+Version: 1.0.0
+Created: 2024-10-24
+LastUpdated: 2024-10-24
+---
+
+## Summary 
+
+This proposal introduces a new resharding implementation and shard layout for
+production networks.
+
+Resharding V3 is a significantly redesigned approach, addressing limitations of
+the previous versions, [Resharding V1][NEP-040] and [Resharding V2][NEP-508].
+The earlier solutions became obsolete due to major protocol changes since
+Resharding V2, including the introduction of Stateless Validation, Single Shard
+Tracking, and Mem-Trie.
+
+The primary objective of Resharding V3 is to increase chain capacity by
+splitting overutilized shards. A secondary aim is to lay the groundwork for
+supporting Dynamic Resharding, Instant Resharding and Shard Merging in future
+updates.
+
+## Motivation
+
+```text
+[Explain why this proposal is necessary, how it will benefit the NEAR protocol or community, and what problems it solves. Also describe why the existing protocol specification is inadequate to address the problem that this NEP solves, and what potential use cases or outcomes.]
+```
+
+## Specification
+
+```text
+[Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.]
+```
+
+Resharding will be scheduled in advance by the NEAR developer team. The new
+shard layout will be hardcoded into the neard binary and linked to the protocol
+version. As the protocol upgrade progresses, resharding will be triggered during
+the post-processing phase of the last block of the epoch. At this point, the
+state of the parent shard will be split between two child shards. From the first
+block of the new protocol version onward, the chain will operate with the new
+shard layout.
+
+There are two key dimensions to consider: state storage and protocol features,
+along with a few additional details.
+
+1) State Storage: Currently, the state of a shard is stored in three distinct
+formats: the state, the flat state, and the mem-trie. Each of these
+representations must be resharded. Logically, resharding is an almost
+instantaneous event that occurs before the first block under the new shard
+layout. However, in practice, some of this work may be deferred to
+post-processing, as long as the chain's view reflects a fully resharded state.
+
+1) Protocol Features: Several protocol features must integrate smoothly with the
+  resharding process, including:
+
+* Stateless Validation: Resharding must be validated and proven through
+  stateless validation mechanisms.
+* State Sync: Nodes must be able to sync the states of the child
+  shards post-resharding.
+* Cross-Shard Traffic: Receipts sent to the parent shard may need to be
+  reassigned to one of the child shards.
+* Receipt Handling: Delayed, postponed, buffered, and promise-yield receipts
+  must be correctly distributed between the child shards.
+* ShardId Semantics: The shard identifiers will become abstract identifiers
+  where today they are number in the 0..num_shards range.
+
+### State Storage - Mem Trie
+
+### State Storage - Flat State
+
+Flat State is a collection of key-value pairs stored on disk and each entry
+contains a reference to its ShardId. When splitting a shard, every item inside
+its Flat State must be correctly reassigned to either one of the new children;
+due to technical limitations such operation can't be completed instantaneously.
+
+Flat State main purposes are allowing the creation of State Sync snapshots and
+the construction of Mem Tries. Fortunately, these two operations can be delayed
+until resharding is completed. Note also that with Mem Tries enabled the chain
+can move forward even if the current status of Flat State is not in sync with
+the latest block. 
+
+For the reason stated above, the chosen strategy is to reshard Flat State in a
+long-running background task. The new shards' states must converge with their
+Mem Tries representation in a reasonable amount of time.
+
+Splitting a shard's Flat State is performed in multiple steps:
+
+1) A post-processing 'split' task is created during the last block of the old
+   shard layout, instantaneously. 
+2) The 'split' task runs in parallel with the chain for a certain amount of
+   time. Inside this routine every key-value pair belonging to the shard being
+   split (also called parent shard) is copied into either the left or the right
+   child Flat State. Entries linked to receipts are handled in a special way.
+3) Once the task is completed, the parent shard Flat State is cleaned up. The
+   children shards Flat States have their state in sync with last block of the
+   old shard layout.
+4) Children shards must apply the delta changes from the first block of the new
+   shard layout until the final block of the canonical chain. This operation is
+   done in another background task to avoid slowdowns while processing blocks.
+5) Children shards Flat States are now ready and can be used to take State Sync
+   snapshots and to reload Mem Tries.
+
+### State Storage - State
+
+### Stateless Validation
+
+### State Sync
+
+### Cross Shard Traffic
+
+### Receipt Handling - Delayed, Postponed, PromiseYield
+
+### Receipt Handling - Buffered
+
+### ShardId Semantics
+
+## Reference Implementation
+
+```text
+[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that:
+
+* Its interaction with other features is clear.
+* Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.)
+* It is reasonably clear how the feature would be implemented.
+* Corner cases are dissected by example.
+* For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges.
+
+The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.]
+```
+
+## Security Implications
+
+```text
+[Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.]
+```
+
+## Alternatives
+
+```text
+[Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?]
+```
+
+## Future possibilities
+
+* Dynamic Resharding - In this proposal, resharding is scheduled in advance and hardcoded within the neard binary. In the future, we aim to enable the chain to dynamically trigger and execute resharding autonomously, allowing it to adjust capacity automatically based on demand.
+* Fast Dynamic Resharding - In the Dynamic Resharding extension, the new shard layout is configured for the second upcoming epoch. This means that a full epoch must pass before the chain transitions to the updated shard layout. In the future, our goal is to accelerate this process by finalizing the previous epoch more quickly, allowing the chain to adopt the new layout as soon as possible.
+* Shard Merging - In this proposal the only allowed resharding operation is shard splitting. In the future, we aim to enable shard merging, allowing underutilized shards to be combined with neighboring shards. This would allow the chain to free up resources and reallocate them where they are most needed.
+
+## Consequences
+
+```text
+[This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.]
+```
+
+### Positive
+
+* p1
+
+### Neutral
+
+* n1
+
+### Negative
+
+* n1
+
+### Backwards Compatibility
+
+```text
+[All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.]
+```
+
+## Unresolved Issues (Optional)
+
+```text
+[Explain any issues that warrant further discussion. Considerations
+
+* What parts of the design do you expect to resolve through the NEP process before this gets merged?
+* What parts of the design do you expect to resolve through the implementation of this feature before stabilization?
+* What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?]
+```
+
+## Changelog
+
+```text
+[The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.]
+```
+
+### 1.0.0 - Initial Version
+
+> Placeholder for the context about when and who approved this NEP version.
+
+#### Benefits
+
+> List of benefits filled by the Subject Matter Experts while reviewing this version:
+
+* Benefit 1
+* Benefit 2
+
+#### Concerns
+
+> Template for Subject Matter Experts review for this version:
+> Status: New | Ongoing | Resolved
+
+|    # | Concern | Resolution | Status |
+| ---: | :------ | :--------- | -----: |
+|    1 |         |            |        |
+|    2 |         |            |        |
+
+## Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
+
+
+<!-- links --> 
+
+[NEP-040]: https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md
+[NEP-508]: https://github.com/near/NEPs/blob/master/neps/nep-0508.md