NEP-508: Resharding v2

neps/nep-0508.md

@@ -0,0 +1,164 @@
+---
+NEP: 508
+Title: Resharding v2
+Authors: Waclaw Banasik, Shreyan Gupta, Yoon Hong
+Status: Draft
+DiscussionsTo: https://github.com/near/nearcore/issues/8992
+Type: Protocol
+Version: 1.0.0
+Created: 2022-09-19
+LastUpdated: 2023-09-19
+---
+
+## Summary
+
+In essence, this NEP is extension of [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md), which focused splitting one shard into multiple shards. 
+
+We are introducing resharding v2, which supports one shard splitting into two within one epoch at pre-determined split boundary. The NEP includes performance improvement to make resharding feasible under the current state as well as actual resharding in mainnet and testnet (To be specific, spliting shard 3 into two). 
+
+While the new approach addresses critical limitations left unsolved in NEP-40 and is expected to remain valid for foreseable future, it does not serve all usecases, such as dynamic resharding. 
+
+
+
+## Motivation
+
+Currently, NEAR protocol has four shards. With more partners onboarding, we started seeing that some shards occasionally become over-crowded. In addition, with state sync and stateless validation, validators do not need to track all shards and validator hardware requirements can be greatly reduced with smaller shard size. 
+
+## Specification
+
+### High level assumptions
+
+* Some form of State sync (centralized or decentralized) is enabled.
+* Flat state is enabled.
+* Shard split boundary is predetermined. In other words, necessity of shard splitting is manually decided.
+* Merkle Patricia Trie is the undelying data structure for the protocol state.
+* Minimal epoch gap between two resharding events is X.
+
+### High level requirements
+
+* Resharding should work even when validators stop tracking all shards.
+* Resharding should work after stateless validation is enabled.
+* Resharding should be fast enough so that both state sync and resharding can happen within one epoch.
+* ~~Resharding should not require additional hardware from nodes.~~
+  * This needs to be assessed during test
+* Resharding should be fault tolerant
+  * Chain must not stall in case of resharding failure.
+  * A validator should be able to recover in case they go offline during resharding.
+    * For now, our aim is at least allowing a validator to join back after resharding is finished. 
+* No transaction or receipt should be lost during resharding.
+* Resharding should work regardless of number of existing shards.
+* There should be no more place (in any apps or tools) where number of shard is hardcoded.
+
+### Out of scope
+
+* Dynamic resharding
+  * automatically scheduling resharding based on shard usage/capacity
+  * automatically determining the shard layout
+* merging shards
+* shard reshuffling
+* shard boundary adjustment
+* Shard determination logic (shard boundary is still determined by string value)
+* Advanced failure handling
+  * If a validator goes offline during resharding, it can join back immediately and move forward as long as enough time is left to reperform resharding.
+* TBD
+
+### Required protocol changes
+
+TBD. e.g. configuration changes we have to introduce
+
+### Required state changes
+
+TBD. e.g. additional/updated data a node has to maintain
+
+### Resharding flow
+
+TBD. how resharding happens at the high level
+
+## Reference Implementation
+
+TBD
+
+## Security Implications
+
+[Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.]
+
+## Alternatives
+
+* Why is this design the best in the space of possible designs?
+  * TBD
+* What other designs have been considered and what is the rationale for not choosing them?
+  * TBD
+* What is the impact of not doing this?
+  * TBD
+
+## Future possibilities
+
+As noted above, dynamic resharding is out of scope for this NEP and should be implemented in the future. Dynamic resharding includes the following but not limited to:
+* automatic determination of split boundary
+* automatic shard splitting and merging based on traffic
+
+Other useful features that can be considered as a follow up:
+* account colocation for low latency across account call
+* shard on demand
+
+## Consequences
+
+### Positive
+
+* Workload across shards will be more evenly distributed.
+* Required space to maintain state (either in memory or in persistent disk) will be smaller.
+* State sync overhead will be smaller.
+* TBD
+
+### Neutral
+
+* Number of shards is expected to increase.
+* Underlying trie structure and data structure are not going to change.
+* Resharding will create dependency on flat storage and state sync.
+
+### Negative
+
+* The resharding process is still not fully automated. Analyzing shard data, determining the split boundary, and triggering an actual shard split all need to be manually curated by a person.
+* During resharding, a node is expected to do more work as it will have to apply changes twice (for the current shard and future shard).
+* Increased potential for apps and tools to break without proper shard layout change handling.
+
+### Backwards Compatibility
+
+We do not expect anything to break with this change. Yet, shard splitting can introduce additional complexity on replayability. For instance, as target shard of a receipt and belonging shard of an account can change with shard splitting, shard splitting must be replayed along with transactions at the exact epoch boundary.
+
+## Unresolved Issues (Optional)
+
+[Explain any issues that warrant further discussion. Considerations
+
+- What parts of the design do you expect to resolve through the NEP process before this gets merged?
+- What parts of the design do you expect to resolve through the implementation of this feature before stabilization?
+- What related issues do you consider out of scope for this NEP that could be addressed in the future independently of the solution that comes out of this NEP?]
+
+## Changelog
+
+[The changelog section provides historical context for how the NEP developed over time. Initial NEP submission should start with version 1.0.0, and all subsequent NEP extensions must follow [Semantic Versioning](https://semver.org/). Every version should have the benefits and concerns raised during the review. The author does not need to fill out this section for the initial draft. Instead, the assigned reviewers (Subject Matter Experts) should create the first version during the first technical review. After the final public call, the author should then finalize the last version of the decision context.]
+
+### 1.0.0 - Initial Version
+
+> Placeholder for the context about when and who approved this NEP version.
+
+#### Benefits
+
+> List of benefits filled by the Subject Matter Experts while reviewing this version:
+
+- Benefit 1
+- Benefit 2
+
+#### Concerns
+
+> Template for Subject Matter Experts review for this version:
+> Status: New | Ongoing | Resolved
+
+|   # | Concern | Resolution | Status |
+| --: | :------ | :--------- | -----: |
+|   1 |         |            |        |
+|   2 |         |            |        |
+
+## Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).