Skip to content

Commit

Permalink
Update nep-0508.md
Browse files Browse the repository at this point in the history
Updating draft
  • Loading branch information
walnut-the-cat authored Sep 19, 2023
1 parent 746ea70 commit 2d80281
Showing 1 changed file with 42 additions and 16 deletions.
58 changes: 42 additions & 16 deletions neps/nep-0508.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,33 +10,53 @@ Created: 2022-09-19
LastUpdated: 2023-09-19
---

## Note
## Summary

Please refer to [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md) for resharding phase 1, where we performed resharding from 1 shard to 4 shards.
In essence, this NEP is extension of [NEP-40](https://github.com/near/NEPs/blob/master/specs/Proposals/0040-split-states.md), which focused splitting one shard into multiple shards.

## Summary
We are introducing the second phase of resharding, which supports one shard splitting into two within one epoch at pre-determined split boundary.

[Provide a short human-readable (~200 words) description of the proposal. A reader should get from this section a high-level understanding about the issue this NEP is addressing.]
While the new approach addresses critical limitations left unsolved in NEP-40 and is expected to remain valid for foreseable future, it does not serve all usecases, such as dynamic resharding.

## Motivation

[Explain why this proposal is necessary, how it will benefit the NEAR protocol or community, and what problems it solves. Also describe why the existing protocol specification is inadequate to address the problem that this NEP solves, and what potential use cases or outcomes.]
Currently, NEAR protocol has four shards. With more partners onboarding, we started seeing that some shards occasionally become over-crowded. In addition, with state sync and stateless validation, validators do not need to track all shards and validator hardware requirements can be greatly reduced with smaller shard size.

## Specification

[Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.]
### High level assumptions

Check failure on line 27 in neps/nep-0508.md

View workflow job for this annotation

GitHub Actions / markdown-lint

Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### High level assumptions"]

neps/nep-0508.md:27 MD022/blanks-around-headings/blanks-around-headers Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### High level assumptions"]
* Some form of State sync (centralized or decentralized) is enabled.

Check failure on line 28 in neps/nep-0508.md

View workflow job for this annotation

GitHub Actions / markdown-lint

Lists should be surrounded by blank lines [Context: "* Some form of State sync (cen..."]

neps/nep-0508.md:28 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "* Some form of State sync (cen..."]
* Flat state is enabled.
* Shard split boundary is predetermined. In other words, necessity of shard splitting is manually decided.
* Merkle Patricia Trie is undelying data structure for the protocol.
* Minimal epoch gap between two resharding events is X.

## Reference Implementation
### High level requirements

Check failure on line 34 in neps/nep-0508.md

View workflow job for this annotation

GitHub Actions / markdown-lint

Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### High level requirements"]

neps/nep-0508.md:34 MD022/blanks-around-headings/blanks-around-headers Headings should be surrounded by blank lines [Expected: 1; Actual: 0; Below] [Context: "### High level requirements"]
* Resharding should work even when validators stop tracking all shards.

Check failure on line 35 in neps/nep-0508.md

View workflow job for this annotation

GitHub Actions / markdown-lint

Lists should be surrounded by blank lines [Context: "* Resharding should work even ..."]

neps/nep-0508.md:35 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "* Resharding should work even ..."]
* Resharding should work after stateless validation is enabled.
* Resharding should be fast enough so that both state sync and resharding can happen within one epoch.
* Resharding should not require additional hardware from nodes.
* Resharding should be fault tolerant
* Chain must not stall in case of resharding failure.
* A validator should be able to recover in case they go offline during resharding.
* No transaction or receipt should be lost during resharding.
* Resharding should work regardless of number of existing shards.

### Required protocol changes

TBD. e.g. configuration changes we have to introduce

[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that:
### Required state changes

- Its interaction with other features is clear.
- Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.)
- It is reasonably clear how the feature would be implemented.
- Corner cases are dissected by example.
- For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges.
TBD. e.g. additional/updated data a node has to maintain

The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.]
### Resharding flow

TBD. how resharding happens at the high level

## Reference Implementation

TBD

## Security Implications

Expand All @@ -48,7 +68,13 @@ The section should return to the examples given in the previous section, and exp

## Future possibilities

[Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.]
As noted above, dynamic resharding is out of scope for this NEP and should be implemented in the future. Dynamic resharding includes the following but not limited to:
* automatic determination of split boundary

Check failure on line 72 in neps/nep-0508.md

View workflow job for this annotation

GitHub Actions / markdown-lint

Lists should be surrounded by blank lines [Context: "* automatic determination of s..."]

neps/nep-0508.md:72 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "* automatic determination of s..."]
* automatic shard splitting and merging based on traffic

Other useful features that can be considered as a follow up:
* account colocation for low latency across account call

Check failure on line 76 in neps/nep-0508.md

View workflow job for this annotation

GitHub Actions / markdown-lint

Lists should be surrounded by blank lines [Context: "* account colocation for low l..."]

neps/nep-0508.md:76 MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "* account colocation for low l..."]
* shard on demand

## Consequences

Expand All @@ -68,7 +94,7 @@ The section should return to the examples given in the previous section, and exp

### Backwards Compatibility

[All NEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. Author must explain a proposes to deal with these incompatibilities. Submissions without a sufficient backwards compatibility treatise may be rejected outright.]
We do not expect anything to break with this change. Yet, shard splitting can introduce additional complexity on replayability. For instance, as target shard of a receipt and belonging shard of an account can change with shard splitting, shard splitting must be replayed along with transactions at the exact epoch boundary.

## Unresolved Issues (Optional)

Expand Down

0 comments on commit 2d80281

Please sign in to comment.