jam-duna traces upd: Reverse trie key bit ordering (#10)

Based on w3f/jamtestvectors#14 Also: * jam-duna: assurances first trace: just 1 epoch * add note re bootstrap services, adjust codec => bin in README
jam-duna · Oct 18, 2024 · bc314fe · bc314fe
1 parent d6a30f5
commit bc314fe
Show file tree

Hide file tree

Showing 425 changed files with 12,970 additions and 11,170 deletions.
diff --git a/README.md b/README.md
@@ -46,20 +46,21 @@ Each validator communicates with other nodes using JAMNP's QUIC protocol. Depend
 
 | `mode`         | `safrole` | `assurance` | `finality` | `conformance`  |
 |---------------|-----------|-------------|------------|----------------|
-| QUIC / JAM Codec | Block, Ticket | WorkPackage, Guarantee, Assurance, EC TBD | TBD: Announcement, Vote, ...  | TBD: Dispute |
 | Tickets: E_T           |   x       |       x     |     x      |     x          |
 | Guarantees: E_G           |           |       x     |     x      |     x          |
 | Assurances: E_A           |           |       x     |     x      |     x          |
 | Preimages: E_P           |           |       x     |     x      |     x          |
 | Refine/Accumulate PVM  |         |       x     |     x      |     x          |
+| Bootstrap Services: Assign |         |     x      |     x       |     x          |
 | Audit/Announcements |     |             |     x      |     x          |
 | GRANDPA       |           |             |     x      |     x          |
 | Disputes: E_D |           |             |            |     x          |
 | BLS           |           |             |            |     x          |
 | BEEFY         |           |             |            |     x          |
 | Authorization Pool/Queue |           |             |            |     x          |
-| Privileged Services |     |             |            |     x          |
+| Bootstrap Services: Other |     |             |            |     x          |
 | State         | C4, C6, C7, C8, C9, C11, C13 | C10, C12 | - | C1, C2, C3, C5 |
+| JAMSNP        | CE128,129,131 | + CE133-135,137,141 | + CE142,143,144,145 | + CE132 |
 | Timeline      | Q4 2024   | Q1 2025     | early Q2 2025 | late Q2 2025 |
 
 We aim for 5-10 teams to successfully establish their own working testnets in Q4, with collaborative efforts beginning around sub0@Devcon7+JAM0 in mid-November.
@@ -83,17 +84,15 @@ directory.
 
 If you have service code (in any language, including privileged services) suitable for `mode=assurances`, put it in `services/${team}/${servicename}` (e.g.  `services/strongly-web3/fib`).  See [services](./services) for examples in progress.
 
-To model submission of work packages utilizing a service, traces can include the receipt by the first validator, e.g in `traces/${mode}/${team}/workpackages/${core}.{codec,json}$`
-
 ## JAM Implementer Submissions
 
-Building a collaborative JAM Testnet requires teams share Docker image URLs, Traces, and Testnet configurations, but not code.
+Building a collaborative JAM Testnet requires teams share Traces (supporting validation independent of JAMSNP) and Docker image URLs (requiring JAMSNP), but not code.
 
 To contribute:
 
-- Submit a PR to add a Docker image URL below.   
-- Add your fully working `docker-compose.yml` to  `testnet/${mode}/${team}` (e.g. `testnet/safrole/jam-duna/docker-compose.yml`)
-- Add your sample trace to `traces/${mode}/${team}` (e.g. `traces/safrole/jam-duna/data`)
+- Submit a PR to add your traces and/or a Docker image URL below.  
+- For traces of blocks and states, add your sample trace to `traces/${mode}/${team}` (e.g. `traces/safrole/jam-duna`)
+- For Docker image urls, add below and a fully working `docker-compose.yml` to  `testnet/${mode}/${team}` (e.g. `testnet/safrole/jam-duna/docker-compose.yml`)
 
 | team          | Docker Image URL                                       |
 |---------------|--------------------------------------------------------|
@@ -130,3 +129,5 @@ To contribute:
 | subjam        | TBD                                                    |
 
 For simplicity, use lowercase for `${team}` and `${mode}`.  Dashes and underscores are ok, spaces/punctations are not.
+
+Everyone must follow JAM Prize rules and can politely refuse collaboration with any team for any reason.
diff --git a/services/README.md b/services/README.md
@@ -4,7 +4,9 @@
 
 JAM Services consist of PVM Byte code placed on-chain with 4 entry points.  
 To support the testing of these JAM Services, JAM Implementation teams are encouraged to place
-JAM Service code, in any language here.  
+JAM Service code, in any language here.  Note that JAM Service code is not necessary for teams to
+collaborate on JAM testnet, but it may support useful debugging.  Services may include a
+_bootstrap service_ included in genesis state.
 
 * Raw assembly, with [polkatool](https://github.com/koute/polkavm/tree/master/tools/polkatool)
 * Rust, see [polkatool](https://github.com/koute/polkavm/tree/master/tools/polkatool)
@@ -21,10 +23,13 @@ Each JAM Service should have raw source code, build instructions, and a _JAM-rea
 * `accumulate` (entry point 10) 
 * `on_transfer` (entry point 15) 
 
-Furthermore, as privileged services themselves are needed to place
-code blobs on-chain and create JAM Services utilizing these code
-blobs, it is critical that teams have a "generic" privileged service
-to have a JAM Testnet in "assurances" mode.  This is an open question.
+Furthermore, a _bootstrap service_ is needed in genesis state to support many different privileged operations:
+* create JAM Services 
+* assigning coretime
+* requesting preimages
+* changing the validator keysets
+
+These always-accumulate/privileged services use the same JAMSNP work package submission+sharing processes as non-privileged service.  Teams are encouraged to share Bootstrap service code along with their genesis configs.
 
 ## Raw PVM Assembly code
 

diff --git a/traces/README.md b/traces/README.md
@@ -7,53 +7,72 @@ each others state transitions (including state roots) and check each others JAM
 
 If your team has a sequence of Blocks + States, please submit a PR going into this directory, containing at minimum easily parsed / decoded files for each `epoch` and $phase`
 
-* Blocks: `$epoch_$phase.json`  `$epoch_$phase.codec` 
-* States: `$epoch_$phase.json`  `$epoch_$phase.codec`
+* Blocks: `$epoch_$phase.json`  `$epoch_$phase.bin` 
+* States: `$epoch_$phase.json`  `$epoch_$phase.bin`
 
 The combination of JSON + codec is useful for debugging purposes.
 
 Here is a sample output:
 ```
-# ls safrole/$TEAM_NAME/blocks/
-0_0.codec   0_11.json  0_5.codec  0_8.json   1_10.codec  1_3.json   1_7.codec  2_0.json    2_2.codec  2_5.json   2_9.codec   3_10.json   3_4.codec  3_7.json
-0_0.json    0_2.codec  0_5.json   0_9.codec  1_10.json   1_4.codec  1_7.json   2_1.codec   2_2.json   2_6.codec  2_9.json    3_11.codec  3_4.json   3_8.codec
-0_1.codec   0_2.json   0_6.codec  0_9.json   1_11.codec  1_4.json   1_8.codec  2_1.json    2_3.codec  2_6.json   3_0.codec   3_11.json   3_5.codec  3_8.json
-0_1.json    0_3.codec  0_6.json   1_0.codec  1_11.json   1_5.codec  1_8.json   2_10.codec  2_3.json   2_7.codec  3_0.json    3_2.codec   3_5.json   3_9.codec
-0_10.codec  0_3.json   0_7.codec  1_0.json   1_2.codec   1_5.json   1_9.codec  2_10.json   2_4.codec  2_7.json   3_1.codec   3_2.json    3_6.codec  3_9.json
-0_10.json   0_4.codec  0_7.json   1_1.codec  1_2.json    1_6.codec  1_9.json   2_11.codec  2_4.json   2_8.codec  3_1.json    3_3.codec   3_6.json   4_0.codec
-0_11.codec  0_4.json   0_8.codec  1_1.json   1_3.codec   1_6.json   2_0.codec  2_11.json   2_5.codec  2_8.json   3_10.codec  3_3.json    3_7.codec  4_0.json
-
-# ls safrole/$TEAM_NAME/state_snapshots/
-0_0.codec   0_11.json  0_5.codec  0_8.json   1_10.codec  1_3.json   1_7.codec  2_0.json    2_2.codec  2_5.json   2_9.codec   3_10.json   3_4.codec  3_7.json
-0_0.json    0_2.codec  0_5.json   0_9.codec  1_10.json   1_4.codec  1_7.json   2_1.codec   2_2.json   2_6.codec  2_9.json    3_11.codec  3_4.json   3_8.codec
-0_1.codec   0_2.json   0_6.codec  0_9.json   1_11.codec  1_4.json   1_8.codec  2_1.json    2_3.codec  2_6.json   3_0.codec   3_11.json   3_5.codec  3_8.json
-0_1.json    0_3.codec  0_6.json   1_0.codec  1_11.json   1_5.codec  1_8.json   2_10.codec  2_3.json   2_7.codec  3_0.json    3_2.codec   3_5.json   3_9.codec
-0_10.codec  0_3.json   0_7.codec  1_0.json   1_2.codec   1_5.json   1_9.codec  2_10.json   2_4.codec  2_7.json   3_1.codec   3_2.json    3_6.codec  3_9.json
-0_10.json   0_4.codec  0_7.json   1_1.codec  1_2.json    1_6.codec  1_9.json   2_11.codec  2_4.json   2_8.codec  3_1.json    3_3.codec   3_6.json   4_0.codec
-0_11.codec  0_4.json   0_8.codec  1_1.json   1_3.codec   1_6.json   2_0.codec  2_11.json   2_5.codec  2_8.json   3_10.codec  3_3.json    3_7.codec  4_0.json
+# ls safrole/$TEAM_NAME/blocks
+349445_0.bin    349445_2.bin   349445_6.bin   349446_0.bin    349446_2.bin   349446_6.bin   349447_0.bin    349447_2.bin   349447_6.bin   349448_0.bin    349448_2.bin   349448_6.bin   349449_0.bin
+349445_0.json   349445_2.json  349445_6.json  349446_0.json   349446_2.json  349446_6.json  349447_0.json   349447_2.json  349447_6.json  349448_0.json   349448_2.json  349448_6.json  349449_0.json
+349445_1.bin    349445_3.bin   349445_7.bin   349446_1.bin    349446_3.bin   349446_7.bin   349447_1.bin    349447_3.bin   349447_7.bin   349448_1.bin    349448_3.bin   349448_7.bin   349449_1.bin
+349445_1.json   349445_3.json  349445_7.json  349446_1.json   349446_3.json  349446_7.json  349447_1.json   349447_3.json  349447_7.json  349448_1.json   349448_3.json  349448_7.json  349449_1.json
+349445_10.bin   349445_4.bin   349445_8.bin   349446_10.bin   349446_4.bin   349446_8.bin   349447_10.bin   349447_4.bin   349447_8.bin   349448_10.bin   349448_4.bin   349448_8.bin
+349445_10.json  349445_4.json  349445_8.json  349446_10.json  349446_4.json  349446_8.json  349447_10.json  349447_4.json  349447_8.json  349448_10.json  349448_4.json  349448_8.json
+349445_11.bin   349445_5.bin   349445_9.bin   349446_11.bin   349446_5.bin   349446_9.bin   349447_11.bin   349447_5.bin   349447_9.bin   349448_11.bin   349448_5.bin   349448_9.bin
+349445_11.json  349445_5.json  349445_9.json  349446_11.json  349446_5.json  349446_9.json  349447_11.json  349447_5.json  349447_9.json  349448_11.json  349448_5.json  349448_9.json
+# ls safrole/$TEAM_NAME/state_snapshots
+349445_0.bin    349445_2.bin   349445_6.bin   349446_0.bin    349446_2.bin   349446_6.bin   349447_0.bin    349447_2.bin   349447_6.bin   349448_0.bin    349448_2.bin   349448_6.bin   349449_0.bin
+349445_0.json   349445_2.json  349445_6.json  349446_0.json   349446_2.json  349446_6.json  349447_0.json   349447_2.json  349447_6.json  349448_0.json   349448_2.json  349448_6.json  349449_0.json
+349445_1.bin    349445_3.bin   349445_7.bin   349446_1.bin    349446_3.bin   349446_7.bin   349447_1.bin    349447_3.bin   349447_7.bin   349448_1.bin    349448_3.bin   349448_7.bin   349449_1.bin
+349445_1.json   349445_3.json  349445_7.json  349446_1.json   349446_3.json  349446_7.json  349447_1.json   349447_3.json  349447_7.json  349448_1.json   349448_3.json  349448_7.json  349449_1.json
+349445_10.bin   349445_4.bin   349445_8.bin   349446_10.bin   349446_4.bin   349446_8.bin   349447_10.bin   349447_4.bin   349447_8.bin   349448_10.bin   349448_4.bin   349448_8.bin
+349445_10.json  349445_4.json  349445_8.json  349446_10.json  349446_4.json  349446_8.json  349447_10.json  349447_4.json  349447_8.json  349448_10.json  349448_4.json  349448_8.json
+349445_11.bin   349445_5.bin   349445_9.bin   349446_11.bin   349446_5.bin   349446_9.bin   349447_11.bin   349447_5.bin   349447_9.bin   349448_11.bin   349448_5.bin   349448_9.bin
+349445_11.json  349445_5.json  349445_9.json  349446_11.json  349446_5.json  349446_9.json  349447_11.json  349447_5.json  349447_9.json  349448_11.json  349448_5.json  349448_9.json
 ```
 
 Teams should be able to take a directory of `$mode/$TEAM_NAME/blocks` and:
 - read a genesis state `genesis.json` for the `$mode`
-- read blocks/*.codec and decode all blocks successfully
+- read blocks/*.bin and decode all blocks successfully with the JAM Codec
 - validate each block, checking the stateroot and block hash at a minimum
 
-Note: For JSON, the intention is for JSON object attribute names to be as
-close as possible to w3f `codec` test vectors.  However, different
-teams are likely to choose slightly different JSON attribute names for
-some state outside the test vectors, so validation between the two
-teams should be based on the JAM codec files instead of the JSON
-content. The JSON should be used solely for supporting conversations
-and ease of debugging.
+Here is a sample run of a `validatetraces` that does the above:
+```
+# ./validatetraces ~/jam-duna/jamtestnet/traces/assurances/jam_duna
+VALIDATED Block 349462_0.bin => State 349462_0.bin | [N3] H_t=4193544 H_r=0910..b69d  EpochMarker(η1=151f..3ead)
+VALIDATED Block 349462_1.bin => State 349462_1.bin | [N3] H_t=4193545 H_r=27d0..5869   |E_T|=3
+VALIDATED Block 349462_2.bin => State 349462_2.bin | [N2] H_t=4193546 H_r=6413..540d   |E_T|=3
+VALIDATED Block 349462_3.bin => State 349462_3.bin | [N1] H_t=4193547 H_r=134b..f686   |E_T|=3  |E_G|=1
+VALIDATED Block 349462_4.bin => State 349462_4.bin | [N4] H_t=4193548 H_r=4329..66d0   |E_T|=3  |E_A|=5
+VALIDATED Block 349462_5.bin => State 349462_5.bin | [N2] H_t=4193549 H_r=5c46..e964   |E_T|=3
+VALIDATED Block 349462_6.bin => State 349462_6.bin | [N1] H_t=4193550 H_r=4aaa..94e2   |E_T|=3
+VALIDATED Block 349462_7.bin => State 349462_7.bin | [N1] H_t=4193551 H_r=9f96..ceb4 
+VALIDATED Block 349462_8.bin => State 349462_8.bin | [N1] H_t=4193552 H_r=3e82..7b05 
+VALIDATED Block 349462_9.bin => State 349462_9.bin | [N3] H_t=4193553 H_r=5838..d51c 
+VALIDATED Block 349462_10.bin => State 349462_10.bin | [N3] H_t=4193554 H_r=f300..38c6   WinningTickets(12)
+VALIDATED Block 349462_11.bin => State 349462_11.bin | [N5] H_t=4193555 H_r=111d..3ca5 
+Trace validation completed successfully.
+```
 
+Note that JAM codec versions are used, not JSON files.  JSON files are
+provided for debugging only.  In particular, JSON object attribute
+names to be as close as possible to w3f `codec` test vectors.
+However, different teams are likely to choose slightly different JSON
+attribute names for some state variables outside the test vectors, so
+validation between two teams should be based on the JAM codec files
+instead of the JSON content.
 
 ## Modes
 
 * For mode `safrole`, only the first 4 epochs are useful or needed following this [JAM Safrole Model](https://docs.google.com/spreadsheets/d/1ueAisCMOx7B-m_fXMLT0FXBxfVzydJyr-udE8jKwDN8/edit?gid=615049643#gid=615049643)
 
 * For mode `assurance`, `finality`, and `conformance` additional epochs may be required using sample work packages.
 
+* Additional modes can be added between based on JAM implementer community interest.
+
 ## Notes
 
 This is a PoC at this point and any team submission should not be taken as authoritative in any way.  
-
diff --git a/...ole/jam-duna/state_snapshots/341165_0.bin → traces/assurances/genesis.bin b/...ole/jam-duna/state_snapshots/341165_0.bin → traces/assurances/genesis.bin