diff --git a/docs/common/core/003-csca-list-proposal-guide.md b/docs/common/core/003-csca-list-proposal-guide.md index e607f232..f19789ea 100644 --- a/docs/common/core/003-csca-list-proposal-guide.md +++ b/docs/common/core/003-csca-list-proposal-guide.md @@ -17,7 +17,7 @@ When obtained the passport signer's root certificate, you can check its presence stored on Rarimo chain in a decentralized manner. This is how you ensure that the passport is signed by an eligible entity. -Technical details can be found in [CSCA List Rarimo Core module](https://github.com/rarimo/rarimo-core/x/cscalist/README.md). +Technical details can be found in [CSCA List Rarimo Core module](../../../x/cscalist/README.md). The keys of CSCA signers can be compromised or updated, ergo list updates must be handled. This is done via proposals. The guide will help you in creating and verifying proposals on list changes. @@ -38,10 +38,11 @@ You can call the CLI from that dir, or add it to your `$PATH`: export PATH=$PATH:$(go env GOPATH)/bin ``` - -Some commands require network RPC in order to fetch state. -You can put it into environment variable for convenient command calls: +All commands require Rarimo home directory set. +Also, some of them require network RPC in order to fetch state. +You can put these into environment variable for convenient command calls: ```bash +export RARIMO_HOME=/path/to/your/rarimo/home export RPC=https://your-rpc ``` @@ -59,7 +60,7 @@ This is why you have to monitor the list manually if you are interested in up-to ### Compare with state ```bash -rarimo-cored --node $RPC query cscalist ldif-tree-diff your-file.ldif +rarimo-cored --home $RARIMO_HOME --node $RPC query cscalist ldif-tree-diff your-file.ldif ``` If roots are different, the list was changed and the proposal must be created to update the tree. @@ -67,54 +68,53 @@ If roots are different, the list was changed and the proposal must be created to Most likely, you have already filled basic proposal data in the block scan (title, description, etc.). You should have the file with CSCA list, see **Download the list** subsection. Assuming the file name is `your-file.ldif`. -Obtain primary data for your proposal: +Next, obtain primary data for your proposal, which depends on the proposal type. + +### ReplaceCSCAListProposal + ```bash cd ~/Downloads # or wherever you have the file -rarimo-cored query cscalist parse-ldif --output-format hash your-file.ldif your-output-file.txt +rarimo-cored --home $RARIMO_HOME query cscalist parse-ldif your-file.ldif your-output-file.txt ``` `your-output-file.txt` will contain the big list of hashes of public keys from the LDIF file. Example: ``` -0x1d4dd579478a38c00f58a4d94263ff2bb0459992c073ebb7a6991194e44157f2 -0x0dcc4019fccc7ad4fbb535a40633cc32f99a18096a736b21e695e35e964209ae -0x01a3d79b678d79a8f912b693c4d57b38cf0e44ef413b7684e92a664e98c911ed +"0x1d4dd579478a38c00f58a4d94263ff2bb0459992c073ebb7a6991194e44157f2", +"0x0dcc4019fccc7ad4fbb535a40633cc32f99a18096a736b21e695e35e964209ae", +"0x01a3d79b678d79a8f912b693c4d57b38cf0e44ef413b7684e92a664e98c911ed" ``` -If this is `ReplaceCSCAListProposal`, just copy the hashes to the proposal `leaves` field. - -For `EditCSCAListProposal` the process is more complicated: +These are the hashes of compatible public keys from the provided LDIF file. +Just copy them to the `leaves` field of your draft proposal. -### Method 1 +### EditCSCAListProposal -1. Obtain the old Master List LDIF file, which tree is stored on-chain at the moment. -If you don't have it, use **Method 2** instead. -2. Optional: ensure that the list from file is the same as on-chain: ```bash -rarimo-cored --node $RPC query cscalist ldif-tree-diff your-old-file.ldif +cd ~/Downloads # or wherever you have the file +rarimo-cored --home $RARIMO_HOME query cscalist prepare-proposal your-file.ldif your-output-file.txt ``` -You expect to see message that the trees' roots are the same. -Otherwise, seek for an actual `your-old-file.ldif`. -3. Extract the hashes from your old file: -```bash -rarimo-cored query cscalist parse-ldif --output-format hash your-old-file.ldif old-output-file.txt + +`your-output-file.txt` will contain two lists of hashes to add and remove. Example: +``` +=== To add === +"0x1d4dd579478a38c00f58a4d94263ff2bb0459992c073ebb7a6991194e44157f2", +"0x0dcc4019fccc7ad4fbb535a40633cc32f99a18096a736b21e695e35e964209ae" +=== To remove === +"0x01a3d79b678d79a8f912b693c4d57b38cf0e44ef413b7684e92a664e98c911ed" ``` -4. Compare `your-output-file.txt` and `old-output-file.txt` to find the differences. -You can use `diff`, `vimdiff` or any preferable tool. -- hashes present only in `your-output-file.txt` are to be added -- hashes present only in `old-output-file.txt` are to be removed -5. Put acquired hashes into respective fields: `toAdd`, `toRemove` +The command has fetched the current state, compared it to the tree from file and provided the differences to consume. +Copy the respective hashes to the `toAdd` and `toRemove` fields of your draft proposal. -### Method 2 +### Submit proposal -1. Fetch the current hashes from the Merkle tree: +Basically, you need a command: ```bash -rarimo-cored --node $RPC query cscalist tree > current-tree.txt +rarimo-cored --home $RARIMO_HOME --node $RPC tx gov submit-proposal draft_proposal.json --from ``` -2. Iterate over the `current-tree.txt` manually and collect `key` values. -You may put them into `old-output-file.txt`. - -3. Perform steps 4 and 5 from **Method 1**. -Congratulations! You should now have your proposal ready for submission. +Refer to Cosmos SDK documentation and other guides to get more details about: +- Account creation +- Generating and managing your keys +- Obtaining `draft_proposal.json` ## Verify proposal @@ -122,16 +122,19 @@ In order to vote for the proposal, you need to ensure that its content correspon Firstly, download the list from [ICAO website](https://pkddownloadsg.icao.int/) and prepare the list of hashes, like described in **Create proposal** section. - Secondly, check whether the proposal makes any difference to the current state: ```bash -rarimo-cored --node $RPC query cscalist ldif-tree-diff your-file.ldif +rarimo-cored --home $RARIMO_HOME --node $RPC query cscalist ldif-tree-diff your-file.ldif ``` -If there is no difference, the proposal does not make sense, and you should vote 'No' or 'NoWithVeto'. +If there is no difference, the proposal either does not make sense, or is fraudulent, and you should vote 'No' or 'NoWithVeto'. +Again, the deeper validation depends on proposal type. If this is `ReplaceCSCAListProposal`, just compare the `leaves` field with `your-output-file.txt`. It is possible that you could have the different order of hashes due to ICAO change of order. Therefore, you may wish to sort both lists, e.g. with `sort` command. -For `EditCSCAListProposal`, you need to perform the same steps as in **Create proposal** section. -Then compare the `toAdd` and `toRemove` fields with the differences between `your-output-file.txt` and `old-output-file.txt`. +For `EditCSCAListProposal`, you should additionally perform: +```bash +rarimo-cored --home $RARIMO_HOME --node $RPC query cscalist prepare-proposal your-file.ldif your-output-file.txt +``` +Then compare the `toAdd` and `toRemove` fields with the differences between `your-output-file.txt` and someone's proposal. -If you get the same results as in the proposal, it is correct, and you can vote for it. +If you get the same results as in the proposal (of any type), it is correct, and you should support it. diff --git a/x/cscalist/client/cli/ldif.go b/x/cscalist/client/cli/ldif.go index 9dd27de6..1d90fedb 100644 --- a/x/cscalist/client/cli/ldif.go +++ b/x/cscalist/client/cli/ldif.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "os" + "strings" "github.com/cosmos/cosmos-sdk/client" "github.com/cosmos/cosmos-sdk/client/flags" @@ -21,12 +22,13 @@ const outputFormatFlag = "output-format" func cmdParseLDIF() *cobra.Command { const ( - rawFormat = "raw" - hashFormat = "hash" - rootFormat = "root" - defaultFormat = hashFormat + rawFormat = "raw" + hashFormat = "hash" + proposalFormat = "proposal" + rootFormat = "root" + defaultFormat = proposalFormat ) - outputFormatValues := fmt.Sprintf("%s, %s, %s", rawFormat, hashFormat, rootFormat) + outputFormatValues := strings.Join([]string{rawFormat, hashFormat, proposalFormat, rootFormat}, ", ") cmd := &cobra.Command{ Use: "parse-ldif [output-file]", @@ -80,12 +82,10 @@ automatically, see ldif-tree-diff)`, } case hashFormat: - for _, pk := range pubKeys { - hash := keccak256.Hash(pk) - if _, err = fmt.Fprintln(dst, hexutil.Encode(hash)); err != nil { - return fmt.Errorf("write to destination: %w", err) - } - } + return printHashes(pubKeys, dst, false) + + case proposalFormat: + return printHashes(pubKeys, dst, true) case rootFormat: sPubKeys := make([]string, len(pubKeys)) @@ -115,11 +115,10 @@ automatically, see ldif-tree-diff)`, func cmdLDIFTreeDiff() *cobra.Command { cmd := &cobra.Command{ Use: "ldif-tree-diff ", - Short: "Build one tree from LDIF file, query another tree from chain and compare roots", + Short: "Build one tree from LDIF file, query tree root from chain and compare roots", Long: `Provide a LDIF file to parse. The list can be manually retrieved from -https://pkddownloadsg.icao.int/. When the file is parsed, one tree is built, -another one is queried. Roots of 2 trees are compared. Use parse-ldif and tree -subcommands to find out the exact differences of data. +https://pkddownloadsg.icao.int/. Use parse-ldif and tree subcommands to find out +the exact differences of data. Use cases: - Automatically validate proposals: ensure that stored and proposed list differ @@ -132,34 +131,20 @@ Use cases: return fmt.Errorf("parse LDIF from file: %w", err) } - pubKeys, err := data.RawPubKeys() - if err != nil { - return fmt.Errorf("extract raw public keys: %w", err) - } - sPubKeys := make([]string, len(pubKeys)) - for i, pk := range pubKeys { - sPubKeys[i] = string(pk) - } - - tree, err := mt.BuildFromRaw(sPubKeys) - if err != nil { - return fmt.Errorf("build Merkle tree: %w", err) - } - cliCtx := client.GetClientContextFromCmd(cmd) cli := types.NewQueryClient(cliCtx) - root, err := getRootNode(cli) + + built, stored, err := getBuiltAndStoredRoots(data, cli) if err != nil { - return fmt.Errorf("get root node: %w", err) + return fmt.Errorf("get built and stored roots: %w", err) } - treeRoot := hexutil.Encode(tree.Root()) - if treeRoot != root.Hash { - fmt.Printf("Trees differ: built_root=%s stored_root=%s\n", treeRoot, root.Hash) + if built != stored { + fmt.Printf("Trees differ: built_root=%s stored_root=%s\n", built, stored) return nil } - fmt.Printf("Tree are same: root=%s\n", treeRoot) + fmt.Printf("Tree are same: root=%s\n", built) return nil }, } @@ -168,6 +153,51 @@ Use cases: return cmd } +func printHashes(pubKeys [][]byte, dst *os.File, forProposal bool) error { + for i, pk := range pubKeys { + hash := keccak256.Hash(pk) + + out := hexutil.Encode(hash) + if forProposal { + var comma string + if i < len(pubKeys)-1 { + comma = "," + } + out = fmt.Sprintf("%q%s", out, comma) + } + + if _, err := fmt.Fprintln(dst, out); err != nil { + return fmt.Errorf("write to destination: %w", err) + } + } + + return nil +} + +func getBuiltAndStoredRoots(data ldif.LDIF, cli types.QueryClient) (string, string, error) { + pubKeys, err := data.RawPubKeys() + if err != nil { + return "", "", fmt.Errorf("extract raw public keys: %w", err) + } + + sPubKeys := make([]string, len(pubKeys)) + for i, pk := range pubKeys { + sPubKeys[i] = string(pk) + } + + tree, err := mt.BuildFromRaw(sPubKeys) + if err != nil { + return "", "", fmt.Errorf("build Merkle tree: %w", err) + } + + root, err := getRootNode(cli) + if err != nil { + return "", "", fmt.Errorf("get root node: %w", err) + } + + return hexutil.Encode(tree.Root()), root.Hash, nil +} + func getRootNode(cli types.QueryClient) (*types.Node, error) { params, err := cli.Params(context.Background(), &types.QueryParamsRequest{}) if err != nil { diff --git a/x/cscalist/client/cli/proposal.go b/x/cscalist/client/cli/proposal.go new file mode 100644 index 00000000..acbf856a --- /dev/null +++ b/x/cscalist/client/cli/proposal.go @@ -0,0 +1,170 @@ +package cli + +import ( + "context" + "fmt" + "os" + "strings" + + "github.com/cosmos/cosmos-sdk/client" + "github.com/cosmos/cosmos-sdk/client/flags" + "github.com/cosmos/cosmos-sdk/types/query" + "github.com/ethereum/go-ethereum/common/hexutil" + "github.com/iden3/go-iden3-crypto/keccak256" + "github.com/rarimo/ldif-sdk/ldif" + "github.com/rarimo/rarimo-core/x/cscalist/types" + "github.com/spf13/cobra" + "google.golang.org/grpc" +) + +func cmdPrepareProposal() *cobra.Command { + cmd := &cobra.Command{ + Use: "prepare-proposal [output-file]", + Short: "Prepare content for EditCSCAListProposal from LDIF file", + Long: `Provide a LDIF file to parse CSCA certificates from it. The list can be manually +retrieved from https://pkddownloadsg.icao.int/. The command builds one tree from +LDIF, fetches another one from chain and selects leaves to add or remove. + +If output-file is provided, the output is written to the specified file instead +of stdout. This is useful to copy-paste from a file. + +Use cases: +- Copy-paste the output to your EditCSCAListProposal draft +- Validate editing proposals: ensure that the leaves to add and remove are the same`, + SuggestionsMinimumDistance: 2, + Args: cobra.RangeArgs(1, 2), + RunE: func(cmd *cobra.Command, args []string) (err error) { + dst := os.Stdout + if len(args) == 2 { + dst, err = os.OpenFile(args[1], os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0644) + if err != nil { + return fmt.Errorf("open destination file: %w", err) + } + defer func() { _ = dst.Close() }() + } + + data, err := ldif.FromFile(args[0]) + if err != nil { + return fmt.Errorf("parse LDIF from file: %w", err) + } + + cliCtx := client.GetClientContextFromCmd(cmd) + cli := types.NewQueryClient(cliCtx) + + built, stored, err := getBuiltAndStoredRoots(data, cli) + if err != nil { + return fmt.Errorf("get built and stored roots: %w", err) + } + + if built == stored { + _, err = fmt.Fprintf(dst, "Tree are same: root=%s. Nothing to add or remove.\n", built) + if err != nil { + return fmt.Errorf("write to destination: %w", err) + } + return nil + } + + oldLeaves, err := fetchHashLeavesFromCosmos(cmd.Context(), cli) + if err != nil { + return fmt.Errorf("fetch hash leaves from cosmos: %w", err) + } + + pubKeys, _ := data.RawPubKeys() // error was handled in getBuiltAndStoredRoots + newLeaves := make([]string, len(pubKeys)) + for i, pk := range pubKeys { + newLeaves[i] = hexutil.Encode(keccak256.Hash(pk)) + } + + toAdd, toRemove := selectProposalHashes(oldLeaves, newLeaves) + + _, err = fmt.Fprintf(dst, "=== To add ===\n%s=== To remove ===\n%s", formatHashes(toAdd), formatHashes(toRemove)) + if err != nil { + return fmt.Errorf("write to destination: %w", err) + } + return nil + }, + } + + flags.AddQueryFlagsToCmd(cmd) + return cmd +} + +// optimized copy of github.com/rarimo/ldif-sdk/utils.FetchHashLeavesFromCosmos +func fetchHashLeavesFromCosmos(ctx context.Context, client types.QueryClient) ([]string, error) { + var ( + limit = uint64(100) + offset = uint64(0) + leaves []string + ) + + for { + resp, err := client.Tree(ctx, &types.QueryTreeRequest{ + Pagination: &query.PageRequest{ + CountTotal: true, + Limit: limit, + Offset: offset, + }, + }, grpc.EmptyCallOption{}) + if err != nil { + return nil, fmt.Errorf("query paginated tree: %w", err) + } + + if cap(leaves) == 0 { + leaves = make([]string, 0, resp.Pagination.Total) + } + + for _, node := range resp.Tree { + leaves = append(leaves, node.Key) + } + + if uint64(len(leaves)) == resp.Pagination.Total { + break + } + + offset += limit + } + + return leaves, nil +} + +func selectProposalHashes(oldLeaves, newLeaves []string) (toAdd []string, toRemove []string) { + toAdd = make([]string, 0, len(newLeaves)) + toRemove = make([]string, 0, len(oldLeaves)) + + for _, leaf := range newLeaves { + if !existsInSlice(oldLeaves, leaf) { + toAdd = append(toAdd, leaf) + } + } + + for _, leaf := range oldLeaves { + if !existsInSlice(newLeaves, leaf) { + toRemove = append(toRemove, leaf) + } + } + + return +} + +func formatHashes(hashes []string) string { + var sb strings.Builder + + for i, h := range hashes { + var comma string + if i < len(hashes)-1 { + comma = "," + } + sb.WriteString(fmt.Sprintf("%q%s\n", h, comma)) + } + + return sb.String() +} + +func existsInSlice(slice []string, elem string) bool { + for _, e := range slice { + if e == elem { + return true + } + } + return false +} diff --git a/x/cscalist/client/cli/query.go b/x/cscalist/client/cli/query.go index e8731757..c2ce6849 100644 --- a/x/cscalist/client/cli/query.go +++ b/x/cscalist/client/cli/query.go @@ -36,6 +36,7 @@ func GetQueryCmd() *cobra.Command { cmdQueryTree(), cmdParseLDIF(), cmdLDIFTreeDiff(), + cmdPrepareProposal(), ) // this line is used by starport scaffolding # 1