Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Commit

Permalink
Overhaul leader CLI such that flags are grouped by subcommand
Browse files Browse the repository at this point in the history
  • Loading branch information
cpubot committed Nov 23, 2023
1 parent 04a39c6 commit 8f20062
Show file tree
Hide file tree
Showing 4 changed files with 182 additions and 110 deletions.
175 changes: 118 additions & 57 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,19 @@ A composition of [`paladin`](https://github.com/0xPolygonZero/paladin) and [`plo
- [Ops](#ops)
- [Worker](#worker)
- [Leader](#leader)
- [Usage](#usage)
- [RPC](#rpc)
- [Verifier](#verifier)
- [Leader Usage](#leader-usage)
- [stdio](#stdio)
- [Jerigon](#jerigon)
- [HTTP](#http)
- [Paladin Runtime](#paladin-runtime)
- [Starting an AMQP enabled cluster](#starting-an-amqp-enabled-cluster)
- [Start worker(s)](#start-workers)
- [Start leader](#start-leader)
- [Starting an in-memory (single process) cluster](#starting-an-in-memory-single-process-cluster)
- [Input mode](#input-mode)
- [stdin](#stdin)
- [HTTP](#http)
- [Jerigon](#jerigon)
- [Verifier](#verifier)
- [RPC](#rpc)
- [Verifier Usage](#verifier-usage)
- [RPC Usage](#rpc-usage)
- [Docker](#docker)


Expand All @@ -34,6 +35,14 @@ worker
└── main.rs
leader
├── Cargo.toml
└── src
└── main.rs
rpc
├── Cargo.toml
└── src
└── main.rs
verifier
├── Cargo.toml
└── src
└── main.rs
```
Expand All @@ -46,98 +55,150 @@ The worker process. Receives proof operations from the leader, and returns the r
### Leader
The leader process. Receives proof generation requests, and distributes them to workers.

## Usage
### RPC

A binary to generate the block trace format expected by the leader.

### Verifier

Leader binary arguments and options are comprised of paladin configuration and input mode:
A binary to verify the correctness of the generated proof.

## Leader Usage

The leader has various subcommands for different io modes. The leader binary arguments are as follows:
```
cargo r --bin leader -- --help
cargo r --release --bin leader -- --help
Usage: leader [OPTIONS]
Usage: leader [OPTIONS] <COMMAND>
Commands:
stdio Reads input from stdin and writes output to stdout
jerigon Reads input from a Jerigon node and writes output to stdout
http Reads input from HTTP and writes output to a directory
help Print this message or the help of the given subcommand(s)
Options:
-m, --mode <MODE>
The input mode. If `std-io`, the input is read from stdin. If `http`, the input is read from HTTP requests. If `jerigon`, the input is read from the `debug_traceBlockByNumber` and `eth_getBlockByNumber` RPC methods from Jerigon [default: std-io] [possible values: std-io, http, jerigon]
-p, --port <PORT>
The port to listen on when using the `http` mode [default: 8080]
-o, --output-dir <OUTPUT_DIR>
The directory to which output should be written (`http` mode only)
--rpc-url <RPC_URL>
The RPC URL to use when using the `jerigon` mode
-b, --block-number <BLOCK_NUMBER>
The block number to use when using the `jerigon` mode
-r, --runtime <RUNTIME>
Specifies the paladin runtime to use [default: amqp] [possible values: amqp, in-memory]
-n, --num-workers <NUM_WORKERS>
Specifies the number of worker threads to spawn (in memory runtime only)
-p, --previous-proof <PREVIOUS_PROOF>
The previous proof output
-h, --help
Print help
-n, --num-workers <NUM_WORKERS> Specifies the number of worker threads to spawn (in memory runtime only)
-r, --runtime <RUNTIME> Specifies the paladin runtime mode [default: amqp] [possible values: amqp, in-memory]
--amqp-uri <AMQP_URI> Specifies the URI for the AMQP broker (AMQP runtime only)
-h, --help Print help
```

### Paladin Runtime
### stdio
The stdio command reads proof input from stdin and writes output to stdout.

Paladin supports both an AMQP and in-memory runtime. The in-memory runtime will emulate a cluster in memory within a single process, and is useful for testing. The AMQP runtime is geared for a production environment. The AMQP runtime requires a running AMQP broker and spinning up worker processes. The AMQP uri can be specified with the `--amqp-uri` flag or be set with the `AMQP_URI` environment variable.
```
cargo r --release --bin leader stdio --help
#### Starting an AMQP enabled cluster
Reads input from stdin and writes output to stdout
##### Start worker(s)
Usage: leader stdio [OPTIONS]
Start worker process(es). The default paladin runtime is AMQP, so no additional flags are required to enable it.
Options:
-f, --previous-proof <PREVIOUS_PROOF> The previous proof output
-h, --help Print help
```

Pull prover input from the rpc binary.
```bash
RUST_LOG=debug cargo r --release --bin worker
cargo r --release --bin rpc fetch --rpc-url http://35.246.1.96:8545 -b 6 > ./input/block_6.json
```

##### Start leader
Pipe the block input to the leader binary.
```bash
cat ./input/block_6.json | cargo r --release --bin leader -- -r in-memory stdio > ./output/proof_6.json
```

Start the leader process with the desired [input mode](#input-mode). The default paladin runtime is AMQP, so no additional flags are required to enable it.
### Jerigon

The Jerigon command reads proof input from a Jerigon node and writes output to stdout.

```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode http --output-dir ./output
```
cargo r --release --bin leader jerigon --help
#### Starting an in-memory (single process) cluster
Reads input from a Jerigon node and writes output to stdout
Paladin can emulate a cluster in memory within a single process. Useful for testing purposes.
Usage: leader jerigon [OPTIONS] --rpc-url <RPC_URL> --block-number <BLOCK_NUMBER>
Options:
-u, --rpc-url <RPC_URL>
-b, --block-number <BLOCK_NUMBER> The block number for which to generate a proof
-f, --previous-proof <PREVIOUS_PROOF> The previous proof output
-h, --help Print help
```

Prove a block.
```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode http --runtime in-memory --output-dir ./output
cargo r --release --bin leader -- -r in-memory jerigon -u <RPC_URL> -b 16 > ./output/proof_16.json
```

### HTTP

The HTTP command reads proof input from HTTP and writes output to a directory.
```
cargo r --release --bin leader http --help
### Input mode
Pass JSON encoded prover input to stdin or over HTTP, or point the leader to a Jerigon RPC endpoint to retrieve the prover input from the `debug_traceBlockByNumber` and `eth_getBlockByNumber` RPC methods.
Reads input from HTTP and writes output to a directory
See [`prover_input.rs`](/leader/src/prover_input.rs) for the input format.
Usage: leader http [OPTIONS] --output-dir <OUTPUT_DIR>
The `std-io` and `http` examples below assume some prover input is stored in `./block_121.json`.
Options:
-p, --port <PORT> The port on which to listen [default: 8080]
-o, --output-dir <OUTPUT_DIR> The directory to which output should be written
-h, --help Print help
```

#### stdin
Pull prover input from the rpc binary.
```bash
cargo r --release --bin rpc fetch --rpc-url http://35.246.1.96:8545 -b 6 > ./input/block_6.json
```

Start the server.
```bash
cat ./block_121.json | RUST_LOG=debug cargo r --release --bin leader > ./output/proof_121.json
RUST_LOG=debug cargo r --release --bin leader http --output-dir ./output
```

#### HTTP
Note that HTTP mode requires a [slightly modified input format](./leader/src/http.rs#L56) from the rest of the commands. In particular, [the previous proof is expected to be part of the payload](./leader/src/http.rs#L58). This is due to the fact that the HTTP mode may handle multiple requests concurrently, and thus the previous proof cannot reasonably be given by a command line argument like the other modes.


Using `jq` we can merge the previous proof and the block input into a single JSON object.

Start the server
```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode http --output-dir ./output
jq -s '{prover_input: .[0], previous: .[1]}' ./input/block_6.json ./output/proof_5.json | curl -X POST -H "Content-Type: application/json" -d @- http://localhost:8080/prove

```

Once initialized, send a request:
### Paladin Runtime

Paladin supports both an AMQP and in-memory runtime. The in-memory runtime will emulate a cluster in memory within a single process, and is useful for testing. The AMQP runtime is geared for a production environment. The AMQP runtime requires a running AMQP broker and spinning up worker processes. The AMQP uri can be specified with the `--amqp-uri` flag or be set with the `AMQP_URI` environment variable.

#### Starting an AMQP enabled cluster

##### Start worker(s)

Start worker process(es). The default paladin runtime is AMQP, so no additional flags are required to enable it.

```bash
curl -X POST -H "Content-Type: application/json" -d @./block_121.json http://localhost:8080/prove
RUST_LOG=debug cargo r --release --bin worker
```

#### Jerigon
##### Start leader

Start the leader process with the desired [command](#leader-usage). The default paladin runtime is AMQP, so no additional flags are required to enable it.

```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode jerigon --runtime in-memory --rpc-url <RPC_URL> --block-number 16 > ./output/proof_16.json
RUST_LOG=debug cargo r --release --bin leader jerigon -u <RPC_URL> -b 16 > ./output/proof_16.json
```

#### Starting an in-memory (single process) cluster

Paladin can emulate a cluster in memory within a single process. Useful for testing purposes.

```bash
cat ./input/block_6.json | cargo r --release --bin leader -- -r in-memory stdio > ./output/proof_6.json
```

## Verifier
## Verifier Usage

A verifier binary is provided to verify the correctness of the generated proof. The verifier expects output in the format generated by the leader. The verifier binary arguments are as follows:
```
Expand All @@ -155,7 +216,7 @@ Example:
cargo r --release --bin verifier -- -f ./output/proof_16.json
```

## RPC
## RPC Usage

An rpc binary is provided to generate the block trace format expected by the leader.
```
Expand Down
82 changes: 48 additions & 34 deletions leader/src/cli.rs
Original file line number Diff line number Diff line change
@@ -1,44 +1,58 @@
use std::path::PathBuf;

use clap::{Parser, ValueEnum, ValueHint};
use clap::{Args, Parser, Subcommand, ValueHint};
use paladin::config::Runtime;

#[derive(Parser)]
pub(crate) struct Cli {
/// The input mode. If `std-io`, the input is read from stdin. If `http`,
/// the input is read from HTTP requests. If `jerigon`, the input is
/// read from the `debug_traceBlockByNumber` and `eth_getBlockByNumber`
/// RPC methods from Jerigon.
#[arg(short, long, value_enum, default_value_t = Mode::StdIo)]
pub(crate) mode: Mode,
/// The port to listen on when using the `http` mode.
#[arg(short, long, default_value_t = 8080)]
pub(crate) port: u16,
/// The directory to which output should be written (`http` mode only).
#[arg(short, long, required_if_eq("mode", "http"), value_hint = ValueHint::DirPath)]
pub(crate) output_dir: Option<PathBuf>,
/// The RPC URL to use when using the `jerigon` mode.
#[arg(long, required_if_eq("mode", "jerigon"), value_hint = ValueHint::Url)]
pub(crate) rpc_url: Option<String>,
/// The block number to use when using the `jerigon` mode.
#[arg(short, long, required_if_eq("mode", "jerigon"))]
pub(crate) block_number: Option<u64>,
/// Specifies the paladin runtime to use.
#[arg(long, short, value_enum, default_value_t = Runtime::Amqp)]
pub(crate) runtime: Runtime,
#[command(subcommand)]
pub(crate) command: Command,

#[clap(flatten)]
pub(crate) runtime: RuntimeGroup,
}

#[derive(Subcommand)]
pub(crate) enum Command {
/// Reads input from stdin and writes output to stdout.
Stdio {
/// The previous proof output.
#[arg(long, short = 'f', value_hint = ValueHint::FilePath)]
previous_proof: Option<PathBuf>,
},
/// Reads input from a Jerigon node and writes output to stdout.
Jerigon {
// The Jerigon RPC URL.
#[arg(long, short = 'u', value_hint = ValueHint::Url)]
rpc_url: String,
/// The block number for which to generate a proof.
#[arg(short, long)]
block_number: u64,
/// The previous proof output.
#[arg(long, short = 'f', value_hint = ValueHint::FilePath)]
previous_proof: Option<PathBuf>,
},
/// Reads input from HTTP and writes output to a directory.
Http {
/// The port on which to listen.
#[arg(short, long, default_value_t = 8080)]
port: u16,
/// The directory to which output should be written.
#[arg(short, long, value_hint = ValueHint::DirPath)]
output_dir: PathBuf,
},
}

#[derive(Args)]
pub(crate) struct RuntimeGroup {
/// Specifies the number of worker threads to spawn (in memory runtime
/// only).
#[arg(long, short)]
pub num_workers: Option<usize>,
/// The previous proof output.
#[arg(long, short = 'f')]
pub previous_proof: Option<PathBuf>,
}

#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Debug, ValueEnum, Default)]
pub(crate) enum Mode {
#[default]
StdIo,
Http,
Jerigon,
pub(crate) num_workers: Option<usize>,
/// Specifies the paladin runtime mode.
#[arg(long, short, value_enum, default_value_t = Runtime::Amqp)]
pub(crate) runtime: Runtime,
/// Specifies the URI for the AMQP broker (AMQP runtime only).
#[arg(long, env = "AMQP_URI", value_hint = ValueHint::Url, required_if_eq("runtime", "amqp"))]
pub amqp_uri: Option<String>,
}
Loading

0 comments on commit 8f20062

Please sign in to comment.