Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Overhaul leader CLI such that flags are grouped by subcommand #2

Merged
merged 1 commit into from
Nov 23, 2023
Merged

Overhaul leader CLI such that flags are grouped by subcommand #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 118 additions & 57 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,19 @@ A composition of [`paladin`](https://github.com/0xPolygonZero/paladin) and [`plo
- [Ops](#ops)
- [Worker](#worker)
- [Leader](#leader)
- [Usage](#usage)
- [RPC](#rpc)
- [Verifier](#verifier)
- [Leader Usage](#leader-usage)
- [stdio](#stdio)
- [Jerigon](#jerigon)
- [HTTP](#http)
- [Paladin Runtime](#paladin-runtime)
- [Starting an AMQP enabled cluster](#starting-an-amqp-enabled-cluster)
- [Start worker(s)](#start-workers)
- [Start leader](#start-leader)
- [Starting an in-memory (single process) cluster](#starting-an-in-memory-single-process-cluster)
- [Input mode](#input-mode)
- [stdin](#stdin)
- [HTTP](#http)
- [Jerigon](#jerigon)
- [Verifier](#verifier)
- [RPC](#rpc)
- [Verifier Usage](#verifier-usage)
- [RPC Usage](#rpc-usage)
- [Docker](#docker)


Expand All @@ -34,6 +35,14 @@ worker
└── main.rs
leader
├── Cargo.toml
└── src
└── main.rs
rpc
├── Cargo.toml
└── src
└── main.rs
verifier
├── Cargo.toml
└── src
└── main.rs
```
Expand All @@ -46,98 +55,150 @@ The worker process. Receives proof operations from the leader, and returns the r
### Leader
The leader process. Receives proof generation requests, and distributes them to workers.

## Usage
### RPC

A binary to generate the block trace format expected by the leader.

### Verifier

Leader binary arguments and options are comprised of paladin configuration and input mode:
A binary to verify the correctness of the generated proof.

## Leader Usage

The leader has various subcommands for different io modes. The leader binary arguments are as follows:
```
cargo r --bin leader -- --help
cargo r --release --bin leader -- --help

Usage: leader [OPTIONS]
Usage: leader [OPTIONS] <COMMAND>

Commands:
stdio Reads input from stdin and writes output to stdout
jerigon Reads input from a Jerigon node and writes output to stdout
http Reads input from HTTP and writes output to a directory
help Print this message or the help of the given subcommand(s)

Options:
-m, --mode <MODE>
The input mode. If `std-io`, the input is read from stdin. If `http`, the input is read from HTTP requests. If `jerigon`, the input is read from the `debug_traceBlockByNumber` and `eth_getBlockByNumber` RPC methods from Jerigon [default: std-io] [possible values: std-io, http, jerigon]
-p, --port <PORT>
The port to listen on when using the `http` mode [default: 8080]
-o, --output-dir <OUTPUT_DIR>
The directory to which output should be written (`http` mode only)
--rpc-url <RPC_URL>
The RPC URL to use when using the `jerigon` mode
-b, --block-number <BLOCK_NUMBER>
The block number to use when using the `jerigon` mode
-r, --runtime <RUNTIME>
Specifies the paladin runtime to use [default: amqp] [possible values: amqp, in-memory]
-n, --num-workers <NUM_WORKERS>
Specifies the number of worker threads to spawn (in memory runtime only)
-p, --previous-proof <PREVIOUS_PROOF>
The previous proof output
-h, --help
Print help
-n, --num-workers <NUM_WORKERS> Specifies the number of worker threads to spawn (in memory runtime only)
-r, --runtime <RUNTIME> Specifies the paladin runtime mode [default: amqp] [possible values: amqp, in-memory]
--amqp-uri <AMQP_URI> Specifies the URI for the AMQP broker (AMQP runtime only)
-h, --help Print help
```

### Paladin Runtime
### stdio
The stdio command reads proof input from stdin and writes output to stdout.

Paladin supports both an AMQP and in-memory runtime. The in-memory runtime will emulate a cluster in memory within a single process, and is useful for testing. The AMQP runtime is geared for a production environment. The AMQP runtime requires a running AMQP broker and spinning up worker processes. The AMQP uri can be specified with the `--amqp-uri` flag or be set with the `AMQP_URI` environment variable.
```
cargo r --release --bin leader stdio --help

#### Starting an AMQP enabled cluster
Reads input from stdin and writes output to stdout

##### Start worker(s)
Usage: leader stdio [OPTIONS]

Start worker process(es). The default paladin runtime is AMQP, so no additional flags are required to enable it.
Options:
-f, --previous-proof <PREVIOUS_PROOF> The previous proof output
-h, --help Print help
```

Pull prover input from the rpc binary.
```bash
RUST_LOG=debug cargo r --release --bin worker
cargo r --release --bin rpc fetch --rpc-url <RPC_URL> -b 6 > ./input/block_6.json
```

##### Start leader
Pipe the block input to the leader binary.
```bash
cat ./input/block_6.json | cargo r --release --bin leader -- -r in-memory stdio > ./output/proof_6.json
```

Start the leader process with the desired [input mode](#input-mode). The default paladin runtime is AMQP, so no additional flags are required to enable it.
### Jerigon

The Jerigon command reads proof input from a Jerigon node and writes output to stdout.

```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode http --output-dir ./output
```
cargo r --release --bin leader jerigon --help

#### Starting an in-memory (single process) cluster
Reads input from a Jerigon node and writes output to stdout

Paladin can emulate a cluster in memory within a single process. Useful for testing purposes.
Usage: leader jerigon [OPTIONS] --rpc-url <RPC_URL> --block-number <BLOCK_NUMBER>

Options:
-u, --rpc-url <RPC_URL>
-b, --block-number <BLOCK_NUMBER> The block number for which to generate a proof
-f, --previous-proof <PREVIOUS_PROOF> The previous proof output
-h, --help Print help
```

Prove a block.
```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode http --runtime in-memory --output-dir ./output
cargo r --release --bin leader -- -r in-memory jerigon -u <RPC_URL> -b 16 > ./output/proof_16.json
```

### HTTP

The HTTP command reads proof input from HTTP and writes output to a directory.
```
cargo r --release --bin leader http --help

### Input mode
Pass JSON encoded prover input to stdin or over HTTP, or point the leader to a Jerigon RPC endpoint to retrieve the prover input from the `debug_traceBlockByNumber` and `eth_getBlockByNumber` RPC methods.
Reads input from HTTP and writes output to a directory

See [`prover_input.rs`](/leader/src/prover_input.rs) for the input format.
Usage: leader http [OPTIONS] --output-dir <OUTPUT_DIR>

The `std-io` and `http` examples below assume some prover input is stored in `./block_121.json`.
Options:
-p, --port <PORT> The port on which to listen [default: 8080]
-o, --output-dir <OUTPUT_DIR> The directory to which output should be written
-h, --help Print help
```

#### stdin
Pull prover input from the rpc binary.
```bash
cargo r --release --bin rpc fetch -u <RPC_URL> -b 6 > ./input/block_6.json
```

Start the server.
```bash
cat ./block_121.json | RUST_LOG=debug cargo r --release --bin leader > ./output/proof_121.json
RUST_LOG=debug cargo r --release --bin leader http --output-dir ./output
```

#### HTTP
Note that HTTP mode requires a [slightly modified input format](./leader/src/http.rs#L56) from the rest of the commands. In particular, [the previous proof is expected to be part of the payload](./leader/src/http.rs#L58). This is due to the fact that the HTTP mode may handle multiple requests concurrently, and thus the previous proof cannot reasonably be given by a command line argument like the other modes.


Using `jq` we can merge the previous proof and the block input into a single JSON object.

Start the server
```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode http --output-dir ./output
jq -s '{prover_input: .[0], previous: .[1]}' ./input/block_6.json ./output/proof_5.json | curl -X POST -H "Content-Type: application/json" -d @- http://localhost:8080/prove

```

Once initialized, send a request:
### Paladin Runtime

Paladin supports both an AMQP and in-memory runtime. The in-memory runtime will emulate a cluster in memory within a single process, and is useful for testing. The AMQP runtime is geared for a production environment. The AMQP runtime requires a running AMQP broker and spinning up worker processes. The AMQP uri can be specified with the `--amqp-uri` flag or be set with the `AMQP_URI` environment variable.

#### Starting an AMQP enabled cluster

##### Start worker(s)

Start worker process(es). The default paladin runtime is AMQP, so no additional flags are required to enable it.

```bash
curl -X POST -H "Content-Type: application/json" -d @./block_121.json http://localhost:8080/prove
RUST_LOG=debug cargo r --release --bin worker
```

#### Jerigon
##### Start leader

Start the leader process with the desired [command](#leader-usage). The default paladin runtime is AMQP, so no additional flags are required to enable it.

```bash
RUST_LOG=debug cargo r --release --bin leader -- --mode jerigon --runtime in-memory --rpc-url <RPC_URL> --block-number 16 > ./output/proof_16.json
RUST_LOG=debug cargo r --release --bin leader jerigon -u <RPC_URL> -b 16 > ./output/proof_16.json
```

#### Starting an in-memory (single process) cluster

Paladin can emulate a cluster in memory within a single process. Useful for testing purposes.

```bash
cat ./input/block_6.json | cargo r --release --bin leader -- -r in-memory stdio > ./output/proof_6.json
```

## Verifier
## Verifier Usage

A verifier binary is provided to verify the correctness of the generated proof. The verifier expects output in the format generated by the leader. The verifier binary arguments are as follows:
```
Expand All @@ -155,7 +216,7 @@ Example:
cargo r --release --bin verifier -- -f ./output/proof_16.json
```

## RPC
## RPC Usage

An rpc binary is provided to generate the block trace format expected by the leader.
```
Expand Down
82 changes: 48 additions & 34 deletions leader/src/cli.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,44 +1,58 @@
use std::path::PathBuf;

use clap::{Parser, ValueEnum, ValueHint};
use clap::{Args, Parser, Subcommand, ValueHint};
use paladin::config::Runtime;

#[derive(Parser)]
pub(crate) struct Cli {
/// The input mode. If `std-io`, the input is read from stdin. If `http`,
/// the input is read from HTTP requests. If `jerigon`, the input is
/// read from the `debug_traceBlockByNumber` and `eth_getBlockByNumber`
/// RPC methods from Jerigon.
#[arg(short, long, value_enum, default_value_t = Mode::StdIo)]
pub(crate) mode: Mode,
/// The port to listen on when using the `http` mode.
#[arg(short, long, default_value_t = 8080)]
pub(crate) port: u16,
/// The directory to which output should be written (`http` mode only).
#[arg(short, long, required_if_eq("mode", "http"), value_hint = ValueHint::DirPath)]
pub(crate) output_dir: Option<PathBuf>,
/// The RPC URL to use when using the `jerigon` mode.
#[arg(long, required_if_eq("mode", "jerigon"), value_hint = ValueHint::Url)]
pub(crate) rpc_url: Option<String>,
/// The block number to use when using the `jerigon` mode.
#[arg(short, long, required_if_eq("mode", "jerigon"))]
pub(crate) block_number: Option<u64>,
/// Specifies the paladin runtime to use.
#[arg(long, short, value_enum, default_value_t = Runtime::Amqp)]
pub(crate) runtime: Runtime,
#[command(subcommand)]
pub(crate) command: Command,

#[clap(flatten)]
pub(crate) runtime: RuntimeGroup,
}

#[derive(Subcommand)]
pub(crate) enum Command {
/// Reads input from stdin and writes output to stdout.
Stdio {
/// The previous proof output.
#[arg(long, short = 'f', value_hint = ValueHint::FilePath)]
previous_proof: Option<PathBuf>,
},
/// Reads input from a Jerigon node and writes output to stdout.
Jerigon {
// The Jerigon RPC URL.
#[arg(long, short = 'u', value_hint = ValueHint::Url)]
rpc_url: String,
/// The block number for which to generate a proof.
#[arg(short, long)]
block_number: u64,
/// The previous proof output.
#[arg(long, short = 'f', value_hint = ValueHint::FilePath)]
previous_proof: Option<PathBuf>,
},
/// Reads input from HTTP and writes output to a directory.
Http {
/// The port on which to listen.
#[arg(short, long, default_value_t = 8080)]
port: u16,
/// The directory to which output should be written.
#[arg(short, long, value_hint = ValueHint::DirPath)]
output_dir: PathBuf,
},
}

#[derive(Args)]
pub(crate) struct RuntimeGroup {
/// Specifies the number of worker threads to spawn (in memory runtime
/// only).
#[arg(long, short)]
pub num_workers: Option<usize>,
/// The previous proof output.
#[arg(long, short = 'f')]
pub previous_proof: Option<PathBuf>,
}

#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Debug, ValueEnum, Default)]
pub(crate) enum Mode {
#[default]
StdIo,
Http,
Jerigon,
pub(crate) num_workers: Option<usize>,
/// Specifies the paladin runtime mode.
#[arg(long, short, value_enum, default_value_t = Runtime::Amqp)]
pub(crate) runtime: Runtime,
/// Specifies the URI for the AMQP broker (AMQP runtime only).
#[arg(long, env = "AMQP_URI", value_hint = ValueHint::Url, required_if_eq("runtime", "amqp"))]
pub amqp_uri: Option<String>,
}
Loading
Loading