vldb24

Benchmark scripts

This directory contains the benchmark scripts used in our SIGMOD'24 and PaPoC'24 submissions. For our protocol implementations, refer to the top-level README.

Each *_benchmark.py script defines a single benchmark (which may include hyperparameter searches and multiple trials). The plotting code and results are in plots/.

The benchmarks automatically provision cloud machines as necessary, using the Hydro CLI. When running these benchmarks, use this branch of Hydroflow.

To run a specific benchmark, run the corresponding *_run.sh file. Note that the path to the jar file in these scripts may need to be adjusted if you are running locally without the NFS. Each script also expects a JSON cluster configuration with information about the cloud provider etc. Here is an example for Dedalus MultiPaxos (<worker image> is the name of a GCP disk image that has access to the NFS as well as Java 8):

{
    "mode": "hydro",
    "env": {
        "cloud": "gcp",
        "project":  <GCP project name>,
        "machine_type": "n2-standard-4",
        "scala_image": <worker image>,
        "hydroflow_image": <worker image>,
        "zone": "us-west1-b",
        "vpc": "eval",
        "user": <worker image username>
    },
    "services": {
        "clients": {
            "type": "scala"
        },
        "leaders": {
            "type": "hydroflow",
            "rust_example": "multipaxos"
        },
        "acceptors": {
            "type": "hydroflow",
            "rust_example": "multipaxos"
        },
        "replicas": {
            "type": "scala"
        }
    }
}

How the benchmarks work

Our benchmarks are derived from Michael Whittaker's, which are described here in quite some detail. Roughly, the following files are involved in each benchmark, using the Paxos benchmarks as an example:

• Implementation: Scala or rust
• Drivers
• Benchmark setup
• Benchmark parameters
• Run script

Implementation

This is the code that represents the "protocol", processing client input and returning some outputs. Since the Scala implementations were written by Michael, we will mainly discuss the rust implementations.

Each protocol has its own directory in rust/examples, which always includes main.rs. Configuration parameters can be provided at the top of each file and incorporated into Dedalus by unwrapping and embedding as Dedalus inputs. The component responsible with communicating with the Scala client must specify custom serialization and deserialization functions, based on the protobuf files, which are used by the Dedalus code. Note that any modification to the protobuf requires recompiling Scala and copying the .jar to the correct location.

Communication channels between components must be explicit. All channels are one-way. Send channels must follow this format and receive channels must follow this format. Note that pthe ports for each channel](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L90) have the format send_to$dest$num or receive_from$src$num, where num is unique per source and destination.

Drivers

This determines what data will be generated by the client, triggering client functions, along with some logging and prometheus setup. Only ClientMain.scala is relevant to us.

Benchmark setup

This defines the global config parameters, which can be reused across components. IP addresses are assigned to components, which can be modified to allow colocation. Configuration parameters are fed into each rust executable and ports are linked between components.

Benchmark parameters

These are the parameters that are changed between each experimental run. This determines the number of machines to be allocated to each component, of which some can participate in each test.

Run script

This is the shell script that executes the benchmark.