Skip to content

Commit

Permalink
Introduce smart_bench_to_csv.sh script, README fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
karolk91 committed Apr 30, 2024
1 parent 73c987b commit 20b3d15
Show file tree
Hide file tree
Showing 3 changed files with 147 additions and 13 deletions.
43 changes: 33 additions & 10 deletions launch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ This directory contains docker related scripts and configurations to compile and

The environment is created as Docker Image (by default named and tagged as `smart-bench:latest`) which leverages multi-stage Docker builds. Helper scripts available in this directory streamline operations to build ([build.sh](./build.sh)) and run ([run.sh](./run.sh)) of the image.

At the first stage of docker build, based upon `docker.io/paritytech/ci-linux:production` as rust enabled compilation env, smart-bench software is being compiled from current sources (this git repository). Resulting binary is copied into second stage / final docker image based on `ubuntu:20.04` image where also other dependencies are configured and installed.
At the first stage of docker build, based upon `docker.io/paritytech/ci-unified:bullseye-1.74.0` as rust enabled compilation env, smart-bench software is being compiled from current sources (this git repository). Resulting binary is copied into second stage / final docker image based on `ubuntu:20.04` image where also other dependencies are configured and installed.

Final image consists of few pieces that are integrated into it:
- `polkadot`, `zombienet` and `polkadot-parachain` pre-compiled binaries taken from offical releases
- `polkadot`, `polkadot-parachain`, `polkadot-execute-worker`, `polkadot-prepare-worker` and `zombienet` pre-compiled binaries taken from offical releases
- `moonbeam` binary provided by custom release created on [this fork](https://github.com/karolk91/moonbeam) (custom fork was required to enable dev rpc module which is not supported by offical releases)
- pre-compiled smart contracts available within this repository at [contracts](../contracts/) dir
- [entrypoint.sh](./entrypoint.sh) script that orchestrates running of the above to eventually receive benchmarking results from `smart-bench` itself. The script accepts set of parameters to be passed as-is to `smart-bench` app. It runs zombienet as a background job and starts smart-bench while providing it with correct parameters to reach to zombienet nodes.
Expand All @@ -17,7 +17,7 @@ Please refer to the [download-bin.sh](./download-bin.sh) script for detailed inf
It is possible to run `smart-bench:latest` using raw `docker run` command with options (see examples below) but usage of `run.sh` can abstract away some of the internals of the image and also result in better compatibility with future image versions. The `run.sh` currently helps with following usage scenarios:
1. use `smart-bench:latest` image as is, composed of specific versions of nodes and contracts
1. use `smart-bench:latest` for what it consists of but provide custom set of pre-compiled contracts
1. use `smart-bench:latest` but provide custom set of node binaries (`polkadot`, `zombienet` `polkadot-parachain`, `moonbeam`)
1. use `smart-bench:latest` but provide custom set of node binaries (`polkadot`, `zombienet` `polkadot-parachain`, `polkadot-execute-worker`, `polkadot-prepare-worker`, `moonbeam`)
1. use `smart-bench:latest` but provide custom zombienet config

Usage scenarios above are possible by providing volume mounts arguments to `docker run`, to override specific locations within the resulting container filesystem. And this is exactly what `run.sh` is doing under the hood for you when providing optional parameters (see usage below) to the script.
Expand All @@ -28,7 +28,7 @@ Usage scenarios above are possible by providing volume mounts arguments to `dock

a) downloads dependencies
```
./downloads-bins.sh
./download-bins.sh
```

b) build smart-bench:latest image
Expand All @@ -43,7 +43,7 @@ VERSION=1.0 ./build.sh


### Step 2. Run image
## `run.sh` help screen:
#### `run.sh` help screen:
```
Usage: ./run.sh OPTION -- ARGUMENTS_TO_SMART_BENCH
Expand Down Expand Up @@ -130,17 +130,40 @@ docker run --rm -it --init -v $PWD/configs:/usr/local/smart-bench/config smart-b
---

## Miscellaneous

### Generate list of pre-funded accounts
Following are optional steps that can be used to re-generate list of pre-funded accounts used for benchmarking

create python environment
```
make setup
```

make sure to activate python env
```
source venv/bin/activate
```

use scripts to generate json list of balances
```
# moonbeam / evm
python eth_addr_generator.py > configs/funded-accounts-evm.json
# substrate / wasm
python substrate_addr_generator.py > configs/funded-accounts-wasm.json
```

### Moonbeam with Dev RPC module enabled build recipe

Following is an example recipe how to build moonbeam binary with Dev RPC module enabled
```
git clone https://github.com/PureStake/moonbeam.git
cd moonbeam && git fetch https://github.com/karolk91/moonbeam master && git cherry-pick decd877
docker run -it --rm -v $PWD:/moonbeam docker.io/paritytech/ci-linux:production /bin/bash
docker run -it --rm -v $PWD:/moonbeam docker.io/paritytech/ci-unified:bullseye-1.74.0 /bin/bash
cd /moonbeam
rustup toolchain install 1.69
rustup default 1.69
rustup override set 1.69
rustup target add wasm32-unknown-unknown --toolchain 1.69
rustup toolchain install 1.74
rustup default 1.74
rustup override set 1.74
rustup target add wasm32-unknown-unknown --toolchain 1.74
cargo build --release
```
36 changes: 33 additions & 3 deletions stats/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,23 @@ This folder contains utilities to create visual graphs based on smart-bench meas
Solution is based upon Grafana and InfluxDB software.

### Theory of operation:
1. Script is spinning up ephemeral environemnt with Grafana, Grafana Renderer and InfluxDB services running by utilizing docker-compose.yml configuration
2. Script translates benchmarking data provided in CSV format into Line Protocol format supported by InfluxDB, then uploads it to the InfluxDB service
3. Script is downloading given Grafana panel id (see supported ids beloew) as PNG image by utlizing Grafana plugin pre-configured in the environemnt
1. Gather smart-bench benchmarking results in CSV format
1. make use of an utility script `smart_bench_to_csv.sh`. Run it against smart-bench software multiple times to gather statistics data. Make sure to use meaningful timestamps. For example (any method to run smart-bench):
```
cargo run --release -- evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934
cargo run --release -- ink-wasm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934
cargo run --release -- sol-wasm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934
```
above will create `benchmark-result.csv` file with all `3` results appended

1. or get existing csv results from [gh-pages branch](https://github.com/
paritytech/smart-bench/blob/gh-pages/benchmark-results.csv)
1. make use of `get_graph.sh` to generate graph as PNG image
1. script is spinning up ephemeral environemnt with Grafana, Grafana Renderer and InfluxDB services running by utilizing docker-compose.yml configuration
1. translates benchmarking data provided in CSV format into Line Protocol format supported by InfluxDB, then uploads it to the InfluxDB service
1. script is downloading given Grafana panel id (see supported ids beloew) as PNG image by utlizing Grafana plugin pre-configured in the environemnt

### Currently supported panel ids with examples:
- `--panel-id=2` - panel to display transactions per seconds (TPS) measurements per platform, per contract type
Expand All @@ -30,3 +44,19 @@ EXAMPLE
./get_graph.sh --panel-id=2 --csv-data=benchmark-result.csv --output=tps.png
```

### `smart_bench_to_csv.sh` help screen:
```
Script to translate smart bench stdout data into csv formatted benchmarking results
Script expects data to be piped from smart-bench application into this script
Usage: smart-bench ..... | ./smart_bench_to_csv.sh ARGS
ARGS
-o, --csv-output (Required) CSV formatted output of smart-bench
-t, --timestamp (Optional) Timestamp to use for benchmark results - if not provided, current time is used
-h, --help Print this help message
EXAMPLE
smart-bench evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934
```

81 changes: 81 additions & 0 deletions stats/smart_bench_to_csv.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!/bin/bash

set -euo pipefail

SCRIPT_NAME="${BASH_SOURCE[0]}"
SCRIPT_PATH=$(dirname "$(realpath -s "${BASH_SOURCE[0]}")")
TIMESTAMP=$(date +%s)

if [ -p /dev/stdin ]; then
PROCESS_LIST=$(ps -u)
CONTRACT_TYPE=$(echo "${PROCESS_LIST}" | grep -v grep | grep -oE "erc20|flipper|incrementer|erc721|erc1155|odd-product|triangle-number|storage-read|storage-write|storage-read-write")
PLATFORM=$(echo "${PROCESS_LIST}" | grep -v grep | grep -oE "ink-wasm|sol-wasm|evm")

echo "${CONTRACT_TYPE}"
echo "${PLATFORM}"
STATS=$(</dev/stdin)
else
echo "No input was found on stdin, skipping!"
fi

function echoerr() { echo "$@" 1>&2; }

function usage {
cat << EOF
Script to translate smart bench stdout data into csv formatted benchmarking results and append to given output file
Script expects data to be piped from smart-bench application into this script
Usage: smart-bench ..... | ${SCRIPT_NAME} ARGS
ARGS
-o, --csv-output (Required) CSV formatted output of smart-bench
-t, --timestamp (Optional) Timestamp to use for benchmark results - if not provided, current time is used
-h, --help Print this help message
EXAMPLE
cargo run --release -- evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ${SCRIPT_NAME} --csv-output=benchmark-result.csv --timestamp=1714515934
EOF
}

function parse_args {
function needs_arg {
if [ -z "${OPTARG}" ]; then
echoerr "No arg for --${OPT} option"
exit 2
fi
}

# shellcheck disable=SC2214
while getopts o:t:h-: OPT; do
# support long options: https://stackoverflow.com/a/28466267/519360
if [ "$OPT" = "-" ]; then # long option: reformulate OPT and OPTARG
OPT="${OPTARG%%=*}" # extract long option name
OPTARG="${OPTARG#"$OPT"}" # extract long option argument (may be empty)
OPTARG="${OPTARG#=}" # if long option argument, remove assigning `=`
fi
case "$OPT" in
o | csv-output) needs_arg && CSV_OUTPUT="${OPTARG}";;
t | timestamp ) needs_arg && TIMESTAMP="${OPTARG}";;
h | help ) usage; exit 0;;
??* ) echoerr "Illegal option --$OPT"; exit 2;; # bad long option
? ) exit 2 ;; # bad short option (error reported via getopts)
esac
done
shift $((OPTIND-1)) # remove parsed options and args from $@ list

[ -n "${CSV_OUTPUT-}" ] || {
echoerr "missing -c/--csv-output arg"
echoerr ""
usage
exit 2
}
}

parse_args "$@"


blocks=$(echo ${STATS} | grep -o 'Total Blocks: [0-9]*' | awk '{print $3}')
extrinsics=$(echo ${STATS} | grep -o 'Total Extrinsics: [0-9]*' | awk '{print $3}')
tps=$(echo ${STATS} | grep -o 'sTPS: [0-9]*' | awk '{print $2}')

echo "${TIMESTAMP}, ${PLATFORM}, not-supp, ${CONTRACT_TYPE}, ${tps}, not-supp, not-supp" >> "${CSV_OUTPUT}"

0 comments on commit 20b3d15

Please sign in to comment.