From 20b3d15206bafaec04928f419a6b0a7c024e2dc7 Mon Sep 17 00:00:00 2001 From: Karol Kokoszka Date: Wed, 1 May 2024 01:01:35 +0200 Subject: [PATCH] Introduce smart_bench_to_csv.sh script, README fixes --- launch/README.md | 43 +++++++++++++++----- stats/README.md | 36 +++++++++++++++-- stats/smart_bench_to_csv.sh | 81 +++++++++++++++++++++++++++++++++++++ 3 files changed, 147 insertions(+), 13 deletions(-) create mode 100755 stats/smart_bench_to_csv.sh diff --git a/launch/README.md b/launch/README.md index 0f071181..a6be8600 100644 --- a/launch/README.md +++ b/launch/README.md @@ -4,10 +4,10 @@ This directory contains docker related scripts and configurations to compile and The environment is created as Docker Image (by default named and tagged as `smart-bench:latest`) which leverages multi-stage Docker builds. Helper scripts available in this directory streamline operations to build ([build.sh](./build.sh)) and run ([run.sh](./run.sh)) of the image. -At the first stage of docker build, based upon `docker.io/paritytech/ci-linux:production` as rust enabled compilation env, smart-bench software is being compiled from current sources (this git repository). Resulting binary is copied into second stage / final docker image based on `ubuntu:20.04` image where also other dependencies are configured and installed. +At the first stage of docker build, based upon `docker.io/paritytech/ci-unified:bullseye-1.74.0` as rust enabled compilation env, smart-bench software is being compiled from current sources (this git repository). Resulting binary is copied into second stage / final docker image based on `ubuntu:20.04` image where also other dependencies are configured and installed. Final image consists of few pieces that are integrated into it: -- `polkadot`, `zombienet` and `polkadot-parachain` pre-compiled binaries taken from offical releases +- `polkadot`, `polkadot-parachain`, `polkadot-execute-worker`, `polkadot-prepare-worker` and `zombienet` pre-compiled binaries taken from offical releases - `moonbeam` binary provided by custom release created on [this fork](https://github.com/karolk91/moonbeam) (custom fork was required to enable dev rpc module which is not supported by offical releases) - pre-compiled smart contracts available within this repository at [contracts](../contracts/) dir - [entrypoint.sh](./entrypoint.sh) script that orchestrates running of the above to eventually receive benchmarking results from `smart-bench` itself. The script accepts set of parameters to be passed as-is to `smart-bench` app. It runs zombienet as a background job and starts smart-bench while providing it with correct parameters to reach to zombienet nodes. @@ -17,7 +17,7 @@ Please refer to the [download-bin.sh](./download-bin.sh) script for detailed inf It is possible to run `smart-bench:latest` using raw `docker run` command with options (see examples below) but usage of `run.sh` can abstract away some of the internals of the image and also result in better compatibility with future image versions. The `run.sh` currently helps with following usage scenarios: 1. use `smart-bench:latest` image as is, composed of specific versions of nodes and contracts 1. use `smart-bench:latest` for what it consists of but provide custom set of pre-compiled contracts -1. use `smart-bench:latest` but provide custom set of node binaries (`polkadot`, `zombienet` `polkadot-parachain`, `moonbeam`) +1. use `smart-bench:latest` but provide custom set of node binaries (`polkadot`, `zombienet` `polkadot-parachain`, `polkadot-execute-worker`, `polkadot-prepare-worker`, `moonbeam`) 1. use `smart-bench:latest` but provide custom zombienet config Usage scenarios above are possible by providing volume mounts arguments to `docker run`, to override specific locations within the resulting container filesystem. And this is exactly what `run.sh` is doing under the hood for you when providing optional parameters (see usage below) to the script. @@ -28,7 +28,7 @@ Usage scenarios above are possible by providing volume mounts arguments to `dock a) downloads dependencies ``` -./downloads-bins.sh +./download-bins.sh ``` b) build smart-bench:latest image @@ -43,7 +43,7 @@ VERSION=1.0 ./build.sh ### Step 2. Run image -## `run.sh` help screen: +#### `run.sh` help screen: ``` Usage: ./run.sh OPTION -- ARGUMENTS_TO_SMART_BENCH @@ -130,17 +130,40 @@ docker run --rm -it --init -v $PWD/configs:/usr/local/smart-bench/config smart-b --- ## Miscellaneous + +### Generate list of pre-funded accounts +Following are optional steps that can be used to re-generate list of pre-funded accounts used for benchmarking + +create python environment +``` +make setup +``` + +make sure to activate python env +``` +source venv/bin/activate +``` + +use scripts to generate json list of balances +``` +# moonbeam / evm +python eth_addr_generator.py > configs/funded-accounts-evm.json + +# substrate / wasm +python substrate_addr_generator.py > configs/funded-accounts-wasm.json +``` + ### Moonbeam with Dev RPC module enabled build recipe Following is an example recipe how to build moonbeam binary with Dev RPC module enabled ``` git clone https://github.com/PureStake/moonbeam.git cd moonbeam && git fetch https://github.com/karolk91/moonbeam master && git cherry-pick decd877 -docker run -it --rm -v $PWD:/moonbeam docker.io/paritytech/ci-linux:production /bin/bash +docker run -it --rm -v $PWD:/moonbeam docker.io/paritytech/ci-unified:bullseye-1.74.0 /bin/bash cd /moonbeam -rustup toolchain install 1.69 -rustup default 1.69 -rustup override set 1.69 -rustup target add wasm32-unknown-unknown --toolchain 1.69 +rustup toolchain install 1.74 +rustup default 1.74 +rustup override set 1.74 +rustup target add wasm32-unknown-unknown --toolchain 1.74 cargo build --release ``` diff --git a/stats/README.md b/stats/README.md index 36cef8a0..ae6e117e 100644 --- a/stats/README.md +++ b/stats/README.md @@ -5,9 +5,23 @@ This folder contains utilities to create visual graphs based on smart-bench meas Solution is based upon Grafana and InfluxDB software. ### Theory of operation: -1. Script is spinning up ephemeral environemnt with Grafana, Grafana Renderer and InfluxDB services running by utilizing docker-compose.yml configuration -2. Script translates benchmarking data provided in CSV format into Line Protocol format supported by InfluxDB, then uploads it to the InfluxDB service -3. Script is downloading given Grafana panel id (see supported ids beloew) as PNG image by utlizing Grafana plugin pre-configured in the environemnt +1. Gather smart-bench benchmarking results in CSV format + 1. make use of an utility script `smart_bench_to_csv.sh`. Run it against smart-bench software multiple times to gather statistics data. Make sure to use meaningful timestamps. For example (any method to run smart-bench): + ``` + cargo run --release -- evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934 + + cargo run --release -- ink-wasm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934 + + cargo run --release -- sol-wasm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934 + ``` + above will create `benchmark-result.csv` file with all `3` results appended + + 1. or get existing csv results from [gh-pages branch](https://github.com/ + paritytech/smart-bench/blob/gh-pages/benchmark-results.csv) +1. make use of `get_graph.sh` to generate graph as PNG image + 1. script is spinning up ephemeral environemnt with Grafana, Grafana Renderer and InfluxDB services running by utilizing docker-compose.yml configuration + 1. translates benchmarking data provided in CSV format into Line Protocol format supported by InfluxDB, then uploads it to the InfluxDB service + 1. script is downloading given Grafana panel id (see supported ids beloew) as PNG image by utlizing Grafana plugin pre-configured in the environemnt ### Currently supported panel ids with examples: - `--panel-id=2` - panel to display transactions per seconds (TPS) measurements per platform, per contract type @@ -30,3 +44,19 @@ EXAMPLE ./get_graph.sh --panel-id=2 --csv-data=benchmark-result.csv --output=tps.png ``` +### `smart_bench_to_csv.sh` help screen: +``` +Script to translate smart bench stdout data into csv formatted benchmarking results +Script expects data to be piped from smart-bench application into this script + +Usage: smart-bench ..... | ./smart_bench_to_csv.sh ARGS + +ARGS + -o, --csv-output (Required) CSV formatted output of smart-bench + -t, --timestamp (Optional) Timestamp to use for benchmark results - if not provided, current time is used + -h, --help Print this help message + +EXAMPLE +smart-bench evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934 +``` + diff --git a/stats/smart_bench_to_csv.sh b/stats/smart_bench_to_csv.sh new file mode 100755 index 00000000..62def777 --- /dev/null +++ b/stats/smart_bench_to_csv.sh @@ -0,0 +1,81 @@ +#!/bin/bash + +set -euo pipefail + +SCRIPT_NAME="${BASH_SOURCE[0]}" +SCRIPT_PATH=$(dirname "$(realpath -s "${BASH_SOURCE[0]}")") +TIMESTAMP=$(date +%s) + +if [ -p /dev/stdin ]; then + PROCESS_LIST=$(ps -u) + CONTRACT_TYPE=$(echo "${PROCESS_LIST}" | grep -v grep | grep -oE "erc20|flipper|incrementer|erc721|erc1155|odd-product|triangle-number|storage-read|storage-write|storage-read-write") + PLATFORM=$(echo "${PROCESS_LIST}" | grep -v grep | grep -oE "ink-wasm|sol-wasm|evm") + + echo "${CONTRACT_TYPE}" + echo "${PLATFORM}" + STATS=$(&2; } + +function usage { + cat << EOF +Script to translate smart bench stdout data into csv formatted benchmarking results and append to given output file +Script expects data to be piped from smart-bench application into this script + +Usage: smart-bench ..... | ${SCRIPT_NAME} ARGS + +ARGS + -o, --csv-output (Required) CSV formatted output of smart-bench + -t, --timestamp (Optional) Timestamp to use for benchmark results - if not provided, current time is used + -h, --help Print this help message + +EXAMPLE +cargo run --release -- evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ${SCRIPT_NAME} --csv-output=benchmark-result.csv --timestamp=1714515934 +EOF +} + +function parse_args { + function needs_arg { + if [ -z "${OPTARG}" ]; then + echoerr "No arg for --${OPT} option" + exit 2 + fi + } + + # shellcheck disable=SC2214 + while getopts o:t:h-: OPT; do + # support long options: https://stackoverflow.com/a/28466267/519360 + if [ "$OPT" = "-" ]; then # long option: reformulate OPT and OPTARG + OPT="${OPTARG%%=*}" # extract long option name + OPTARG="${OPTARG#"$OPT"}" # extract long option argument (may be empty) + OPTARG="${OPTARG#=}" # if long option argument, remove assigning `=` + fi + case "$OPT" in + o | csv-output) needs_arg && CSV_OUTPUT="${OPTARG}";; + t | timestamp ) needs_arg && TIMESTAMP="${OPTARG}";; + h | help ) usage; exit 0;; + ??* ) echoerr "Illegal option --$OPT"; exit 2;; # bad long option + ? ) exit 2 ;; # bad short option (error reported via getopts) + esac + done + shift $((OPTIND-1)) # remove parsed options and args from $@ list + + [ -n "${CSV_OUTPUT-}" ] || { + echoerr "missing -c/--csv-output arg" + echoerr "" + usage + exit 2 + } +} + +parse_args "$@" + + +blocks=$(echo ${STATS} | grep -o 'Total Blocks: [0-9]*' | awk '{print $3}') +extrinsics=$(echo ${STATS} | grep -o 'Total Extrinsics: [0-9]*' | awk '{print $3}') +tps=$(echo ${STATS} | grep -o 'sTPS: [0-9]*' | awk '{print $2}') + +echo "${TIMESTAMP}, ${PLATFORM}, not-supp, ${CONTRACT_TYPE}, ${tps}, not-supp, not-supp" >> "${CSV_OUTPUT}"