Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce CSV related tools to master #69

Merged
merged 6 commits into from
May 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ jobs:
STATS=$(./run.sh -- ${{ matrix.type }} ${{ matrix.contract }} ${TEST_PARAMS})
BLOCKS=$(echo ${STATS} | grep -o 'Total Blocks: [0-9]*' | awk '{print $3}')
EXTRINSICS=$(echo ${STATS} | grep -o 'Total Extrinsics: [0-9]*' | awk '{print $3}')
TPS=$(echo ${STATS} | grep -o 'sTPS: [0-9]*' | awk '{print $2}')
TPS=$(echo ${STATS} | grep -o '[0-9]\+\.[0-9]\{2\}' | awk '{print $2}')
echo "Blocks: ${BLOCKS}"
echo "Extrinsics: ${EXTRINSICS}"
echo "TPS: ${TPS}"
Expand Down
19 changes: 10 additions & 9 deletions launch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ This directory contains docker related scripts and configurations to compile and

The environment is created as Docker Image (by default named and tagged as `smart-bench:latest`) which leverages multi-stage Docker builds. Helper scripts available in this directory streamline operations to build ([build.sh](./build.sh)) and run ([run.sh](./run.sh)) of the image.

At the first stage of docker build, based upon `docker.io/paritytech/ci-linux:production` as rust enabled compilation env, smart-bench software is being compiled from current sources (this git repository). Resulting binary is copied into second stage / final docker image based on `ubuntu:20.04` image where also other dependencies are configured and installed.
At the first stage of docker build, based upon `docker.io/paritytech/ci-unified:bullseye-1.74.0` as rust enabled compilation env, smart-bench software is being compiled from current sources (this git repository). Resulting binary is copied into second stage / final docker image based on `ubuntu:20.04` image where also other dependencies are configured and installed.

Final image consists of few pieces that are integrated into it:
- `polkadot`, `zombienet` and `polkadot-parachain` pre-compiled binaries taken from offical releases
- `polkadot`, `polkadot-parachain` and `zombienet` pre-compiled binaries taken from offical releases
- `moonbeam` binary provided by custom release created on [this fork](https://github.com/karolk91/moonbeam) (custom fork was required to enable dev rpc module which is not supported by offical releases)
Copy link
Contributor

@smiasojed smiasojed May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which moonbeam branch is compatible with polkadot v1.0.0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the same moonbeam version for both branches

- pre-compiled smart contracts available within this repository at [contracts](../contracts/) dir
- [entrypoint.sh](./entrypoint.sh) script that orchestrates running of the above to eventually receive benchmarking results from `smart-bench` itself. The script accepts set of parameters to be passed as-is to `smart-bench` app. It runs zombienet as a background job and starts smart-bench while providing it with correct parameters to reach to zombienet nodes.
Expand All @@ -28,7 +28,7 @@ Usage scenarios above are possible by providing volume mounts arguments to `dock

a) downloads dependencies
```
./downloads-bins.sh
./download-bins.sh
```

b) build smart-bench:latest image
Expand All @@ -43,7 +43,7 @@ VERSION=1.0 ./build.sh


### Step 2. Run image
## `run.sh` help screen:
#### `run.sh` help screen:
```
Usage: ./run.sh OPTION -- ARGUMENTS_TO_SMART_BENCH

Expand Down Expand Up @@ -130,17 +130,18 @@ docker run --rm -it --init -v $PWD/configs:/usr/local/smart-bench/config smart-b
---

## Miscellaneous

Copy link
Contributor

@smiasojed smiasojed May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not using prefunded account on master - master is compatible with polkadot 1.0.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will remove the section, thanks for spotting this

### Moonbeam with Dev RPC module enabled build recipe

Following is an example recipe how to build moonbeam binary with Dev RPC module enabled
```
git clone https://github.com/PureStake/moonbeam.git
cd moonbeam && git fetch https://github.com/karolk91/moonbeam master && git cherry-pick decd877
docker run -it --rm -v $PWD:/moonbeam docker.io/paritytech/ci-linux:production /bin/bash
docker run -it --rm -v $PWD:/moonbeam docker.io/paritytech/ci-unified:bullseye-1.74.0 /bin/bash
cd /moonbeam
rustup toolchain install 1.69
rustup default 1.69
rustup override set 1.69
rustup target add wasm32-unknown-unknown --toolchain 1.69
rustup toolchain install 1.74
rustup default 1.74
rustup override set 1.74
rustup target add wasm32-unknown-unknown --toolchain 1.74
cargo build --release
```
18 changes: 18 additions & 0 deletions src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ mod integration_tests;
mod stats;
mod wasm;

use std::fmt::Display;

// export for use by contract! macro
use clap::Parser;
pub use stats::{collect_block_stats, print_block_info, BlockInfo};
Expand Down Expand Up @@ -51,12 +53,28 @@ pub enum Contract {
StorageReadWrite,
}

impl Display for TargetPlatform {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", clap::ArgEnum::to_possible_value(self).unwrap_or("unknown".into()).get_name())
}
}

impl Display for Contract {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", clap::ArgEnum::to_possible_value(self).unwrap_or("unknown".into()).get_name())
}
}

#[tokio::main]
async fn main() -> color_eyre::Result<()> {
color_eyre::install()?;
let cli = Cli::parse();
tracing_subscriber::fmt::init();

println!("Smart-bench run parameters:");
println!("Platform: {}", cli.chain);
println!("Contracts: {}", cli.contracts.iter().map(|arg| arg.to_string()).collect::<Vec<_>>().join("+"));

match cli.chain {
TargetPlatform::InkWasm => wasm::exec(cli).await,
TargetPlatform::SolWasm => wasm::exec(cli).await,
Expand Down
50 changes: 46 additions & 4 deletions stats/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,36 @@ This folder contains utilities to create visual graphs based on smart-bench meas

Solution is based upon Grafana and InfluxDB software.

### Theory of operation:
1. Script is spinning up ephemeral environemnt with Grafana, Grafana Renderer and InfluxDB services running by utilizing docker-compose.yml configuration
2. Script translates benchmarking data provided in CSV format into Line Protocol format supported by InfluxDB, then uploads it to the InfluxDB service
3. Script is downloading given Grafana panel id (see supported ids beloew) as PNG image by utlizing Grafana plugin pre-configured in the environemnt
### Theory of operation

1. Gather smart-bench benchmarking results in CSV format
Make use of an utility script `smart_bench_to_csv.sh`. Run it against smart-bench software multiple times to gather statistics data. Make sure to use meaningful timestamps. For example (any method to run smart-bench):

```
cargo run --release -- evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934

cargo run --release -- ink-wasm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934

cargo run --release -- sol-wasm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934
```

or

```
../launch/run.sh -- evm flipper --instance-count 1 --call-count 1500 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934

../launch/run.sh -- ink-wasm flipper --instance-count 1 --call-count 1500 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934

../launch/run.sh -- sol-wasm flipper --instance-count 1 --call-count 1500 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934
```

above will create `benchmark-result.csv` file with all `3` results appended

or get existing csv results from [gh-pages branch](https://github.com/paritytech/smart-bench/tree/gh-pages)
2. Make use of `get_graph.sh` to generate graph as PNG image
- script is spinning up ephemeral environemnt with Grafana, Grafana Renderer and InfluxDB services running by utilizing docker-compose.yml configuration
- translates benchmarking data provided in CSV format into Line Protocol format supported by InfluxDB, then uploads it to the InfluxDB service
- script is downloading given Grafana panel id (see supported ids beloew) as PNG image by utlizing Grafana plugin pre-configured in the environemnt

### Currently supported panel ids with examples:
- `--panel-id=2` - panel to display transactions per seconds (TPS) measurements per platform, per contract type
Expand All @@ -30,3 +56,19 @@ EXAMPLE
./get_graph.sh --panel-id=2 --csv-data=benchmark-result.csv --output=tps.png
```

### `smart_bench_to_csv.sh` help screen:
```
Script to translate smart bench stdout data into csv formatted benchmarking results
Script expects data to be piped from smart-bench application into this script

Usage: smart-bench ..... | ./smart_bench_to_csv.sh ARGS

ARGS
-o, --csv-output (Required) CSV formatted output of smart-bench
-t, --timestamp (Optional) Timestamp to use for benchmark results - if not provided, current time is used
-h, --help Print this help message

EXAMPLE
smart-bench evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ./smart_bench_to_csv.sh --csv-output=benchmark-result.csv --timestamp=1714515934
```

23 changes: 18 additions & 5 deletions stats/get_graph.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,19 @@ DOCKER_COMPOSE_FILE="docker-compose.yml"
HOST="localhost"
GRAFANA_DASHBOARD_JSON="grafana-provisioning/dashboards/tps.json"

check_command() {
command -v "$1" >/dev/null 2>&1 && { $1 --version >/dev/null 2>&1; }
}

if check_command "docker compose"; then
DOCKER_COMPOSE="docker compose"
elif check_command "docker-compose"; then
DOCKER_COMPOSE="docker-compose"
else
echo "Neither 'docker compose' nor 'docker-compose' could be found or are functional on your system."
exit 1
fi

function echoerr() { echo "$@" 1>&2; }

function usage {
Expand Down Expand Up @@ -125,24 +138,24 @@ wait_for_containers() {
local retry_interval=1

echo "Waiting for all Docker containers to be running..."
retry_command "$max_retries" "$retry_interval" docker compose -f "$DOCKER_COMPOSE_FILE" \
retry_command "$max_retries" "$retry_interval" ${DOCKER_COMPOSE} -f "$DOCKER_COMPOSE_FILE" \
ps --services --filter "status=running" | grep -qvx " "
}

start_containers() {
echo "Starting docker containers"
docker compose -f "$DOCKER_COMPOSE_FILE" up -d
${DOCKER_COMPOSE} -f "$DOCKER_COMPOSE_FILE" up -d
wait_for_containers
}

stop_containers() {
echo "Stopping docker containers"
docker compose -f "$DOCKER_COMPOSE_FILE" down
${DOCKER_COMPOSE} -f "$DOCKER_COMPOSE_FILE" down
}

print_containers_logs() {
echo "Container logs -- START"
docker-compose -f "$DOCKER_COMPOSE_FILE" logs
${DOCKER_COMPOSE} -f "$DOCKER_COMPOSE_FILE" logs
echo "Container logs -- END"
}

Expand All @@ -158,7 +171,7 @@ convert_csv_to_line_protocol() {
gsub(/ /,"\\ ", field);
return field
}{
printf "%s,platform=%s,parachain_ver=%s,contract_type=%s,contract_compiler_ver=\"%s\" tx_per_sec=%s %s\n",
printf "%s,platform=\"%s\",parachain_ver=\"%s\",contract_type=\"%s\",contract_compiler_ver=\"%s\" tx_per_sec=%s %s\n",
"tps",
trim($2),
trim($3),
Expand Down
75 changes: 75 additions & 0 deletions stats/smart_bench_to_csv.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
#!/bin/bash
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could add output-csv option to smart-bench?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, thats good idea, we can think of this pull request as some intermediate step towards better solution

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, lets create an issue for improvement

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this might be a better idea just to add the option to output as csv to smart-bench itself.


set -euo pipefail

SCRIPT_NAME="${BASH_SOURCE[0]}"
SCRIPT_PATH=$(dirname "$(realpath -s "${BASH_SOURCE[0]}")")
TIMESTAMP=$(date +%s)

if [ -p /dev/stdin ]; then
STATS=$(</dev/stdin)
else
echo "No input was found on stdin, skipping!"
fi

function echoerr() { echo "$@" 1>&2; }

function usage {
cat << EOF
Script to translate smart bench stdout data into csv formatted benchmarking results and append to given output file
Script expects data to be piped from smart-bench application into this script

Usage: smart-bench ..... | ${SCRIPT_NAME} ARGS

ARGS
-o, --csv-output (Required) CSV formatted output of smart-bench
-t, --timestamp (Optional) Timestamp to use for benchmark results - if not provided, current time is used
-h, --help Print this help message

EXAMPLE
cargo run --release -- evm flipper --instance-count 1 --call-count 1500 --url ws://localhost:9988 | ${SCRIPT_NAME} --csv-output=benchmark-result.csv --timestamp=1714515934
EOF
}

function parse_args {
function needs_arg {
if [ -z "${OPTARG}" ]; then
echoerr "No arg for --${OPT} option"
exit 2
fi
}

# shellcheck disable=SC2214
while getopts o:t:h-: OPT; do
# support long options: https://stackoverflow.com/a/28466267/519360
if [ "$OPT" = "-" ]; then # long option: reformulate OPT and OPTARG
OPT="${OPTARG%%=*}" # extract long option name
OPTARG="${OPTARG#"$OPT"}" # extract long option argument (may be empty)
OPTARG="${OPTARG#=}" # if long option argument, remove assigning `=`
fi
case "$OPT" in
o | csv-output) needs_arg && CSV_OUTPUT="${OPTARG}";;
t | timestamp ) needs_arg && TIMESTAMP="${OPTARG}";;
h | help ) usage; exit 0;;
??* ) echoerr "Illegal option --$OPT"; exit 2;; # bad long option
? ) exit 2 ;; # bad short option (error reported via getopts)
esac
done
shift $((OPTIND-1)) # remove parsed options and args from $@ list

[ -n "${CSV_OUTPUT-}" ] || {
echoerr "missing -c/--csv-output arg"
echoerr ""
usage
exit 2
}
}

parse_args "$@"


platform=$(echo ${STATS} | grep -o 'Platform: [a-z0-9-]*' | awk '{print $2}')
contract_types=$(echo ${STATS} | grep -o 'Contracts: [+a-z0-9-]*' | awk '{print $2}')
tps=$(echo ${STATS} | grep -o 'sTPS: [0-9]\+\.[0-9]\{2\}' | awk '{print $2}')

echo "${TIMESTAMP}, ${platform}, n/a, ${contract_types}, ${tps}, n/a, n/a" >> "${CSV_OUTPUT}"
Loading