regression and benchmark testing between jerigon and zkevm #751

temaniarpit27 · 2024-10-26T15:56:25Z

temaniarpit27 · 2024-10-27T09:33:46Z

.github/workflows/cron_jerigon_zero_testing.yml

+      - name: Run Erigon Network
+        run: |
+          cd ..
+          tar xf "$(pwd)/zk_evm/test_data/erigon-data.tar.gz" || {


@Nashtare @praetoriansentry what would be best way to store the file? currently added with repo

It is not big, I think it can stay with repo.

we are using a shorter data (672) blocks, if we use that, its probably fine but if we choose to go with a larger set of data, we may need to think

temaniarpit27 · 2024-10-27T09:35:04Z

.github/workflows/cron_jerigon_zero_testing.yml

+        run: |
+          export ETH_RPC_URL="http://localhost:8545"
+          rm -rf proofs/* circuits/* ./proofs.json test.out verify.out leader.out
+          random_numbers=($(shuf -i 1-500 -n 5))


@Nashtare @praetoriansentry currently using random 5 blocks for integration testing, any pref for block count

I don't think it is a good idea to take random 5 blocks. Should be all 500 probably, if we run it once a week. Maybe 10 or 100, if 500 is too much.

ya its just for testing, we will choose whatever we think is fine. i was thinking somewhere around 100

temaniarpit27 · 2024-10-27T09:35:44Z

scripts/jerigon_zero_benchmark.sh

+echo "$(date +"%Y-%m-%d %H:%M:%S")" &>> "$OUTPUT_LOG"
+
+# Define the blocks to process
+blocks=(100 200 300 400 500)


@praetoriansentry @Nashtare currently using random blocks, any idea which blocks to use for benchmark testing

Maybe we could read the list from the config file (CSV to make it simple?).

yes this is just a placeholder. we will choose block numbers first and if thats too much, we will put that in config file

temaniarpit27 · 2024-10-27T09:37:06Z

@praetoriansentry @Nashtare currently using erigon.stopped.0.to.672.tar.gz as erigon data. Anything else we need to use. This is a test network data which has 672 blocks

temaniarpit27 · 2024-10-27T09:37:15Z

.github/workflows/cron_jerigon_zero_testing.yml

+name: Jerigon Zero Testing
+
+on:
+  # Uncomment when ready to run on a schedule


temaniarpit27 · 2024-10-27T09:39:22Z

Sample output in log file

2024-10-27 09:25:52
Processing block: 1
Proving duration for block 1: 2.523 seconds, performance time: 1.467795802, performance user time: 2.039623000, performance system time: 0.119509000
Processing block: 2
Proving duration for block 2: 1.511 seconds, performance time: 1.498711832, performance user time: 1.577669000, performance system time: 0.125585000
Processing completed at: 2024-10-27 09:25:59

2024-10-27 09:32:03
Processing block: 1
Proving duration for block 1: 2.334 seconds, performance time: 1.487433304, performance user time: 1.674814000, performance system time: 0.129767000
Processing block: 2
Proving duration for block 2: 1.563 seconds, performance time: 1.551011885, performance user time: 1.970833000, performance system time: 0.193141000
Processing completed at: 2024-10-27 09:32:10

@praetoriansentry @Nashtare let me know what more data we want to add to this.

atanmarko · 2024-10-29T11:28:25Z

.github/workflows/cron_jerigon_zero_testing.yml

+jobs:
+  jerigon_zero_testing:
+    name: Jerigon Zero Testing - Integration and Benchmarking
+    runs-on: zero-ci


zero-ci is shared among other tasks. Ben created zero-reg - stronger, and for use only with benchmarks. We should just make sure that we schedule checks at different time.

atanmarko · 2024-10-29T11:30:11Z

.github/workflows/cron_jerigon_zero_testing.yml

+  push:
+    branches: [develop]
+  pull_request:
+    branches:
+      - "**"


This will be removed when PR is debugged, right?

yes, this is just for testing

atanmarko · 2024-10-29T11:31:15Z

.github/workflows/cron_jerigon_zero_testing.yml

+        run: |
+          export ETH_RPC_URL="http://localhost:8545"
+          rm -rf proofs/* circuits/* ./proofs.json test.out verify.out leader.out
+          random_numbers=($(shuf -i 1-500 -n 5))


I don't think it is a good idea to take random 5 blocks. Should be all 500 probably, if we run it once a week. Maybe 10 or 100, if 500 is too much.

atanmarko · 2024-10-29T11:33:01Z

.github/workflows/cron_jerigon_zero_testing.yml

+      - name: Run Erigon Network
+        run: |
+          cd ..
+          tar xf "$(pwd)/zk_evm/test_data/erigon-data.tar.gz" || {


It is not big, I think it can stay with repo.

atanmarko · 2024-10-29T11:42:36Z

scripts/jerigon_zero_benchmark.sh

+    # Run performance stats
+    if ! perf stat -e cycles ./target/release/leader --runtime in-memory --load-strategy monolithic --block-batch-size "$BLOCK_BATCH_SIZE" --proof-output-dir "$PROOF_OUTPUT_DIR" stdio < "output_${block}.json" &> "$BLOCK_OUTPUT_LOG"; then
+        echo "Performance command failed for block: $block" &>> "$OUTPUT_LOG"
+        cat "$BLOCK_OUTPUT_LOG" &>> "$OUTPUT_LOG"


I would use one output file to append performance measurements, and the other for error log details. This way, any error that happens will dump big log to the previous performance results and would make them hard to read.

atanmarko · 2024-10-29T11:45:23Z

scripts/jerigon_zero_benchmark.sh

+    set +o pipefail
+    if ! cat "$BLOCK_OUTPUT_LOG" | grep "Successfully wrote to disk proof file " | awk '{print $NF}' | tee "$PROOFS_FILE_LIST"; then
+        echo "Proof list not generated for block: $block. Check the log for details." &>> "$OUTPUT_LOG"
+        cat "$BLOCK_OUTPUT_LOG" &>> "$OUTPUT_LOG"


Is $BLOCK_OUTPUT_LOG the whole leader output terminal log? We should just keep minimal details and time measurements, as we want to track changes over weeks - not to append whole debug/info leader log to the results file. One liner with the block that error-ed in results file, and separate file with whole error log.

this is printed only in case of failure, if a block proving fails. otherwise we wont know why a block failed
if it succeeds, then we are printing only details about time measurements.
maybe, we can move failure blocks to a new file or something like that

You can print 'Proof list not generated for block: $block. Check the log for details."' so that we know the block failed. Then you check the error logs in the other file.

atanmarko · 2024-10-29T11:51:38Z

scripts/jerigon_zero_benchmark.sh

+    echo "Processing block: $block" &>> "$OUTPUT_LOG"
+
+    # Fetch block data
+    if ! ./target/release/rpc --rpc-url "$ETH_RPC_URL" fetch --start-block "$block" --end-block "$block" > "output_${block}.json"; then


output_${block}.json is a bit misleading. Maybe witness_${block}.json?

atanmarko · 2024-10-29T11:54:50Z

scripts/jerigon_zero_benchmark.sh

+process_block() {
+    local block=$1
+
+    echo "Processing block: $block" &>> "$OUTPUT_LOG"


I would keep the results file minimal - maybe one line with date/time of measurement, and the timings for the blocks. Would be easy that way to open file and see how the performances are changed over weeks and months.

#751 (comment)

this is the expected output
let me know if we need to reduce more. i think this also gives clear info along with being concise

Line Processing block: 1 is redundant, does not bring any new information.

github-actions bot added the ci label Oct 26, 2024

temaniarpit27 force-pushed the arpit/704 branch 15 times, most recently from 45eb44d to 581be7e Compare October 26, 2024 19:03

draft for integration testing

3e0ff27

temaniarpit27 force-pushed the arpit/704 branch from 581be7e to 3e0ff27 Compare October 26, 2024 19:06

temaniarpit27 changed the title ~~draft for integration testing~~ regression and benchmark testing between jerigon and zkevm Oct 27, 2024

temaniarpit27 force-pushed the arpit/704 branch 11 times, most recently from 9967181 to 18f7058 Compare October 27, 2024 09:17

temaniarpit27 force-pushed the arpit/704 branch 3 times, most recently from 0dceebc to 4dc1e8f Compare October 27, 2024 09:28

draft for benchmarking

e254c8a

temaniarpit27 force-pushed the arpit/704 branch from 4dc1e8f to e254c8a Compare October 27, 2024 09:32

temaniarpit27 commented Oct 27, 2024

View reviewed changes

temaniarpit27 requested a review from praetoriansentry October 27, 2024 09:37

temaniarpit27 marked this pull request as ready for review October 27, 2024 09:37

temaniarpit27 requested review from 0xaatif, atanmarko and Nashtare as code owners October 27, 2024 09:37

atanmarko reviewed Oct 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

regression and benchmark testing between jerigon and zkevm #751

regression and benchmark testing between jerigon and zkevm #751

temaniarpit27 commented Oct 26, 2024 •

edited

Loading

temaniarpit27 Oct 27, 2024 •

edited

Loading

atanmarko Oct 29, 2024

temaniarpit27 Oct 29, 2024

temaniarpit27 Oct 27, 2024 •

edited

Loading

atanmarko Oct 29, 2024 •

edited

Loading

temaniarpit27 Oct 29, 2024

temaniarpit27 Oct 27, 2024

atanmarko Oct 29, 2024

temaniarpit27 Oct 29, 2024 •

edited

Loading

temaniarpit27 commented Oct 27, 2024

temaniarpit27 Oct 27, 2024

temaniarpit27 commented Oct 27, 2024

atanmarko Oct 29, 2024

atanmarko Oct 29, 2024

temaniarpit27 Oct 29, 2024

atanmarko Oct 29, 2024 •

edited

Loading

atanmarko Oct 29, 2024

atanmarko Oct 29, 2024

atanmarko Oct 29, 2024 •

edited

Loading

temaniarpit27 Oct 29, 2024

atanmarko Oct 29, 2024

atanmarko Oct 29, 2024

atanmarko Oct 29, 2024

temaniarpit27 Oct 29, 2024

atanmarko Oct 29, 2024

regression and benchmark testing between jerigon and zkevm #751

Are you sure you want to change the base?

regression and benchmark testing between jerigon and zkevm #751

Conversation

temaniarpit27 commented Oct 26, 2024 • edited Loading

temaniarpit27 Oct 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

temaniarpit27 Oct 27, 2024 • edited Loading

Choose a reason for hiding this comment

atanmarko Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

temaniarpit27 Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

temaniarpit27 commented Oct 27, 2024

Choose a reason for hiding this comment

temaniarpit27 commented Oct 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atanmarko Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atanmarko Oct 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

temaniarpit27 commented Oct 26, 2024 •

edited

Loading

temaniarpit27 Oct 27, 2024 •

edited

Loading

temaniarpit27 Oct 27, 2024 •

edited

Loading

atanmarko Oct 29, 2024 •

edited

Loading

temaniarpit27 Oct 29, 2024 •

edited

Loading

atanmarko Oct 29, 2024 •

edited

Loading

atanmarko Oct 29, 2024 •

edited

Loading