Skip to content

Commit

Permalink
[CI] do not run multiple iterations for the determinism test (#16712)
Browse files Browse the repository at this point in the history
## Description 

A small followup to #16707, emit more info from the script and only run
the determinism test (last step) once.

## Test Plan 

https://github.com/MystenLabs/sui/actions/runs/8311292117

---
If your changes are not user-facing and do not break anything, you can
skip the following section. Otherwise, please briefly describe what has
changed under the Release Notes section.

### Type of Change (Check all that apply)

- [ ] protocol change
- [ ] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration

### Release notes
  • Loading branch information
mwtian authored Mar 17, 2024
1 parent 1cedbb8 commit 03dc1da
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 3 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/simulator-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ jobs:
# Run simulator tests
- name: Run simtest
run: |
tsh -i ${{ steps.auth.outputs.identity-file }} --ttl 120 ssh ubuntu@simtest-01 "source ~/.bashrc && source ~/.cargo/env && cd ~/sui && RUSTUP_MAX_RETRIES=10 CARGO_TERM_COLOR=always CARGO_INCREMENTAL=0 CARGO_NET_RETRY=10 RUST_BACKTRACE=short RUST_LOG=off NUM_CPUS=24 MSIM_TEST_NUM=${{ env.TEST_NUM }} ./scripts/simtest/simtest-run.sh"
tsh -i ${{ steps.auth.outputs.identity-file }} --ttl 120 ssh ubuntu@simtest-01 "source ~/.bashrc && source ~/.cargo/env && cd ~/sui && RUSTUP_MAX_RETRIES=10 CARGO_TERM_COLOR=always CARGO_INCREMENTAL=0 CARGO_NET_RETRY=10 RUST_BACKTRACE=short RUST_LOG=off NUM_CPUS=24 TEST_NUM=${{ env.TEST_NUM }} ./scripts/simtest/simtest-run.sh"
notify:
name: Notify
Expand Down
30 changes: 28 additions & 2 deletions scripts/simtest/simtest-run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,20 @@ SIMTEST_LOGS_DIR=~/simtest_logs
LOG_DIR="${SIMTEST_LOGS_DIR}/${DATE}"
LOG_FILE="$LOG_DIR/log"

# By default run 30 iterations for each test. But allow overrides.
${MSIM_TEST_NUM:=30}
# By default run 1 iteration for each test, if not specified.
: ${TEST_NUM:=1}

echo ""
echo "================================================"
echo "Running e2e simtests with $TEST_NUM iterations"
echo "================================================"
date

# This command runs many different tests, so it already uses all CPUs fairly efficiently, and
# don't need to be done inside of the for loop below.
# TODO: this logs directly to stdout since it is not being run in parallel. is that ok?
MSIM_TEST_SEED="$SEED" \
MSIM_TEST_NUM=${TEST_NUM} \
MSIM_WATCHDOG_TIMEOUT_MS=60000 \
scripts/simtest/cargo-simtest simtest \
--color always \
Expand All @@ -50,6 +57,12 @@ scripts/simtest/cargo-simtest simtest \
--profile simtestnightly \
-E "$TEST_FILTER" 2>&1 | tee "$LOG_FILE"

echo ""
echo "============================================="
echo "Running $NUM_CPUS stress simtests in parallel"
echo "============================================="
date

for SUB_SEED in `seq 1 $NUM_CPUS`; do
SEED="$SUB_SEED$DATE"
LOG_FILE="$LOG_DIR/log-$SEED"
Expand All @@ -72,10 +85,18 @@ done
# wait for all the jobs to end
wait

echo ""
echo "==========================="
echo "Running determinism simtest"
echo "==========================="
date

# Check for determinism in stress simtests
LOG_FILE="$LOG_DIR/determinism-log"
echo "Using MSIM_TEST_SEED=$SEED, logging to $LOG_FILE"

MSIM_TEST_SEED="$SEED" \
MSIM_TEST_NUM=1 \
MSIM_WATCHDOG_TIMEOUT_MS=60000 \
MSIM_TEST_CHECK_DETERMINISM=1
scripts/simtest/cargo-simtest simtest \
Expand All @@ -85,7 +106,12 @@ scripts/simtest/cargo-simtest simtest \
--profile simtestnightly \
-E "$TEST_FILTER" 2>&1 | tee "$LOG_FILE"

echo ""
echo "============================================="
echo "All tests completed, checking for failures..."
echo "============================================="
date

grep -qHn FAIL "$LOG_DIR"/*

# if grep found no failures exit now
Expand Down

0 comments on commit 03dc1da

Please sign in to comment.