Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Add more sims test scenarios #23278

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+319 −173
Open

test: Add more sims test scenarios #23278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
71c45a7
More sims tests
alpe Jan 9, 2025
c29ec4d
Merge branch 'main' into alex/sims2_v2_dev_tests
aljo242 Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 28 additions & 33 deletions scripts/build/simulations.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@

#? test-sim-nondeterminism: Run non-determinism test for simapp
test-sim-nondeterminism:
# @echo "Running non-determinism test..."
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout=30m -tags='sims' -run TestAppStateDeterminism \
# -NumBlocks=100 -BlockSize=200 -Period=0
@echo "Running non-determinism test..."
@cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -timeout=30m -tags='sims' -run TestAppStateDeterminism \
-NumBlocks=100 -BlockSize=200


# Requires an exported plugin. See store/streaming/README.md for documentation.
#
Expand All @@ -18,51 +19,45 @@ test-sim-nondeterminism:
test-sim-nondeterminism-streaming:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we planning to bring these tests back as well?

# @echo "Running non-determinism-streaming test..."
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout=30m -tags='sims' -run TestAppStateDeterminism \
# -NumBlocks=100 -BlockSize=200 -Period=0 -EnableStreaming=true
# -NumBlocks=100 -BlockSize=200 -EnableStreaming=true

test-sim-custom-genesis-fast:
# @echo "Running custom genesis simulation..."
# @echo "By default, ${HOME}/.simapp/config/genesis.json will be used."
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout=30m -tags='sims' -run TestFullAppSimulation -Genesis=${HOME}/.simapp/config/genesis.json \
# -NumBlocks=100 -BlockSize=200 -Seed=99 -Period=5 -SigverifyTx=false
# -NumBlocks=100 -BlockSize=200 -Seed=99 -SigverifyTx=false

test-sim-import-export:
# @echo "Running application import/export simulation. This may take several minutes..."
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout 20m -tags='sims' -run TestAppImportExport \
# -NumBlocks=50 -Period=5
# @echo "Running application import/export simulation. This may take several minutes..."
# @cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -timeout 20m -tags='sims' -run TestAppImportExport \
# -NumBlocks=50

test-sim-after-import:
# @echo "Running application simulation-after-import. This may take several minutes..."
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout 30m -tags='sims' -run TestAppSimulationAfterImport \
# -NumBlocks=50 -Period=5
@echo "Running application simulation-after-import. This may take several minutes..."
@cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -timeout 30m -tags='sims' -run TestAppSimulationAfterImport \
-NumBlocks=50

test-sim-custom-genesis-multi-seed:
# @echo "Running multi-seed custom genesis simulation..."
# @echo "By default, ${HOME}/.simapp/config/genesis.json will be used."
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout 30m -tags='sims' -run TestFullAppSimulation -Genesis=${HOME}/.simapp/config/genesis.json \
# -NumBlocks=400 -Period=5
# @cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -timeout 30m -tags='sims' -run TestFullAppSimulation -Genesis=${HOME}/.simapp/config/genesis.json \
# -NumBlocks=400

test-sim-multi-seed-long:
# @echo "Running long multi-seed application simulation. This may take awhile!"
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout=2h -tags='sims' -run TestFullAppSimulation \
# -NumBlocks=150 -Period=50

test-sim-multi-seed-short: test-v2-sim
# @echo "Running short multi-seed application simulation. This may take awhile!"
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -timeout 30m -tags='sims' -run TestFullAppSimulation \
# -NumBlocks=50 -Period=10 -FauxMerkle=true
@echo "Running long multi-seed application simulation. This may take awhile!"
@cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -timeout=2h -tags='sims' -run TestFullAppSimulation \
-NumBlocks=150

.Phony: test-v2-sim
test-v2-sim:
@echo "Running short multi-seed application simulation. This may take awhile!"
@cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -timeout 30m -tags='sims' -run TestSimsAppV2 \
# -NumBlocks=50 -Period=10 -FauxMerkle=true
test-sim-multi-seed-short:
@echo "Running short multi-seed application simulation. This may take awhile!"
@cd ${CURRENT_DIR}/simapp/v2 && go test -failfast -mod=readonly -timeout 30m -tags='sims' -run TestFullAppSimulation \
-NumBlocks=50

test-sim-benchmark-invariants:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this as invariants have been removed

# @echo "Running simulation invariant benchmarks..."
# cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -benchmem -bench=BenchmarkInvariants -tags='sims' -run=^$ \
# -Enabled=true -NumBlocks=1000 -BlockSize=200 \
# -Period=1 -Commit=true -Seed=57 -v -timeout 24h
# -Commit=true -Seed=57 -v -timeout 24h

.PHONY: \
test-sim-nondeterminism \
Expand All @@ -81,20 +76,20 @@ SIM_COMMIT ?= true

#? test-sim-fuzz: Run fuzz test for simapp
test-sim-fuzz:
@echo "Running application fuzz for numBlocks=2, blockSize=20. This may take awhile!"
# @echo "Running application fuzz for numBlocks=2, blockSize=20. This may take awhile!"
#ld flags are a quick fix to make it work on current osx
@cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -json -tags='sims' -ldflags="-extldflags=-Wl,-ld_classic" -timeout=60m -fuzztime=60m -run=^$$ -fuzz=FuzzFullAppSimulation -GenesisTime=1714720615 -NumBlocks=2 -BlockSize=20
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -json -tags='sims' -ldflags="-extldflags=-Wl,-ld_classic" -timeout=60m -fuzztime=60m -run=^$$ -fuzz=FuzzFullAppSimulation -GenesisTime=1714720615 -NumBlocks=2 -BlockSize=20

#? test-sim-benchmark: Run benchmark test for simapp
test-sim-benchmark:
@echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
@cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -tags='sims' -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
# @echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -tags='sims' -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
-Enabled=true -NumBlocks=$(SIM_NUM_BLOCKS) -BlockSize=$(SIM_BLOCK_SIZE) -Commit=$(SIM_COMMIT) -timeout 24h


test-sim-profile:
@echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
@cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -tags='sims' -benchmem -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
# @echo "Running application benchmark for numBlocks=$(SIM_NUM_BLOCKS), blockSize=$(SIM_BLOCK_SIZE). This may take awhile!"
# @cd ${CURRENT_DIR}/simapp && go test -failfast -mod=readonly -tags='sims' -benchmem -run=^$$ $(.) -bench ^BenchmarkFullAppSimulation$$ \
-Enabled=true -NumBlocks=$(SIM_NUM_BLOCKS) -BlockSize=$(SIM_BLOCK_SIZE) -Commit=$(SIM_COMMIT) -timeout 24h -cpuprofile cpu.out -memprofile mem.out

.PHONY: test-sim-profile test-sim-benchmark test-sim-fuzz
Expand Down
Loading
Loading