feat(tests): port ethereum/tests test cases (tracking issue)

[danceratopz]

This is a top-level issue for tracking the manual porting of ethereum/tests test cases to execution-spec-tests:

Contributions to manually port individual test cases are very welcome!

If you're already familiar with ethereum/tests and execution-spec-tests, jump to the issue tracker below!

Background

ethereum/tests was the repository used to define EVM test cases from Frontier up to and including The Merge. The test cases are defined as YAML (and sometimes JSON) files in the ./src/ sub-directory and the JSON "test fixtures" (fully-populated tests that can be executed against clients) are generated using ethereum/retesteth. These JSON artifacts are regenerated when required and added to the repository, for example, in the ./GeneralStateTests/ sub-directory.

From Shanghai on, new test cases (in particular for new features included in hard-forks) have been defined in Python in this repository (execution-spec-tests). The existing test cases are still important to execute against clients, however, and porting ethereum/tests to execution-spec-tests ensures that the test cases can be continually maintained and filled (generated) for newer forks. It also has the advantage that client teams will only need to obtain test fixture releases from one source.

Automated Port to Python

Although there will be an effort to automate ("transpile") the remaining test cases from YAML (respectively JSON) to Python; contributions to manually port individual test cases can be very beneficial and are welcome!

The benefits of manual porting are:

1. Reduce the number of test cases by combing multiple YAML (resp. JSON) test cases in one Python test function (via parametrization).
2. Potentially improve coverage by parametrizing the Python version.
3. Code/docstrings are likely higher quality than an automated conversion.
4. The test can be better organized within the ./tests folder in execution-spec-tests by the fork and the EIP in which the functionality was introduced.

Process

1. Pick a single or a set of YAML test cases in ethereum/tests to port and create an issue in this repository - optional, but definitely recommended if porting tests from the tracker below to avoid duplicate work.
2. Create a PR with the ported tests:
   a. Add the list of ported YAML files to converted-ethereum-tests.txt.
   b. If the tests can't currently be filled, please explain the issue (feel free to also open a Discussion).

Help with ethereum/tests

The test formats are described in the ethereum/tests online docs. Note that state filler test cases in YAML can be parametrized, see the docs here.

Hints to Get Started with ethereum/execution-spec-tests

0. Start here: Getting Started → Installation.
1. Use uv run et make test to create a template test module (currently a WIP in ✨ feat(et): add et make test for interactively creating new tests #950).
2. Where to put the test? See how tests are organized in Writing Tests → Adding a New Test.
3. State vs Blockchain test? If in doubt try to write a state test, see Writing Tests → Types of Tests → Deciding on a Test Type.
4. Tests are filled via EELS by default (no further installation necessary; it's a Python dependency and got installed in Step 0), see Filling → Getting Started.
5. Parametrizing test functions is encouraged, see Writing Tests → Writing a New Test → Parametrizing Tests.
6. The Opcode mini-lang defined in this repository is the preferred way to define bytecode in tests (if feasible). For a very simple example, see test_chainid() - there are other examples in the tests.

Feel free to reach out for help or guidance in the Ethereum R&D discord or via DM; see Getting Help in the docs.

Test Cases

Preview Give feedback

1. Add test cases for any single opcode (porting an entire test case from ethereum/tests can be overwhelming; instead add your own test cases for a single opcode from https://www.evm.codes/).
2. src/GeneralStateTestsFiller/VMTests/vmTests - swapFiller.yml - WIP: feat(tests): SwapFiller.yml  #1012
3. src/GeneralStateTestsFiller/VMTests/vmTests - dupFiller.yml.
4. src/GeneralStateTestsFiller/VMTests/vmTests - pushFiller.yml - WIP: ✨feat(tests): PUSH* opcodes #974
5. src/GeneralStateTestsFiller/VMTests/vmTests (Note: DUP already ported).
6. src/GeneralStateTestsFiller/stLogTests/.
7. src/GeneralStateTestsFiller/stRandom and src/GeneralStateTestsFiller/stRandom2 (see Easy to convert test from ethereum/tests  #1016 for details)
8. src/GeneralStateTestsFiller/stZeroKnowledge and src/GeneralStateTestsFiller/stZeroKnowledge2 (see Easy to convert test from ethereum/tests  #1016 for details)
9. src/GeneralStateTestsFiller/stPreCompiledContracts1 and src/GeneralStateTestsFiller/stPreCompiledContracts2 (see Easy to convert test from ethereum/tests  #1016 for details)
10. src/GeneralStateTestsFiller/stQuadraticComplexityTest (see Easy to convert test from ethereum/tests  #1016 for details)
11. src/GeneralStateTestsFiller/stSStoreTest (see Easy to convert test from ethereum/tests  #1016 for details)
12. src/GeneralStateTestsFiller/stTimeConsuming (see Easy to convert test from ethereum/tests  #1016 for details)
13. src/GeneralStateTestsFiller/stWalletTest (see Easy to convert test from ethereum/tests  #1016 for details)
14. src/GeneralStateTestsFiller/stCallCodes, src/GeneralStateTestsFiller/stCallDelegateCodesCallCodeHomestead and src/GeneralStateTestsFiller/stCallDelegateCodesHomestead (see Easy to convert test from ethereum/tests  #1016 for details)
15. src/GeneralStateTestsFiller/stEIP150singleCodeGasPrices (see Easy to convert test from ethereum/tests  #1016 for details)

Some Examples of Ported Tests

ethereum/tests	ethereum/execution-spec-tests
src/GeneralStateTestsFiller/VMTests/vmTests/dupFiller.yml	tests/frontier/opcodes/test_dup.py
GeneralStateTests/stCreate2/call_outsize_then_create2_successful_then_returndatasize.json, GeneralStateTests/stCreate2/call_then_create2_successful_then_returndatasize.json	tests/constantinople/eip1014_create2/test_create_returndata.py

[winsvega]

I still remain quite skeptical about automated port. I would rather focus on rewriting the tests. there are not so many if you take combinations and vector tests (precompile inputs) and random tests out. (which are easy to do in python too)
then remaining are either simple cases or heavy cases.

[pacrob]

I'll take porting swapFiller.yml if nobody's working on it yet. NM, Looking for a better option 😁

OK, how about the 3 calldata tests - copy, load, and size? @danceratopz

[bomanaps]

Please can I take on swapFiller.yml? @danceratopz

[danceratopz]

Please can I take on swapFiller.yml? @danceratopz

Yes, @bomanaps! Please make an issue when you get started and I can link it in the tracker here to avoid anyone else working on it :)

[Giszter]

Hi @winsvega Sorry I haven't written in so long, but I got sick and then Christmas..As I see from docs the best idea will be to start with tag: 'good first issue'. In this case can I take for example: https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller/stEIP150singleCodeGasPrices or maybe rc/GeneralStateTestsFiller/stRandom and src/GeneralStateTestsFiller/stRandom2? Or maybe there is something more important ?
ps. Happy New Year ;)

[winsvega]

That seems to only check the gas price of individual opcodes. That should be already covered in pyspec.

Take a look at #1016

[Giszter]

That seems to only check the gas price of individual opcodes. That should be already covered in pyspec.

Take a look at #1016

So maybe this one stWalletTest
?

[reedsa]

@danceratopz @winsvega If the Zero Knowledge and/or Precompiled Contracts are available, I'll dig into those.

Edit: Focusing solely on precompiled contracts. ZK tests can be worked by someone else or I can circle back once I've nailed things down.

[winsvega]

That seems to only check the gas price of individual opcodes. That should be already covered in pyspec.

Take a look at #1016

So maybe this one stWalletTest
?

stWallet test is not an evm test. This is a legacy test for a smart contract, I just kept it as a legacy, and it probably doesn't cover anything evm related.

[Giszter]

hm so in this case @winsvega can you suggest which one I can take? maybe it will be easier to choose correct one ;)

[winsvega]

StRandom