✨(gentest): make gentest great

[danceratopz]

The gentest CLI is a really nice tool that can generate and write a Python (pytest) test module to disk given a type 0 transaction hash on a live network. This generates a state_test from the mainnet transaction 0xa41f...06be:

uv run gentest 0xa41f343be7a150b740e5c939fa4d89f3a2850dbe21715df96b612fc20d1906be tests/paris/test_0xa41f.py

Currently, due to its limited transaction type support (type 0 only), it's of limited use. This meta ticket is to track progress of making gentest an easy-to-use CLI tool that can generate test cases for specific issues observed on devnets, and potentially, to allow benchmarking of an EVM's transaction processing using blockchain_tests created from real mainnet data.

While working on the issues below, we should consider and try to ensure that gentest also works on Verkle and EOF testnets.

gentest Tasks

Preview Give feedback

1. ✨(gentest): Better gentest specific configuration #893
   1 of 1
   scope:gentest type:feat +2
2. ✨(gentest): allow custom headers in gentest config file and add an example config file #891
   scope:gentest type:feat +2
3. 🔀(gentest): use jinja2 templates to structure python output #863
   scope:gentest type:refactor +2
4. 🐞(gentest): generate test cases as a state_test for single transactions #864
   scope:gentest type:bug +2
5. ✨(gentest): ensure the generated python test module passes tox checks #859
   scope:gentest type:feat +2
6. 🧪(gentest): add testing to ci for the gentest cli #860
   scope:gentest type:test +2
7. ✨(gentest): add support for other transaction types #861
   scope:gentest type:feat +2
8. ✨(gentest): use evm_bytes to generate python opcode in tests #862
   scope:gentest type:feat +2
9. ✨(gentest): create blockchain_test test cases from multiple transaction hashes or block hash #865
   scope:gentest type:feat +2
10. ✨(gentest): Add support for chain_id #908
    scope:gentest type:feat +2

[danceratopz]

@raxhvl said he's happy to pick this up! ☺️

Thanks @raxhvl! Above I tried to outline the most obvious tasks for gentest. Please see these as suggestions - if you have any other ideas or inputs, just go for it! I'm also happy to jump in and help out if need be.

[raxhvl]

Thanks @danceratopz I will comment on individual issues as I start working on them.

[danceratopz]

The testinit command has now become et make test - the tasks related to make test are now tracked here: #973

[chfast]

Hey there, I've just become aware of this source of execution data: https://ethpandaops.io/data/xatu/schema/canonical_execution_/#canonical_execution_storage_reads
It may be easier to access these instead of going through a node RPC.

[raxhvl]

Thanks @chfast!

[danceratopz]

Hey there, I've just become aware of this source of execution data: https://ethpandaops.io/data/xatu/schema/canonical_execution_/#canonical_execution_storage_reads It may be easier to access these instead of going through a node RPC.

Thanks @chfast, this is a great idea! I think we'll have to go with RPC support for now, as xatu is unlikely to add devnet support to the database. But it is definitely something we can consider down the line once gentest is more feature complete.

[chfast]

I've done some research on this, and I've found some issues:

• they use some specific data format that I'm not familiar with, but it is not JSON
• there is no data for mainnet (at least not in all categories); e.g. there is not data about storage.

So in general, this is probably not good data source.