dev: load test the relayer fleet on staging [2]

[Eikix]

Describe the enhancement request

load test the relayer fleet on staging and thus include in the test strategy the ability to assert whether the relayer fleet is robust and resilient to increased load on the testnet.

in the current version of kakarot, relayers are susceptible to get their nonces de-sync'd (though I still don't understand in which case they get their nonce out of sync with Starknet Sepolia), and they reset their nonce every 1 minute with the real state of the chain.

Describe Preferred Solution

add to test strategy on staging environment (and prod?) the ability to spam the chain and check whether relayer fleet holds the load.

WHEN I launch a staging integration test suite, THEN I am able to assess that the relayer fleet is robust towards load. That means that with extremely high probability, relayer transactions are neither rejected nor slowed down by nonce errors. This probability must not become lower with increased load.

---

Context

worst case scenario: Let X be the time in second it takes for the relayer fleet to heal itself from having bad nonces. Then, when Kakarot Sepolia is under heavy load, the average time to relayer successfully a transaction is X.
this happens IF under heavy load, all relayers tend to "corrupt" their nonce. Then, transactions get picked up in the mempool and with high probability are rejected until the nonces "hashmap" of the RPC service is healed.

desired outcome: WHEN I submit an EVM transaction with Kakarot RPC, THEN with >99% probability, the relayed Starknet transaction that wraps my EVM transaction is NOT rejected. If it is rejected, then under a second, my transaction is sent again by another relayer and the corrupted relayer is healed. An alarm is set off

[greged93]

Investigate stress testing tools for EVM chains.

Duration: 2 hours