simulated.Backend leaks goroutines

[topliceanu]

System information

Geth version: v1.14.12
OS & Version: MacOS

Expected behaviour

I'm in the process of writing tests that use the simulated backend and I want to make sure that my code does not leak goroutines, so I'm using uber/goleak.

I noticed that uber/goleak reports that the simulated backend leaks goroutines.

Given this test:

package main

import (
        "testing"

        "github.com/ethereum/go-ethereum/core/types"
        "github.com/ethereum/go-ethereum/ethclient/simulated"
        "go.uber.org/goleak"
)

func TestSimulatedBackendLeak(t *testing.T) {
        defer goleak.VerifyNone(t)
        genesisData := types.GenesisAlloc{}
        simulatedBackend := simulated.NewBackend(genesisData)
        defer simulatedBackend.Close()
}

and the latest stable versions of go-ethereum and goleak at the time of writing:

go 1.23.2

require (
        github.com/ethereum/go-ethereum v1.14.12
        go.uber.org/goleak v1.3.0
)

I expected no leaked goroutines to be reported by goleak.

Actual behaviour

I'm getting an error from goleak.
I expected that when Close() returns, all the goroutines started by the simulated backend are terminated. Instead I get this stacktrace:

✗ go test main_test.go
--- FAIL: TestSimulatedBackendLeak (0.45s)
    main_test.go:16: found unexpected goroutines:
        [Goroutine 3 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 4 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 5 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 6 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 7 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 8 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 9 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 10 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 11 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 12 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 13 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 14 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 15 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 16 in state chan receive, with github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache on top of the stack:
        github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:63 +0x34
        created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/core/sender_cacher.go:55 +0x78
         Goroutine 29 in state chan receive, with github.com/ethereum/go-ethereum/eth/gasprice.NewOracle.func1 on top of the stack:
        github.com/ethereum/go-ethereum/eth/gasprice.NewOracle.func1()
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/eth/gasprice/gasprice.go:126 +0x78
        created by github.com/ethereum/go-ethereum/eth/gasprice.NewOracle in goroutine 51
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/eth/gasprice/gasprice.go:124 +0x8f0
         Goroutine 33 in state chan receive, with github.com/ethereum/go-ethereum/eth/filters.(*FilterAPI).timeoutLoop on top of the stack:
        github.com/ethereum/go-ethereum/eth/filters.(*FilterAPI).timeoutLoop(0x14000624750, 0x0?)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/eth/filters/api.go:93 +0x70
        created by github.com/ethereum/go-ethereum/eth/filters.NewFilterAPI in goroutine 51
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/ethereum/[email protected]/eth/filters/api.go:81 +0xe0
         Goroutine 99 in state select, with github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain on top of the stack:
        github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x14000603180)
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_state.go:110 +0xe8
        created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 51
        	/Users/alexandrutopliceanu/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:149 +0x3cc
        ]
FAIL
FAIL	command-line-arguments	0.875s
FAIL

Steps to reproduce the behaviour

I prepared a small repo showcasing the issue, https://github.com/topliceanu/simulatedbackendleak.
Run it with go test main_test.go.

Backtrace

See above 🆙

When submitting logs: please submit them as text and not screenshots.

[jwasinger]

I can see that a good number of these "leaking" routines are because of the SenderCacher which runs signature recovery in parallel on background goroutines and is instantiated as a global variable. However, I'm not sure about the others. Continuing to investigate...