Linux + hardhat + hardhat-network-helpers setup deterministically panics

[n3xt-auz]

Using the following versions

"devDependencies": {
    "@nomicfoundation/hardhat-network-helpers": "^1.0.11",
    "@nomicfoundation/hardhat-toolbox": "^5.0.0",
    "@nomiclabs/hardhat-ethers": "^2.2.3",
    "@openzeppelin/contracts": "^5.0.2",
    "@openzeppelin/contracts-upgradeable": "^5.0.2",
    "@openzeppelin/hardhat-upgrades": "^3.2.0",
    "@typechain/ethers-v6": "^0.5.1",
    "@typechain/hardhat": "^9.1.0",
    "ethers6": "npm:ethers@6",
    "hardhat": "^2.22.6",
    "hardhat-deploy": "^0.12.4",
    "hardhat-deploy-ethers": "^0.4.2",
    "hardhat-gas-reporter": "^1.0.8",
    "solidity-coverage": "^0.8.12",
    "tsx": "^4.16.2",
    "typechain": "^8.3.2"
  }

with a pretty straightforward Hardhat configuration:

import 'hardhat-deploy';
import 'hardhat-deploy-ethers';
import '@nomicfoundation/hardhat-toolbox';
import '@openzeppelin/hardhat-upgrades';

import { HardhatUserConfig } from 'hardhat/config';
import { accounts, namedAccounts } from './test-wallets';

const config: HardhatUserConfig = {
  defaultNetwork: 'hardhat',
  typechain: {
    discriminateTypes: true,
    outDir: './src/typechain-types',
    node16Modules: false,
    target: 'ethers-v6',
  },
  solidity: {
    compilers: [{ version: '0.8.20', settings: {} }],
  },
  namedAccounts: namedAccounts,
  networks: {
    hardhat: {
      mining: {
        auto: true,
        interval: 1000,
      },
      throwOnTransactionFailures: true,
      throwOnCallFailures: true,
      allowUnlimitedContractSize: true,
      blockGasLimit: 30000000,
      gas: 12450000,
      initialBaseFeePerGas: 30000000,
      gasPrice: 30000000,
      accounts: accounts,
    }
  },
};

export default config;

a simple ERC20 contract with Proxy and running some tests using the loadFixture() construct, I reliably get (100+ runs) the following panic:

# thread '<unnamed>' panicked at /build/crates/edr_provider/src/interval.rs:92:18:
# Failed to send cancellation signal: ()
# stack backtrace:
#    0:     0x7ff061931f85 - <unknown>
#    1:     0x7ff0619599fb - <unknown>
#    2:     0x7ff06192f3ef - <unknown>
#    3:     0x7ff061931d5e - <unknown>
#    4:     0x7ff061932e89 - <unknown>
#    5:     0x7ff061932bcd - <unknown>
#    6:     0x7ff061933323 - <unknown>
#    7:     0x7ff061933204 - <unknown>
#    8:     0x7ff061932449 - <unknown>
#    9:     0x7ff061932f37 - <unknown>
#   10:     0x7ff060e55973 - <unknown>
#   11:     0x7ff060e55e26 - <unknown>
#   12:     0x7ff061166d18 - <unknown>
#   13:     0x7ff061100e1a - <unknown>
#   14:     0x7ff06110045d - <unknown>
#   15:     0x7ff060feb6fa - <unknown>
#   16:           0xf15f6a - _ZN15node_napi_env__13CallFinalizerEPFvP10napi_env__PvS2_ES2_S2_
#   17:           0xeefddb - _ZThn40_N6v8impl9Reference8FinalizeEv
#   18:           0xf1a872 - _ZZN4node11Environment11CloseHandleI11uv_handle_sZN6v8impl12_GLOBAL__N_118ThreadSafeFunction26CloseHandlesAndMaybeDeleteEbEUlPS2_E_EEvPT_T0_ENUlS6_E_4_FUNES6_
#   19:          0x1e34661 - uv__finish_close
#                                at /home/iojs/build/ws/out/../deps/uv/src/unix/core.c:351:5
#   20:          0x1e34661 - uv__run_closing_handles
#                                at /home/iojs/build/ws/out/../deps/uv/src/unix/core.c:365:5
#   21:          0x1e34661 - uv_run
#                                at /home/iojs/build/ws/out/../deps/uv/src/unix/core.c:463:5
#   22:           0xeca0a0 - _ZN4node11Environment14CleanupHandlesEv
#   23:           0xeca15c - _ZN4node11Environment10RunCleanupEv
#   24:           0xe698d1 - _ZN4node15FreeEnvironmentEPNS_11EnvironmentE
#   25:           0xfc0c80 - _ZN4node16NodeMainInstance3RunEv
#   26:           0xf11fff - _ZN4node5StartEiPPc
#   27:     0x7ff09b029d90 - <unknown>
#   28:     0x7ff09b029e40 - __libc_start_main
#   29:           0xe62b9e - _start
#   30:                0x0 - <unknown>
# fatal runtime error: failed to initiate panic, error 5
# Subtest: e2e/TOKEN.test.ts
not ok 1 - e2e/TOKEN.test.ts
  ---
  duration_ms: 28517.016558
  location: '/home/runner/work/kimono/kimono/packages/blockchain/e2e/TOKEN.test.ts:1:1'
  failureType: 'testCodeFailure'
  exitCode: ~
  signal: 'SIGABRT'
  error: 'test failed'
  code: 'ERR_TEST_FAILURE'
  ...
1..5

The tests themselves run fine, all of them pass. I have tried running with jest and simple node:assert, always to the same outcome.

Note. I will have a public repo for reproduceability, but this may take me a little while.

[fvictorio]

Uhm, this is really interesting. Please let us know when you have a repro and we'll look into it.

[n3xt-auz]

@fvictorio please see https://github.com/n3xtio/repro-edr-panic for a minimal reproduction setup.

We consistently see the issue with the latest ubuntu runner in GH actions, as well as on a linux VM (DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS").
This is not a problem on Mac OS setups.

[fvictorio]

Thank you @n3xt-auz. Apparently, this is caused by the combination of interval mining and node:test. Removing interval mining, or using hh test (which uses mocha under the hood) seems to work fine.

More minimal repro here: https://github.com/fvictorio/edr-590

[n3xt-auz]

Thanks, we've "solved" this on our end to use an OSX runner for now as we rely on automining for our purposes. Appreciate your investigation and moving it to a "to do" 👍🏻 .