[Bug]: Node AppHash'ing after v19.x.x upgrade

[a26nine]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Our cosmoshub-4 archive nodes stopped progressing after the v19 upgrade. So, we downloaded the archive snapshot from QuickSync. The nodes progressed smoothly for a while, but then it AppHash'd. We waited for a few days and downloaded another snapshot from the same source, and the results were same again; the node AppHash'd after some time. Once more, we waited for a few days for a new snapshot, got it, and got AppHash'd again.

The most recent AppHash happened on v19.2.0:

Sep 18 08:42:30 cosmovisor[7778]: 8:42AM ERR Error in validation err="wrong Block.Header.AppHash.  Expected FD8867280ABAD4DF361A7285B14822FC3ABE20CB4BDF8B87DDDF152C9A772450, got A4E3A67B70A3FF6C5C6EA6C9B2BDA694E886C6D2B1710F51182843C5AA5A887B" module=blocksync

I am not sure who/what is the culprit here—the snapshot, the binary, or something else?

We rolled back a few times and cleared the wasm directory before starting gaiad. We also tried running with the pre-built binaries supplied in the Releases section. But, none of it helped.

Our build process:

• Clone https://github.com/cosmos/gaia
• Build & Install it with make install (Current go version is 1.22.6)

Long Version:

./gaiad version --long
build_deps:
- cloud.google.com/[email protected]
- cloud.google.com/go/[email protected]
- cloud.google.com/go/auth/[email protected]
- cloud.google.com/go/compute/[email protected]
- cloud.google.com/go/[email protected]
- cloud.google.com/go/[email protected]
- cosmossdk.io/[email protected] => github.com/informalsystems/cosmos-sdk/[email protected]
- cosmossdk.io/client/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/[email protected]
- cosmossdk.io/tools/[email protected]
- cosmossdk.io/tools/[email protected]
- cosmossdk.io/x/[email protected]
- cosmossdk.io/x/[email protected]
- cosmossdk.io/x/[email protected]
- cosmossdk.io/x/[email protected]
- cosmossdk.io/x/[email protected]
- cosmossdk.io/x/[email protected]
- filippo.io/[email protected]
- github.com/99designs/[email protected] => github.com/cosmos/[email protected]
- github.com/CosmWasm/[email protected]
- github.com/CosmWasm/wasmvm/[email protected]
- github.com/DataDog/[email protected]+incompatible
- github.com/aws/[email protected]
- github.com/beorn7/[email protected]
- github.com/bgentry/[email protected]
- github.com/bgentry/[email protected]
- github.com/bits-and-blooms/[email protected]
- github.com/btcsuite/btcd/btcec/[email protected]
- github.com/cenkalti/backoff/[email protected]
- github.com/cespare/xxhash/[email protected]
- github.com/chzyer/[email protected]
- github.com/cockroachdb/apd/[email protected]
- github.com/cockroachdb/[email protected]
- github.com/cockroachdb/[email protected]
- github.com/cockroachdb/[email protected]
- github.com/coinbase/[email protected]
- github.com/cometbft/[email protected]
- github.com/cometbft/[email protected]
- github.com/cosmos/[email protected]
- github.com/cosmos/[email protected]
- github.com/cosmos/[email protected]
- github.com/cosmos/[email protected] => github.com/cosmos/[email protected]
- github.com/cosmos/[email protected]
- github.com/cosmos/[email protected]
- github.com/cosmos/[email protected]
- github.com/cosmos/[email protected]
- github.com/cosmos/ibc-apps/middleware/packet-forward-middleware/[email protected]
- github.com/cosmos/ibc-apps/modules/rate-limiting/[email protected]
- github.com/cosmos/ibc-go/modules/[email protected]
- github.com/cosmos/ibc-go/[email protected]
- github.com/cosmos/ics23/[email protected]
- github.com/cosmos/interchain-security/[email protected]
- github.com/cosmos/[email protected]
- github.com/creachadair/[email protected]
- github.com/creachadair/[email protected]
- github.com/davecgh/[email protected]
- github.com/decred/dcrd/dcrec/secp256k1/[email protected]
- github.com/desertbit/[email protected]
- github.com/distribution/[email protected]
- github.com/dvsekhvalnov/[email protected]
- github.com/emicklei/[email protected]
- github.com/fatih/[email protected]
- github.com/felixge/[email protected]
- github.com/fsnotify/[email protected]
- github.com/getsentry/[email protected]
- github.com/go-kit/[email protected]
- github.com/go-kit/[email protected]
- github.com/go-logfmt/[email protected]
- github.com/go-logr/[email protected]
- github.com/go-logr/[email protected]
- github.com/godbus/[email protected]
- github.com/gogo/[email protected]
- github.com/gogo/[email protected]
- github.com/golang/[email protected]
- github.com/golang/[email protected]
- github.com/golang/[email protected]
- github.com/golang/[email protected]
- github.com/google/[email protected]
- github.com/google/[email protected]
- github.com/google/[email protected]
- github.com/google/[email protected]
- github.com/google/[email protected]
- github.com/google/[email protected]
- github.com/googleapis/[email protected]
- github.com/googleapis/gax-go/[email protected]
- github.com/gorilla/[email protected]
- github.com/gorilla/[email protected]
- github.com/gorilla/[email protected]
- github.com/grpc-ecosystem/[email protected]
- github.com/grpc-ecosystem/[email protected]
- github.com/gsterjov/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/golang-lru/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hashicorp/[email protected]
- github.com/hdevalence/[email protected]
- github.com/huandu/[email protected]
- github.com/iancoleman/[email protected]
- github.com/iancoleman/[email protected]
- github.com/improbable-eng/[email protected]
- github.com/jmespath/[email protected]
- github.com/klauspost/[email protected]
- github.com/kr/[email protected]
- github.com/kr/[email protected]
- github.com/lib/[email protected]
- github.com/magiconair/[email protected]
- github.com/manifoldco/[email protected]
- github.com/mattn/[email protected]
- github.com/mattn/[email protected]
- github.com/minio/[email protected]
- github.com/mitchellh/[email protected]
- github.com/mitchellh/[email protected]
- github.com/mitchellh/[email protected]
- github.com/mtibben/[email protected]
- github.com/oasisprotocol/[email protected]
- github.com/oklog/[email protected]
- github.com/opencontainers/[email protected]
- github.com/pelletier/go-toml/[email protected]
- github.com/pkg/[email protected]
- github.com/pmezard/[email protected]
- github.com/prometheus/[email protected]
- github.com/prometheus/[email protected]
- github.com/prometheus/[email protected]
- github.com/prometheus/[email protected]
- github.com/rakyll/[email protected]
- github.com/rcrowley/[email protected]
- github.com/rogpeppe/[email protected]
- github.com/rs/[email protected]
- github.com/rs/[email protected]
- github.com/sagikazarmark/[email protected]
- github.com/skip-mev/[email protected]
- github.com/spf13/[email protected]
- github.com/spf13/[email protected]
- github.com/spf13/[email protected]
- github.com/spf13/[email protected]
- github.com/spf13/[email protected]
- github.com/stretchr/[email protected]
- github.com/subosito/[email protected]
- github.com/syndtr/[email protected] => github.com/syndtr/[email protected]
- github.com/tendermint/[email protected]
- github.com/tidwall/[email protected]
- github.com/ulikunitz/[email protected]
- [email protected]
- go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]
- go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]
- go.opentelemetry.io/[email protected]
- go.opentelemetry.io/otel/[email protected]
- go.opentelemetry.io/otel/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- golang.org/x/[email protected]
- google.golang.org/[email protected]
- google.golang.org/[email protected]
- google.golang.org/genproto/googleapis/[email protected]
- google.golang.org/genproto/googleapis/[email protected]
- google.golang.org/[email protected]
- google.golang.org/[email protected]
- gopkg.in/[email protected]
- gopkg.in/[email protected]
- gopkg.in/[email protected]
- gotest.tools/[email protected]
- nhooyr.io/[email protected]
- pgregory.net/[email protected]
- sigs.k8s.io/[email protected]
build_tags: netgo
commit: 1d52b3d434d5d78561c3628ef351b88890ec7f7f
cosmos_sdk_version: v0.50.9-lsm
go: go version go1.22.6 linux/amd64
name: gaia
server_name: gaiad
version: v19.2.0

Gaia Version

v19.2.0

How to reproduce?

• Download archive snapshot from QuickSync
• Unpack it
• Build & Install relevant gaiad binary
• Start gaiad

The node will AppHash after some time.

[MSalopek]

Thank you for raising this concern, I'm sorry you are facing issues.

Have you tried any other node (default, pruned) from quicksync?

We can try and get in contact with quicksync and help them debug.

It is weird if this is happening frequently but we have had reports of apphashes tied to wasm directories ever since we have introduced cosmwasm.

If possible, it would be beneficial to try and replicate this on a smaller node (to reduce debugging times). If this issue happens with other quicksync snapshots but not on polkachu or nodestake it could point to a slight misconfiguration in quicksync's export procedure that can be mitigated.

[a26nine]

Thank you for raising this concern, I'm sorry you are facing issues.

Have you tried any other node (default, pruned) from quicksync?

We can try and get in contact with quicksync and help them debug.

It is weird if this is happening frequently but we have had reports of apphashes tied to wasm directories ever since we have introduced cosmwasm.

If possible, it would be beneficial to try and replicate this on a smaller node (to reduce debugging times). If this issue happens with other quicksync snapshots but not on polkachu or nodestake it could point to a slight misconfiguration in quicksync's export procedure that can be mitigated.

I forgot to mention, we downloaded Polkachu's pruned snapshot, and it's running fine with the same binary without any issues.

[a26nine]

@MSalopek, did you get a chance to check with the QuickSync team?

[mayank-daga]

@MSalopek even i am getting similar issues

[MSalopek]

ChainLayer has been contacted. Updates will be posted as they reach us.

[MSalopek]

The issue seems to be solved on Quicksync's end.

Feel free to resync from the newest snapshot.

@mayank-daga @a26nine

[a26nine]

The issue seems to be solved on Quicksync's end.

Feel free to resync from the newest snapshot.

@mayank-daga @a26nine

No, it's not resolved. We are running pruned nodes for now.