[Bug]: After upgrading to v19.1.0 Node is throwing peer issue

[mayank-daga]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Hi
I am getting following logs after upgrading to v19.1.0

6:57AM INF starting node with ABCI CometBFT in-process module=server
6:57AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
6:57AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
6:57AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
6:57AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
6:57AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
6:57AM INF service start impl=EventBus module=events msg="Starting EventBus service"
6:57AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
7:03AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
7:03AM INF ABCI Handshake App Info hash=96977406B9BCC19F78614EAFB8E9FE6368A780668E51C586FADBCE1EA2CD58E7 height=22004611 module=consensus protocol-version=0 software-version=HEAD-4106e7a673da18b518fd81231a8e8b99bbf0fd0d
7:03AM INF ABCI Replay Blocks appHeight=22004611 module=consensus stateHeight=22004611 storeHeight=22004611
7:03AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=96977406B9BCC19F78614EAFB8E9FE6368A780668E51C586FADBCE1EA2CD58E7 appHeight=22004611 module=consensus
7:03AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.11
7:03AM INF This node is not a validator addr=46D05282E6580EBAD176AA6B5F287FC60F39977C module=consensus pubKey=PubKeyEd25519{9D82E4177952344DFB862689F4FCA8A7FF8E4015526C80AE5C959DB86FEE20EF}
7:03AM INF P2P Node ID ID=92ac45d8a7d21907382b832c57dc1201738b237c file=/gaia/.gaia/config/node_key.json module=p2p
7:03AM INF Adding persistent peers addrs=["[email protected]:26090","[email protected]:13456","[email protected]:14956","[email protected]:26656"] module=p2p
7:03AM INF Adding unconditional peer ids ids=[] module=p2p
7:03AM INF Add our address to book [email protected]:26656 book=/gaia/.gaia/config/addrbook.json module=p2p
7:03AM INF service start impl=Node module=server msg="Starting Node service"
7:03AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
7:03AM INF serve module=rpc-server msg="Starting RPC HTTP server on [::]:26657"
7:03AM INF service start impl=PEX module=pex msg="Starting PEX service"
7:03AM INF service start book=/gaia/.gaia/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
7:03AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
7:03AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
7:03AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
7:03AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
7:03AM INF Reactor module=consensus waitSync=true
7:03AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
7:03AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
7:03AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
7:03AM INF Saving AddrBook to file book=/gaia/.gaia/config/addrbook.json module=p2p size=1241
7:03AM INF service start impl="Peer{MConn{51.210.118.45:26656} a9380db8758f9b78cbe4253ccdacc992cf80f6f6 out}" module=p2p msg="Starting Peer service" [email protected]:26656
7:03AM INF service start impl=MConn{51.210.118.45:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
7:03AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{51.210.118.45:26656} module=p2p [email protected]:26656
7:03AM INF service stop impl=MConn{51.210.118.45:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
7:03AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{51.210.118.45:26656} a9380db8758f9b78cbe4253ccdacc992cf80f6f6 out}"
7:03AM INF service stop impl="Peer{MConn{51.210.118.45:26656} a9380db8758f9b78cbe4253ccdacc992cf80f6f6 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
7:03AM INF service start impl="Peer{MConn{176.9.101.44:26656} 505f4467926cdad29932c44dc5ea7a5da6982f48 out}" module=p2p msg="Starting Peer service" [email protected]:26656
7:03AM INF service start impl=MConn{176.9.101.44:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
7:03AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{176.9.101.44:26656} module=p2p [email protected]:26656
7:03AM INF service stop impl=MConn{176.9.101.44:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
7:03AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{176.9.101.44:26656} 505f4467926cdad29932c44dc5ea7a5da6982f48 out}"
7:03AM INF service stop impl="Peer{MConn{176.9.101.44:26656} 505f4467926cdad29932c44dc5ea7a5da6982f48 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
7:03AM INF service start impl="Peer{MConn{37.27.56.238:5000} e6d729935d77865bf79fc051b2a9d1a263f1fcdc out}" module=p2p msg="Starting Peer service" [email protected]:5000
7:03AM INF service start impl=MConn{37.27.56.238:5000} module=p2p msg="Starting MConnection service" [email protected]:5000
7:03AM INF service start impl="Peer{MConn{18.143.53.64:26656} c5ccf0aaf81e5e523a56bd676822c9f73abcb833 out}" module=p2p msg="Starting Peer service" [email protected]:26656
7:03AM INF service start impl=MConn{18.143.53.64:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
7:03AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{18.143.53.64:26656} module=p2p [email protected]:26656
7:03AM INF service stop impl=MConn{18.143.53.64:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
7:03AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{18.143.53.64:26656} c5ccf0aaf81e5e523a56bd676822c9f73abcb833 out}"
7:03AM INF service stop impl="Peer{MConn{18.143.53.64:26656} c5ccf0aaf81e5e523a56bd676822c9f73abcb833 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
7:03AM INF service start impl="Peer{MConn{46.4.80.48:14956} f527c58fb1c30eca8c4226edd7495c9bc2ec69b5 out}" module=p2p msg="Starting Peer service" [email protected]:14956
7:03AM INF service start impl=MConn{46.4.80.48:14956} module=p2p msg="Starting MConnection service" [email protected]:14956
7:03AM INF starting gRPC server... address=0.0.0.0:9090 module=grpc-server
7:03AM INF starting API server... address=tcp://0.0.0.0:1317 module=api-server
7:03AM INF serve module=api-server msg="Starting RPC HTTP server on [::]:1317"
7:03AM ERR Error dialing peer err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p
7:03AM INF Reconnecting to peer [email protected]:26090 module=p2p
7:03AM ERR Error dialing peer err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p
7:03AM INF Reconnecting to peer [email protected]:13456 module=p2p
7:03AM INF service start impl="Peer{MConn{192.69.220.226:26656} f956e0b6368d9712f173a8459679fd4413a3a3d1 out}" module=p2p msg="Starting Peer service" [email protected]:26656
7:03AM INF service start impl=MConn{192.69.220.226:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
7:03AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=0
7:03AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=0
7:03AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=1
7:03AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=1
7:03AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=2
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=2
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=3
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=3
7:04AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=3 numToDial=7
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=4
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=4
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=5
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=5
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=6
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=6
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=7
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=7
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=8
7:04AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=3 numToDial=7
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=8
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=9
7:04AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=9
7:04AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=10
7:05AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=11
7:05AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=10
7:05AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=12
7:05AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=11
7:05AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=3 numToDial=7
7:05AM INF service start impl="Peer{MConn{164.68.107.188:26656} cfd785a4224c7940e9a10f6c1ab24c343e923bec out}" module=p2p msg="Starting Peer service" [email protected]:26656
7:05AM INF service start impl=MConn{164.68.107.188:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
7:05AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{164.68.107.188:26656} module=p2p [email protected]:26656
7:05AM INF service stop impl=MConn{164.68.107.188:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
7:05AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{164.68.107.188:26656} cfd785a4224c7940e9a10f6c1ab24c343e923bec out}"
7:05AM INF service stop impl="Peer{MConn{164.68.107.188:26656} cfd785a4224c7940e9a10f6c1ab24c343e923bec out}" module=p2p msg="Stopping Peer service" [email protected]:26656
7:05AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=13
7:05AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=12
7:05AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=14
7:05AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=13
7:05AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=15
7:05AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=14
7:05AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=16
7:05AM INF Saving AddrBook to file book=/gaia/.gaia/config/addrbook.json module=p2p size=1241
7:05AM INF Ensure peers module=pex numDialing=1 numInPeers=0 numOutPeers=3 numToDial=6
7:05AM INF service start impl="Peer{MConn{74.118.140.203:26656} 66248ea4d70c4b8fd6c9ba848895a0332a3a345e out}" module=p2p msg="Starting Peer service" [email protected]:26656
7:05AM INF service start impl=MConn{74.118.140.203:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
7:05AM INF Error reconnecting to peer. Trying again [email protected]:26090 err="dial tcp 142.93.235.166:26090: i/o timeout" module=p2p tries=15
7:05AM ERR Error in validation err="wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=blocksync
7:05AM ERR Stopping peer for error err="reactor validation error: wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=p2p peer="Peer{MConn{74.118.140.203:26656} 66248ea4d70c4b8fd6c9ba848895a0332a3a345e out}"
7:05AM INF service stop impl=MConn{74.118.140.203:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
7:05AM INF service stop impl="Peer{MConn{74.118.140.203:26656} 66248ea4d70c4b8fd6c9ba848895a0332a3a345e out}" module=p2p msg="Stopping Peer service" [email protected]:26656

It synced some blocks and started giving this issue

Gaia Version

v19.1.0

How to reproduce?

No response

[mayank-daga]

HI @mpoke can you help here

[mayank-daga]

HI @mpoke just following up

[mayank-daga]

Hi just following if anyone can help

[mpoke]

Hi @mayank-daga. Sorry for the late reply. At the time you opened this issue, the Cosmos Hub was running Gaia v19.2.0. Since yesterday, the Hub is running Gaia v20.0.0. Please try again with these versions.

[mayank-daga]

Hi @mpoke i have tried with v19.2.0 and v20.0.0 but getting same issue i have added logs:

5:23AM INF starting node with ABCI CometBFT in-process module=server
5:24AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
5:24AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start impl=EventBus module=events msg="Starting EventBus service"
5:24AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
usa-dekalia@usazuadvpl00010:/NewDisk70Machine/node-atom$ sudo docker logs 5bc66246b5c4
5:23AM INF starting node with ABCI CometBFT in-process module=server
5:24AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
5:24AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start impl=EventBus module=events msg="Starting EventBus service"
5:24AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
usa-dekalia@usazuadvpl00010:/NewDisk70Machine/node-atom$ sudo docker logs 5bc66246b5c4
5:23AM INF starting node with ABCI CometBFT in-process module=server
5:24AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
5:24AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start impl=EventBus module=events msg="Starting EventBus service"
5:24AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
usa-dekalia@usazuadvpl00010:/NewDisk70Machine/node-atom$ sudo docker logs 5bc66246b5c4
5:23AM INF starting node with ABCI CometBFT in-process module=server
5:24AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
5:24AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
5:24AM INF service start impl=EventBus module=events msg="Starting EventBus service"
5:24AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
5:41AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
5:41AM INF ABCI Handshake App Info hash=96977406B9BCC19F78614EAFB8E9FE6368A780668E51C586FADBCE1EA2CD58E7 height=22004611 module=consensus protocol-version=0 software-version=v20.0.0
5:41AM INF ABCI Replay Blocks appHeight=22004611 module=consensus stateHeight=22004611 storeHeight=22004611
5:41AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=96977406B9BCC19F78614EAFB8E9FE6368A780668E51C586FADBCE1EA2CD58E7 appHeight=22004611 module=consensus
5:41AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.11
5:41AM INF This node is not a validator addr=46D05282E6580EBAD176AA6B5F287FC60F39977C module=consensus pubKey=PubKeyEd25519{9D82E4177952344DFB862689F4FCA8A7FF8E4015526C80AE5C959DB86FEE20EF}
5:41AM INF P2P Node ID ID=92ac45d8a7d21907382b832c57dc1201738b237c file=/gaia/.gaia/config/node_key.json module=p2p
5:41AM INF Adding persistent peers addrs=["[email protected]:14956","[email protected]:14956","[email protected]:13456","[email protected]:26656","[email protected]:26656","[email protected]:14956","[email protected]:26656","[email protected]:26656"] module=p2p
5:41AM INF Adding unconditional peer ids ids=[] module=p2p
5:41AM INF Add our address to book [email protected]:26656 book=/gaia/.gaia/config/addrbook.json module=p2p
5:41AM INF service start impl=Node module=server msg="Starting Node service"
5:41AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
5:41AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
5:41AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
5:41AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
5:41AM INF serve module=rpc-server msg="Starting RPC HTTP server on [::]:26657"
5:41AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
5:41AM INF Reactor module=consensus waitSync=true
5:41AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
5:41AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
5:41AM INF service start impl=PEX module=pex msg="Starting PEX service"
5:41AM INF service start book=/gaia/.gaia/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
5:41AM INF Saving AddrBook to file book=/gaia/.gaia/config/addrbook.json module=p2p size=1206
5:41AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
5:41AM INF service start impl="Peer{MConn{135.148.123.8:14956} 2441e90fcb341fcd5bebec15b54e346cdca64a9b out}" module=p2p msg="Starting Peer service" [email protected]:14956
5:41AM INF service start impl=MConn{135.148.123.8:14956} module=p2p msg="Starting MConnection service" [email protected]:14956
5:41AM INF service start impl="Peer{MConn{202.8.10.36:26656} 641ea77a7c3ddd8efa61f18d5d85acda6295aeca out}" module=p2p msg="Starting Peer service" [email protected]:26656
5:41AM INF service start impl=MConn{202.8.10.36:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
5:41AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{202.8.10.36:26656} module=p2p [email protected]:26656
5:41AM INF service stop impl=MConn{202.8.10.36:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
5:41AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{202.8.10.36:26656} 641ea77a7c3ddd8efa61f18d5d85acda6295aeca out}"
5:41AM INF service stop impl="Peer{MConn{202.8.10.36:26656} 641ea77a7c3ddd8efa61f18d5d85acda6295aeca out}" module=p2p msg="Stopping Peer service" [email protected]:26656
5:41AM INF service start impl="Peer{MConn{104.237.53.202:14956} 1af4a7c20e75edabd42bec9631d9cbde29dc0528 out}" module=p2p msg="Starting Peer service" [email protected]:14956
5:41AM INF service start impl=MConn{104.237.53.202:14956} module=p2p msg="Starting MConnection service" [email protected]:14956
5:41AM INF service start impl="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}" module=p2p msg="Starting Peer service" [email protected]:26656
5:41AM INF service start impl=MConn{54.177.181.170:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
5:41AM INF starting gRPC server... address=0.0.0.0:9090 module=grpc-server
5:41AM INF starting API server... address=tcp://0.0.0.0:1317 module=api-server
5:41AM INF serve module=api-server msg="Starting RPC HTTP server on [::]:1317"
5:41AM INF service start impl="Peer{MConn{165.22.236.46:26656} cab9590d22a12bf1cb310456f5a07e48803c71de out}" module=p2p msg="Starting Peer service" [email protected]:26656
5:41AM INF service start impl=MConn{165.22.236.46:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
5:41AM ERR Error dialing peer err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p
5:41AM INF Reconnecting to peer [email protected]:13456 module=p2p
5:41AM ERR Error in validation err="wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=blocksync
5:41AM ERR Stopping peer for error err="reactor validation error: wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=p2p peer="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}"
5:41AM INF service stop impl=MConn{54.177.181.170:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
5:41AM ERR Stopping peer for error err="read tcp 172.17.0.2:39186->54.177.181.170:26656: use of closed network connection" module=p2p peer="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}"
5:41AM INF service stop impl="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
5:41AM INF Reconnecting to peer [email protected]:26656 module=p2p
5:41AM ERR error while stopping peer error="already stopped" module=p2p
5:41AM INF service start impl="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}" module=p2p msg="Starting Peer service" [email protected]:26656
5:41AM INF service start impl=MConn{54.177.181.170:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
5:41AM INF service start impl="Peer{MConn{5.9.95.101:14956} def524b19130cbbc671ea6635d18f426f4afaa96 out}" module=p2p msg="Starting Peer service" [email protected]:14956
5:41AM INF service start impl=MConn{5.9.95.101:14956} module=p2p msg="Starting MConnection service" [email protected]:14956
5:41AM ERR Error dialing peer err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p
5:41AM INF Reconnecting to peer [email protected]:26656 module=p2p
5:41AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=0
5:41AM ERR Error dialing peer err="auth failure: secret conn failed: read tcp 172.17.0.2:42482->66.165.248.66:26656: i/o timeout" module=p2p
5:41AM INF Reconnecting to peer [email protected]:26656 module=p2p
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=0
5:41AM ERR Error dialing peer err="auth failure: secret conn failed: read tcp 172.17.0.2:57984->202.61.240.62:14956: i/o timeout" module=p2p
5:41AM INF Reconnecting to peer [email protected]:14956 module=p2p
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42512->66.165.248.66:26656: i/o timeout" module=p2p tries=0
5:41AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58008->202.61.240.62:14956: i/o timeout" module=p2p tries=0
5:41AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=1
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=1
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42534->66.165.248.66:26656: i/o timeout" module=p2p tries=1
5:41AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=2
5:41AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58030->202.61.240.62:14956: i/o timeout" module=p2p tries=1
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=2
5:41AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=3
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42560->66.165.248.66:26656: i/o timeout" module=p2p tries=2
5:41AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58068->202.61.240.62:14956: i/o timeout" module=p2p tries=2
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=3
5:41AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=4
5:41AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=5 numToDial=5
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=4
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42598->66.165.248.66:26656: i/o timeout" module=p2p tries=3
5:41AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58094->202.61.240.62:14956: i/o timeout" module=p2p tries=3
5:41AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=5
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=5
5:41AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=6
5:41AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42636->66.165.248.66:26656: i/o timeout" module=p2p tries=4
5:41AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58132->202.61.240.62:14956: i/o timeout" module=p2p tries=4
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=6
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=7
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=7
5:42AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58150->202.61.240.62:14956: i/o timeout" module=p2p tries=5
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42658->66.165.248.66:26656: i/o timeout" module=p2p tries=5
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=8
5:42AM INF Ensure peers module=pex numDialing=1 numInPeers=0 numOutPeers=5 numToDial=4
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=8
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42692->66.165.248.66:26656: i/o timeout" module=p2p tries=6
5:42AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58196->202.61.240.62:14956: i/o timeout" module=p2p tries=6
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=9
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=9
5:42AM ERR Error in validation err="wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=blocksync
5:42AM ERR Stopping peer for error err="reactor validation error: wrong Block.Header.LastResultsHash. Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451, got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=p2p peer="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}"
5:42AM INF service stop impl="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
5:42AM INF service stop impl=MConn{54.177.181.170:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
5:42AM INF Reconnecting to peer [email protected]:26656 module=p2p
5:42AM ERR failed to add block err="requested block #22004660 from [], not f05ddce65f1e75babe01d05fef1bce5d8ffe0972" module=blocksync peer="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}"
5:42AM INF service start impl="Peer{MConn{54.177.181.170:26656} f05ddce65f1e75babe01d05fef1bce5d8ffe0972 out}" module=p2p msg="Starting Peer service" [email protected]:26656
5:42AM INF service start impl=MConn{54.177.181.170:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42720->66.165.248.66:26656: i/o timeout" module=p2p tries=7
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=10
5:42AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58224->202.61.240.62:14956: i/o timeout" module=p2p tries=7
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=10
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=11
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42758->66.165.248.66:26656: i/o timeout" module=p2p tries=8
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=11
5:42AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58256->202.61.240.62:14956: i/o timeout" module=p2p tries=8
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=12
5:42AM INF Ensure peers module=pex numDialing=1 numInPeers=0 numOutPeers=5 numToDial=4
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42782->66.165.248.66:26656: i/o timeout" module=p2p tries=9
5:42AM INF Reconnecting to peer [email protected]:26656 module=p2p
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=0
5:42AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58296->202.61.240.62:14956: i/o timeout" module=p2p tries=9
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=13
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42816->66.165.248.66:26656: i/o timeout" module=p2p tries=10
5:42AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=1
5:42AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58326->202.61.240.62:14956: i/o timeout" module=p2p tries=10
5:42AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=14
5:43AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=2
5:43AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42846->66.165.248.66:26656: i/o timeout" module=p2p tries=11
5:43AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=15
5:43AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58352->202.61.240.62:14956: i/o timeout" module=p2p tries=11
5:43AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=3
5:43AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42874->66.165.248.66:26656: i/o timeout" module=p2p tries=12
5:43AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=16
5:43AM INF Saving AddrBook to file book=/gaia/.gaia/config/addrbook.json module=p2p size=1206
5:43AM INF Ensure peers module=pex numDialing=1 numInPeers=0 numOutPeers=5 numToDial=4
5:43AM INF service start impl="Peer{MConn{148.113.163.223:26656} d101f4966835550592c1602e96eba57bb8853f67 out}" module=p2p msg="Starting Peer service" [email protected]:26656
5:43AM INF service start impl=MConn{148.113.163.223:26656} module=p2p msg="Starting MConnection service" [email protected]:26656
5:43AM INF Connection is closed @ recvRoutine (likely by the other side) conn=MConn{148.113.163.223:26656} module=p2p [email protected]:26656
5:43AM INF service stop impl=MConn{148.113.163.223:26656} module=p2p msg="Stopping MConnection service" [email protected]:26656
5:43AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{148.113.163.223:26656} d101f4966835550592c1602e96eba57bb8853f67 out}"
5:43AM INF service stop impl="Peer{MConn{148.113.163.223:26656} d101f4966835550592c1602e96eba57bb8853f67 out}" module=p2p msg="Stopping Peer service" [email protected]:26656
5:43AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58376->202.61.240.62:14956: i/o timeout" module=p2p tries=12
5:43AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=4
5:43AM INF Error reconnecting to peer. Trying again [email protected]:13456 err="dial tcp 37.27.61.38:13456: i/o timeout" module=p2p tries=17
5:43AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="auth failure: secret conn failed: read tcp 172.17.0.2:42906->66.165.248.66:26656: i/o timeout" module=p2p tries=13
5:43AM INF Error reconnecting to peer. Trying again [email protected]:26656 err="dial tcp 35.230.171.24:26656: i/o timeout" module=p2p tries=5
5:43AM INF Error reconnecting to peer. Trying again [email protected]:14956 err="auth failure: secret conn failed: read tcp 172.17.0.2:58414->202.61.240.62:14956: i/o timeout" module=p2p tries=13

[mayank-daga]

Hi @mpoke any update on this

[mayank-daga]

Hi @mpoke any updates

[MSalopek]

Hey, the node has actually apphashed. With sdk v50, peering will remain live and the node will not shut down completely.
See this part of the logs:

5:41AM ERR Error in validation err="wrong Block.Header.LastResultsHash.
Expected 1DFDD836111C80D452125CAD1A9C6A10EEF7CDC5092B9DCF36C4202F8B054451,
got D68A3A4E9C99AFBCBB09BE2381C4BC0F59609E25D0DE0471E6AAECC7436FC988" module=blocksync

We are already discussing this issue here:
#3344

More information will be provided in that thread.

Short overview:
There seems to be an issue with quicksync archive snaps. Since the iteration time takes days (there's >20TB of data) the issue is progressing very slowly as it takes almost a week to try something out.