[Docs] upgrade/chain halt recovery #837

okdas · 2024-09-24T23:48:06Z

Summary

Performed the first upgrade on the Alpha TestNet. Add some documentation changes to prevent some issues in the future.

Issue

N/A

Type of change

Select one or more from the following:

New feature, functionality or library
Consensus breaking; add the consensus-breaking label if so. See [Infra] Automatically add the consensus-breaking label #791 for details
Bug fix
Code health or cleanup
Documentation
Other (specify)

Testing

Documentation: make docusaurus_start; only needed if you make doc changes
Unit Tests: make go_develop_and_test
LocalNet E2E Tests: make test_e2e
DevNet E2E Tests: Add the devnet-test-e2e label to the PR.

Sanity Checklist

I have tested my changes using the available tooling
I have commented my code
I have performed a self-review of my own code; both comments & source code
I create and reference any new tickets, if applicable
I have left TODOs throughout the codebase, if applicable

okdas · 2024-09-26T22:04:54Z

Items to add:

Configure cosmovisor to create backups before upgrade
Have more bump validators to 5.
Instructions on how to test the upgrade locally. Thinking a script w/o LocalNet.

Olshansk · 2024-10-15T15:46:36Z

@okdas Let's add the details here as well: https://x.com/olshansky/status/1846211059989778741

okdas · 2024-10-17T01:20:31Z

@Olshansk can I get a review on this PR, please?

I think I want to investigate/add information about the new Cosmovisor feature mentioned in cosmos-sdk slack, but I'll look into that later. :)

github-actions · 2024-10-17T20:35:50Z

The CI will now also run the e2e tests on devnet, which increases the time it takes to complete all CI checks.

You may need to run make trigger_ci to submit an empty commit that'll trigger the tests.

GCP workloads (requires changing the namespace to 837)
Grafana network dashboard for devnet-issue-837

Olshansk · 2024-10-21T17:01:30Z

tools/installer/full-node.sh

-    # Use the direct download link for the latest release
-    LATEST_RELEASE_URL="https://github.com/pokt-network/poktroll/releases/latest/download/poktroll_linux_${ARCH}.tar.gz"
+    # Get the version genesis started from
+    POKTROLLD_VERSION=$(curl -s https://raw.githubusercontent.com/pokt-network/pocket-network-genesis/master/poktrolld/testnet-validated.init-version)


Is this always going to point github?

#PUC or add TODO to answer the question.

I rephrased this part, but I don't fully understand what you're asking.
If you think that there's a better way, please let me know. I thought we could use the app_version from genesis.json (looks like this: "app_version": "0.0.9") but decided I'd rather store this information in a separate file.

Olshansk · 2024-10-21T17:02:52Z

docusaurus/docs/develop/developer_guide/recovery_from_chain_halt.md

+
+Read more about [upgrade contingency plans](../../protocol/upgrades/contigency_plans.md).
+
+### Manual binary replacement (preferred)


Please update this section w/ links to the binaries for easier access.

There are no binaries to point to. Rephrased.

docusaurus/docs/develop/developer_guide/recovery_from_chain_halt.md

Olshansk · 2024-10-21T17:05:30Z

docusaurus/docs/develop/developer_guide/recovery_from_chain_halt.md

+
+Instead, we need **social consensus** to manually replace the binary and get the chain moving.
+
+Currently this involves synching the network from genesis breaking a way to sync the network from genesis without human interaction, but there are some plans to make the process less painful in the future.


I don't fully understand what you're trying to say with this sentence. #PUC

LOL that sentence makes no sense. The original was more comprehensible.

docusaurus/docs/develop/developer_guide/recovery_from_chain_halt.md

Olshansk · 2024-10-21T18:13:18Z

docusaurus/docs/protocol/upgrades/contigency_plans.md

+In such a case, we need to:
+
+- Roll back validators to the backup (a snapshot is taken by `cosmovisor` automatically prior to upgrade, if `UNSAFE_SKIP_BACKUP` is set to `false`).
+- Skip the upgrade handler and store migrations with `--unsafe-skip-upgrade=$upgradeHeightNumber`.


Please show full command (or link to script).

I don't know what this means

Just like with --halt-height= there's no command that will work for all use cases. I'll add an example, but it's not very usable.

#PUC

Bullet points

Consider making it a separate section below

docusaurus/docs/protocol/upgrades/contigency_plans.md

Olshansk · 2024-10-21T18:27:32Z

docusaurus/docs/protocol/upgrades/upgrade_procedure.md


-   - The old binary should be compiled to work before the upgrade.
-   - The new binary should contain the upgrade logic to be executed immediately after the node is started using the new binary.
+9. Wait until the height is reached and the old node dies due to the error: `ERR UPGRADE "v0.0.9-2" NEEDED at height`, which is expected.


I did a little bit of partial cleanup, but can you do another look through this and make it so copy-pastable that any idiot can follow?

For example, instead of "from the new version", just add the appropriate cd ../... in there.

How do I know what the new version path on the developer's system? It might be a primary poktroll repo on the developer machine, it might not be. ¯\_(ツ)_/¯

Maybe an instruction where they run poktroll version to ensure it's correct?

It's one of the steps in the upgrade playbook for Morse:

docusaurus/docs/protocol/upgrades/upgrade_procedure.md

Olshansk · 2024-10-21T18:27:47Z

docusaurus/docs/protocol/upgrades/upgrade_procedure.md


-   - The old binary should be compiled to work before the upgrade.
-   - The new binary should contain the upgrade logic to be executed immediately after the node is started using the new binary.
+9. Wait until the height is reached and the old node dies due to the error: `ERR UPGRADE "v0.0.9-2" NEEDED at height`, which is expected.


How do I observe the height?

It's all on the screen when you run the command

#PUC? Screenshot? Video?

Olshansk · 2024-11-06T19:16:26Z

@okdas Friendly reminder on this when you're back.

Making it easier to inspect/find/identify chain halts on testnet would be great as well!

okdas added 2 commits September 24, 2024 13:11

--wip-- [skip ci]

bd12a8b

document learnings and more checks

4ea092e

okdas self-assigned this Sep 24, 2024

okdas added the documentation Improvements or additions to documentation label Sep 24, 2024

okdas added this to the Shannon Beta TestNet Launch milestone Sep 24, 2024

Merge branch 'main' into dk-upgrade-learnings

341b987

okdas requested a review from Olshansk September 24, 2024 23:50

okdas changed the title ~~[Upgrades] Document learnings from the past upgrade~~ [Upgrades] Document learnings from the last upgrade Sep 25, 2024

ca-certs are needed for relayminer

3d86aef

okdas and others added 2 commits October 14, 2024 15:26

Merge branch 'main' into dk-upgrade-learnings

94d8af3

LocalNet upgrade procedure

cb073bd

DK and others added 4 commits October 15, 2024 16:19

--wip-- [skip ci]

3fe1fea

--wip-- [skip ci]

525eb57

spell checking

4e8c7dd

Merge branch 'main' into dk-upgrade-learnings

9d5696d

okdas marked this pull request as ready for review October 17, 2024 01:18

okdas changed the title ~~[Upgrades] Document learnings from the last upgrade~~ [Upgrades] Document upgrade/chain halt recovery Oct 17, 2024

okdas changed the title ~~[Upgrades] Document upgrade/chain halt recovery~~ [Docs] upgrade/chain halt recovery Oct 17, 2024

okdas added the devnet-test-e2e label Oct 17, 2024

github-actions bot added devnet push-image CI related - pushes images to ghcr.io labels Oct 17, 2024

okdas and others added 5 commits October 17, 2024 13:36

Empty commit

dd631e6

Merge with main

9d0bc9c

Partial review

83a24aa

Partial review

50cc08e

Partial review

3bddc15

Partial review

498d9d8

Olshansk requested changes Oct 21, 2024

View reviewed changes

okdas and others added 2 commits October 21, 2024 17:59

Merge branch 'main' into dk-upgrade-learnings

7ecf403

requested changes

be35f1a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Docs] upgrade/chain halt recovery #837

[Docs] upgrade/chain halt recovery #837

okdas commented Sep 24, 2024

okdas commented Sep 26, 2024

Olshansk commented Oct 15, 2024

okdas commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

Olshansk Oct 21, 2024

okdas Oct 24, 2024

Olshansk Oct 21, 2024

okdas Oct 24, 2024

Olshansk Oct 21, 2024

okdas Oct 23, 2024 •

edited

Loading

Olshansk Oct 21, 2024

okdas Oct 24, 2024

Olshansk Oct 24, 2024

Olshansk Oct 21, 2024

okdas Oct 24, 2024

Olshansk Oct 24, 2024

Olshansk Oct 21, 2024

okdas Oct 24, 2024

Olshansk Oct 24, 2024

Olshansk commented Nov 6, 2024


		Read more about [upgrade contingency plans](../../protocol/upgrades/contigency_plans.md).

		### Manual binary replacement (preferred)


		Instead, we need social consensus to manually replace the binary and get the chain moving.

		Currently this involves synching the network from genesis breaking a way to sync the network from genesis without human interaction, but there are some plans to make the process less painful in the future.

[Docs] upgrade/chain halt recovery #837

Are you sure you want to change the base?

[Docs] upgrade/chain halt recovery #837

Conversation

okdas commented Sep 24, 2024

Summary

Issue

Type of change

Testing

Sanity Checklist

okdas commented Sep 26, 2024

Olshansk commented Oct 15, 2024

okdas commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

okdas Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Olshansk commented Nov 6, 2024

okdas Oct 23, 2024 •

edited

Loading