PostgreSQL issue - PANIC : replication checkpoint has wrong magic

[jun3280net]

Hi there,

I just tried reloading teslamate after noticing my VM environment had gone down. However, now I'm receiving a POSTGRESQL error related to loading the database.

database_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
database_1 |
database_1 | 2021-05-14 08:54:33.423 UTC [1] LOG: starting PostgreSQL 12.6 (Debian 12.6-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
database_1 | 2021-05-14 08:54:33.429 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
database_1 | 2021-05-14 08:54:33.433 UTC [1] LOG: listening on IPv6 address "::", port 5432
database_1 | 2021-05-14 08:54:33.436 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
database_1 | 2021-05-14 08:54:33.453 UTC [26] LOG: database system was interrupted; last known up at 2021-05-14 01:54:52 UTC
database_1 | 2021-05-14 08:54:33.477 UTC [26] PANIC: replication checkpoint has wrong magic 1650864225 instead of 307747550
database_1 | 2021-05-14 08:54:34.455 UTC [28] FATAL: the database system is starting up
database_1 | 2021-05-14 08:54:34.459 UTC [29] FATAL: the database system is starting up
teslamate_1 | 2021-05-14 08:54:34.502 [error] Postgrex.Protocol (#PID<0.166.0>) failed to connect: ** (Postgrex.Error) FATAL 57P03 (cannot_connect_now) the database system is starting up
teslamate_1 | 2021-05-14 08:54:34.503 [error] Postgrex.Protocol (#PID<0.167.0>) failed to connect: ** (Postgrex.Error) FATAL 57P03 (cannot_connect_now) the database system is starting up
database_1 | 2021-05-14 08:54:34.952 UTC [1] LOG: startup process (PID 26) was terminated by signal 6: Aborted
database_1 | 2021-05-14 08:54:34.952 UTC [1] LOG: aborting startup due to startup process failure
database_1 | 2021-05-14 08:54:34.973 UTC [1] LOG: database system is shut down

I've captured the log images below:

This is definitely outside the realm of my understanding, but I figured I might be able to backup the database and restore it. However, backup is not possible as it complains the container is restarting :

Any suggestions on how to troubleshoot this issue would be greatly appreciated.

Thank you

[jun3280net]

Hopefully, this helps other people out who run into this situation. I found the following link which provides some insight into how to address the problem :

https://www.postgresql-archive.org/PANIC-replication-checkpoint-has-wrong-magic-0-instead-of-td6146772.html

My steps are as follows :

• docker-compose down or CTRL + C to shut down the Teslamate, Postgres, Grafana
• With "elevated privileges / root" I navigated to the directory in Linux Mint : /var/lib/docker/volumes/teslamate_teslamate-db/_data/pg_logical/
• Then I renamed the replorigin_checkpoint file to replorigin_checkpoint_corrupted, as starting up the docker-compose will generate a new replorigin_checkpoint file. In theory, you could delete the file for the same effect - but renaming the file gives you the ability to revert back to your previous state if this doesn't work for you.

(this screenshot includes the newly generated replorigin_checkpoint file)

• docker-compose up
• All fixed :)

[an-suz]

Great, thanks for this. For anyone trying to fix the problem on MacOS (as I was), there's an extra step to find the Docker volume: https://ekartco.com/2019/12/enter-docker-vm-on-macos-catalina-ssh-xhyve/

[BlackWolfWoof]

You saved my butt, thanks a lot 🖤
Was using postgres for Immich

[mohammadrz003]

worked like a charm!
thanks man.

[cotunjr]

after 4 years, thysm. Fixed it with a just simple change.