[bug]: Waiting Close Channel - ChansStatusDataLoss

[sperolightning]

Background

it happened that there was a power failure and subsequently it appeared that there was a "waiting close channel". The node is synchronized and I tried to perform the forced closure command as read in other posts:
lncli closechannel – force “channel_points” 1
but the following message appears
rpc error appears: code = Unknown desc = cannot close channel with state: ChanStatusLocalDataLoss
Other user resolved by contacting the other node forcing closure. In this case the node is Kraken, and I imagine it is not easy to contact and solve the issue. I wonder if there is a software solution to solve the problem?
Thanks for your advices

Your environment

Umbrel Version 0.5.3 on a Raspberry 4b 8GB HDD 2TB.

Steps to reproduce

lncli closechannel – force “channel_points” 1

Expected behaviour

closure of channel

Actual behaviour

rpc error appears: code = Unknown desc = cannot close channel with state: ChanStatusLocalDataLoss

[guggero]

It sounds to me like the power outage caused data corruption in the channel DB and lnd noticed on startup that its commitment height didn't match the remote node's height and went into safe mode for that channel (ChanStatusLocalDataLoss).
The following should work:

1. Make sure you have a backup of your channel.backup file on another machine than just your Umbrel. Creating a backup of the channel.db file might also be a good idea here in case things go sideways (just never restore that file to its original location, only use it with chantools when necessary).
2. Export the channel backup for the channel in question: lncli exportchanbackup --chanpoint <xxxx:y> --output_file /tmp/chan1.backup
3. Try restoring that single channel from backup: lncli restorechanbackup --single_backup /tmp/chan1.backup.
4. Check the log for the channel in question. If it says something like "already exists in channel DB" then you need to remove it from the DB first:
   1. Shut down lnd and make sure it stays shut down and isn't just restarted by Umbrel (might need to consult their docs to figure out how to do that).
   2. Run chantools removechannel --channel <xxxx:y> --channeldb <your_channel_db>
   3. Run lncli exportchanbackup --chanpoint <xxxx:y> --output_file /tmp/chan1.backup and check the logs.

The above should ask the remote node to force close the channel from their end.

But in cases with data corruption to the channel.DB I recommend to start with a fresh DB just to make sure there aren't other areas affected. If you decide to go with that option, you can simplify the steps somehow. Just close all normally operating channels cooperatively (that should save on fees compared to using the backup), then stop lnd and remove the channel.db, restart and use the full channel.backup file to get the remaining, borked channel back.

[guggero]

I'm also going to convert this to a discussion as this is the current expected behavior of lnd if it detects data corruption. We are actively working on migrating to another DB format that is less prone to corrupting on sudden power loss.

[caiomsbr]

Hello Guggero, I had my password here reset... I sent you some tweets asking for help...

That same situation has hapenned to my node, Had a surge, channel.db corrupt, deleted channel.db broght lnd back up, used lncli restorechanbackup and managed to get funds back from all channels except one that's with bcash_is_trash node.

I tried this:
Export the channel backup for the channel in question: lncli exportchanbackup --chanpoint xxxx:y --output_file /tmp/chan1.backup
Try restoring that single channel from backup: lncli restorechanbackup --single_backup /tmp/chan1.backup.

And I got:

[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack single backups: could not derive private key for legacy channel revocation root format: unable to derive private key

Can you help me with that?

[guggero]

@caiomsbr it sounds like you're encountering #7301 (comment) (since bcash_is_trash is a known CLN node).
You can follow these steps to get the channel force closed from the other side. Though it requires some technical know-how and running a custom version of lnd. So maybe it's easier to instead contact the node operator (I'm sure there are contact details available on some of the graph explorers, as the node is quite well known) and ask them to manually force close.

I'm not sure what you're trying to achieve with exporting the channel backup? Are you doing that on the same node where you did the initial channel backup restore with the "full" file? Then exporting and importing it again won't do anything anyway.

[sperolightning]

If it is still needed. I solved this way, but by accident. My umbrella went out for some reason. Then I tried to recover satoshi stuck in a closed channel. I closed all the channels and tried to recover the channels, opening the db backups of the saved channels one at a time to restore the situation before the channel closed. Finally I found all the satoshis.

[caiomsbr]

I was trying to get the a close channel signal on bcash side, because on their end I still show as active node since i'm getting forward reports on LNDg, I sent them a message through lightningnetworkplus website, but Idk if they are ever going to read it...

[guggero]

@caiomsbr we are working on a solution to initiate the force close with those nodes. See latest comments #7301. Once that is working as expected, you should be able to run that. The alternative is to wait until the next CLN release gets out and the remote node updates, then the recover should work normally.

[stargati]

I'm actually on Umbrel and have this exact same issue with the exact same node. Had the issue with another CLN node as well, but I managed to contact them and they force closed manually, which resolved the issue.

I assume I won't be able to run the custom binary. Is there no other option for Umbrel users other than waiting (hoping) for CLN to change this? @guggero, will your workaround be implemented in the final release of v0.16.0-beta?

Similar to @caiomsbr, I have found absolutely no way to contact this operator, despite being a well known CLN node. At my wit's end trying to recover the funds. It's the only channel that hasn't closed after a recent recovery.

[guggero]

Given that this is indeed a problem for users running Umbrel and other node packaging solutions where running a custom version of lnd is not easy, I created a specific new command in chantools to solve this.

Steps:

• Download the latest version of chantools (v0.10.7) (can be any machine, use the darwin release for MacOS).
• Run the following command (replacing the peer address with the full address of the remote peer and the channel point with the channel in question):

chantools triggerforceclose --peer 039d9c531877266e190af2d51795b6104a1e91a188dad4b9ccf3ae4154075dd8e3@localhost:9747 --channel_point 8e3efb6f5537d964c1c02c0587f75fc6995888675083400d4b10c02f47ad3592:0

• The tool will ask you for your seed, which is required to initiate the encrypted p2p communication with the peer.
• If the channel is observed by lnd (e.g. showing up in the list of lncli pendingchannels), then just wait until the force close transaction is confirmed and the channel should automatically update the status there.
• If the channel isn't showing up in the list of pending channels, you can use the chantools sweepremoteclosed command to sweep the funds.

@caiomsbr this might be interesting to you as well.

[stargati]

@guggero my man, you are a godsend. can confirm this works beautifully and funds are now secured on-chain. thank you so much for putting this together!! is there any way I can send you a lightning tip?

[guggero]

There is a tip address in my Nostr profile. But don't worry about it, I'm happy the tool solved your issue.

[caiomsbr]

I got it, ok. My last dumb question: Just as soon as CLN updates, should I use the whole multichan backup (and close all the channels again) or there is a way to retrieve just bcash (CLN) channel from the multifile and initiate the recover from there? Because I didnt backup single channel file just for bcash then!

[caiomsbr]

I have reopened another channels and I got my node working again, except CLN which is in waiting for close (but the network thinks is open and active). There are some things on that step by step I don't understand, but I think with some chat I can perform!

[guggero]

Ah, wait, you're on Umbrel, so it won't be easy for you to run a custom version of lnd...

[caiomsbr]

No, i'm on mynodebtc.com (MyNode)

[guggero]

Right, sorry. Can you run a custom lnd binary in MyNode? Then this should work.

[caiomsbr]

actually I don't know! It has a prompt with root access to file systems and linux commands, but I don't know if mynode is gonna perform automatic actions upon building another lnd on that system!