Pending channels with limbo balance.

[sanjay-shah]

Background

I have a lot of pending-channel with limbo balance. How can I recover or release the limbo balance?

Your environment

• version of lnd: 0.16.4-beta commit=v0.16.4-beta
• which operating system (uname -a on *Nix) : Linux lnd-2 5.4.0-155-generic #172-Ubuntu SMP Fri Jul 7 16:10:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux. Ubuntu 20.04.4 LTS
• version of bitcoind: v0.21.1
• any other relevant environment details

Steps to reproduce

lncli pendingchannels:

pending-channels.txt

When I try to rebroadcast by force close using lncli closechannel --funding_txid 355923a7ffb50ceab012b3d3359384de1e83db76eec4264c5242098173b12423 --output_index 1 --force
I get error:
[lncli] rpc error: code = Unknown desc = channel is already in the process of being force closed

On the block explorer, I can see both channel opening and closing transaction: https://mempool.space/lightning/channel/803239423648333825

Expected behaviour

Funds should move out of limbo balance.

Actual behaviour

Funds are still in limbo balance.

[ziggie1984]

To recover funds use chantools commands : sweeptimelock and sweepremoteclosed these should recover the most of your funds.

[guggero]

It looks like most of those channels (only looked at a couple of them) already fully resolved on-chain. So it appears like your sync with the chain backend might not be fully working or up to date. Try restarting everything (including bitcoind, and perhaps consider updating the version). If that doesn't work, try starting lnd with the reset-wallet-transactions=true option to force a full wallet rescan.

[saubyk]

Hi @sanjay-shah hope your queries are addressed. Moving this issue to a discussion, as there is nothing really to fix here for us.

[sanjay-shah]

Thanks @guggero @saubyk @ziggie1984

I'm re-syncing lnd with --reset-wallet-transactions. Will let you know if the issue gets fixed. Thanks again for your help.

[sanjay-shah]

Hi @guggero, I did restart lnd with lnd --reset-wallet-transactions. All the block were re-synced. I still see the same result of pending channels lncli pendingchannels. I'm trying to get this node ready for accepting payments for our upcoming Sports betting app exclusively on lightning :) and is scheduled to go live next weekend. I really need to get rid of all the false positive results of pendingchannels before that. Can I share any other information to troubleshoot this issue? Not sure if this is related, I also see one active HTLC with the expiration height of 788279:

[sanjay-shah]

{
"total_limbo_balance": "45275309",
"pending_open_channels": [
],
"pending_closing_channels": [
],
"pending_force_closing_channels": [
{
"channel": {
"remote_node_pub": "0347b33e82e93fb6516a6372e5b070716604c23617bf5b7489dce7d7f00e786562",
"channel_point": "4d381eda118ae2c76a8e48f53b52956ea473cab83c2d6d488ae3be347f24ed7b:1",
"capacity": "1000000",
"local_balance": "414011",
"remote_balance": "582237",
"local_chan_reserve_sat": "0",
"remote_chan_reserve_sat": "0",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"closing_txid": "429ac24b2aee8c2befd51b65bd116321e28d34da612f35feefb115d12f489ac1",
"limbo_balance": "414341",
"maturity_height": 798850,
"blocks_til_maturity": -10418,
"recovered_balance": "0",
"pending_htlcs": [
],
"anchor": "LIMBO"
},
{
"channel": {
"remote_node_pub": "02f863083c2972e0e673bb1650c05387a638cef0ec9a3539cba1a61380cd090f93",
"channel_point": "f72efd91bb5d355477fb18b6fbbf8b641be5a8cdb17cfb2446d748864b8cb9b7:0",
"capacity": "2531745",
"local_balance": "57279",
"remote_balance": "2471369",
"local_chan_reserve_sat": "0",
"remote_chan_reserve_sat": "0",
"initiator": "INITIATOR_REMOTE",
"commitment_type": "ANCHORS"
},
"closing_txid": "757efd0600737f2c09cd2257233cfc1dae467a73f01b2a292f41f32401370463",
"limbo_balance": "57279",
"maturity_height": 798964,
"blocks_til_maturity": -10304,
"recovered_balance": "0",
"pending_htlcs": [
],
"anchor": "LIMBO"
},
{
"channel": {
"remote_node_pub": "027835a73fc7c5a1d0a08a47f2d8c15a314ae63ad3195a2ac425e4035937fad5f9",
"channel_point": "0d4cb80ae727d4f8c675d19e4181d314d9e4686ee01b8e900083e23ee09eddca:1",
"capacity": "100000",
"local_balance": "37158",
"remote_balance": "59108",
"local_chan_reserve_sat": "0",
"remote_chan_reserve_sat": "0",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"closing_txid": "2b31c8a9a3ccb4578e7142665b1baf816079591e650449dbd0bd4bcf6064fa61",
"limbo_balance": "37158",
"maturity_height": 795682,
"blocks_til_maturity": -13586,
"recovered_balance": "0",
"pending_htlcs": [
],
"anchor": "LOST"
}
],
"waiting_close_channels": [
{
"channel": {
"remote_node_pub": "020605d79106b5a27a94436c57f56c4d04015c902aac9739f71d5ed9b18e24d3d1",
"channel_point": "22366bb6076a69fa3c0bc03256ccd751f15cccafd5fc5f83a4716f1dec335787:1",
"capacity": "3000000",
"local_balance": "1527021",
"remote_balance": "1471752",
"local_chan_reserve_sat": "30000",
"remote_chan_reserve_sat": "30000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "1527021",
"commitments": {
"local_txid": "8fc2f76ed8ce00ecb42b029f09af24e4770ca0c60f8fb2cafef843b17828d2f0",
"remote_txid": "93cd478b1bd34bbb4702310c2c959b15618f8a50d62c6b4bde4389b11badf3aa",
"remote_pending_txid": "",
"local_commit_fee_sat": "565",
"remote_commit_fee_sat": "565",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266",
"channel_point": "7e26f401a44a5398529032ac50e9a6182a5719fdc0f5630451c2aa7c6c49d585:0",
"capacity": "271918",
"local_balance": "267344",
"remote_balance": "2928",
"local_chan_reserve_sat": "2719",
"remote_chan_reserve_sat": "2719",
"initiator": "INITIATOR_LOCAL",
"commitment_type": "ANCHORS"
},
"limbo_balance": "267344",
"commitments": {
"local_txid": "286d1a06df7ed99dae48636f42ad9bee7d1fcd15e0e31db09517bf68b634e2d1",
"remote_txid": "118c90c81de9d46c3f371e8cb434e2037d2903c76ad199cf03b4a8ebefb8631f",
"remote_pending_txid": "",
"local_commit_fee_sat": "1645",
"remote_commit_fee_sat": "1645",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3",
"channel_point": "8f700abae16bc91a5dfad1209e8737b49dd4fa601c79a205eae4afc8b27d2130:0",
"capacity": "2000000",
"local_balance": "1919757",
"remote_balance": "27339",
"local_chan_reserve_sat": "20000",
"remote_chan_reserve_sat": "20000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": 3
},
"limbo_balance": "1919757",
"commitments": {
"local_txid": "f553993b913cc0c0cbb432f3479bc32225c71e8718482857c17e2d78a677bb9e",
"remote_txid": "7cf2aa309d3fdb33d504b1caccb8ab28c4454cc3f802fe758761bbdc68bc3e2a",
"remote_pending_txid": "847cbbd8db261907ecd8e67f60f294707a24db99a72a5cae7fbc715a863b8faf",
"local_commit_fee_sat": "3240",
"remote_commit_fee_sat": "3670",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "024bfaf0cabe7f874fd33ebf7c6f4e5385971fc504ef3f492432e9e3ec77e1b5cf",
"channel_point": "69b5e3e79636c5cdef5e4b8934e8135324730da9faa392649e8b56260dcd24ad:0",
"capacity": "10000000",
"local_balance": "4301576",
"remote_balance": "5670968",
"local_chan_reserve_sat": "100000",
"remote_chan_reserve_sat": "100000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "4301576",
"commitments": {
"local_txid": "187cca2426475a8608b76f58b36a934a9de6219b3425c0809ee7f69359c5745c",
"remote_txid": "f57f3d712671a9acadc2d75839da39fd349867ccbd5193af7399611b18b34272",
"remote_pending_txid": "51de466f61a35ee019b8a97d3d5bdc556e3cf78128e8fee997c73f20616a22ef",
"local_commit_fee_sat": "1648",
"remote_commit_fee_sat": "1867",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "026b2d2a6184ac6dd5aa75321304ece1a6615ea5125770d877ca126392b294abfc",
"channel_point": "8a28892960e7cad555182f0f509d426bf5b61c395f9c8eb7ca52a32fce38c335:0",
"capacity": "1000000",
"local_balance": "0",
"remote_balance": "996530",
"local_chan_reserve_sat": "10000",
"remote_chan_reserve_sat": "10000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "0",
"commitments": {
"local_txid": "32e23788e2711e35e1c8b2d1bda47c2ca33e12613f880359125fbbec98f7871d",
"remote_txid": "77fa4557198e28a76de1a7fc0356e555ba3ea84827d313484acf70557ecc4e69",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "026b2d2a6184ac6dd5aa75321304ece1a6615ea5125770d877ca126392b294abfc",
"channel_point": "a07cfc81242626ba3833325f9ebbc6fd06c6923262cbef1d41b80d666b5cd147:1",
"capacity": "100000",
"local_balance": "0",
"remote_balance": "96530",
"local_chan_reserve_sat": "1000",
"remote_chan_reserve_sat": "1000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "0",
"commitments": {
"local_txid": "c10bde5ee7d4b062fcbe5a97d95297ea5324687234bd59808933622fac092168",
"remote_txid": "1699cdbf3bd62325c520e0c11760654edea4d32d6c7afb8c71a40508d4b6fd38",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "026b2d2a6184ac6dd5aa75321304ece1a6615ea5125770d877ca126392b294abfc",
"channel_point": "f99abafb9ceda7a82688f8e74630bb57a19ecb05543efda98360dc604a3c15da:0",
"capacity": "80000",
"local_balance": "0",
"remote_balance": "76530",
"local_chan_reserve_sat": "800",
"remote_chan_reserve_sat": "800",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "0",
"commitments": {
"local_txid": "4d2915fe97391ad28dd565b7bdbbb57781b263896f315f0ab8d1c992874d19c2",
"remote_txid": "30f7408e627e1766c86832c5e2b1a4c0ca383755d75ce3665f4effb4e4801ff5",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "026b2d2a6184ac6dd5aa75321304ece1a6615ea5125770d877ca126392b294abfc",
"channel_point": "962aadd7fc7e1435a9dafb92141923254d754a81626515f181af291c88e4d4f4:1",
"capacity": "90000",
"local_balance": "0",
"remote_balance": "86530",
"local_chan_reserve_sat": "900",
"remote_chan_reserve_sat": "900",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "0",
"commitments": {
"local_txid": "c59de9481afc7ee56c531f31ae28644e1846b82235b553a149b19e9e88761a80",
"remote_txid": "c033a485e43706245fdaf38358d0c00760f4be28f7544cc4be8ea900914f9186",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "026e44acf41fcfa19d092a297a7e8452f6ac5eee677ed0f7f2e5d4c7c632467224",
"channel_point": "89a151a6e263c864f81b7ba7a7f42736c2f2129b91eb6dba9402027699ba8ab3:0",
"capacity": "1000000",
"local_balance": "971238",
"remote_balance": "25291",
"local_chan_reserve_sat": "10000",
"remote_chan_reserve_sat": "10000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "971238",
"commitments": {
"local_txid": "ca99f1ce776ae9c12eeb594b5bf195393fa1d3b3506f6d55b5e2867f0ede887e",
"remote_txid": "18c521958fe8da609f51745441692afeb5bba3c4b088c3e6dc4c39e0ab82705d",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "027df5ab17cf5aa7bcd51e95b19ab11ac7be9cb423fbf0e1bf7d796eebea19cb2d",
"channel_point": "5f66870b77388f9a7b66f3157dde918aa6958ada02bda1b2479741ad7f55df4f:1",
"capacity": "80000",
"local_balance": "16787",
"remote_balance": "61789",
"local_chan_reserve_sat": "800",
"remote_chan_reserve_sat": "800",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "16787",
"commitments": {
"local_txid": "aaf710c007c73cdcb6568f3ff71d63e82ae88b5e75c51a3a1079040f514dcf95",
"remote_txid": "e95a6dd8ff2d8abba1e2ac4e84278ab3d3819b1075ae900637758e3523a6749a",
"remote_pending_txid": "",
"local_commit_fee_sat": "763",
"remote_commit_fee_sat": "763",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "027df5ab17cf5aa7bcd51e95b19ab11ac7be9cb423fbf0e1bf7d796eebea19cb2d",
"channel_point": "94fa60cf7e509a5b5970cde19a94b84ead9da5392e1e37076ed81938b20699d9:1",
"capacity": "125000",
"local_balance": "9606",
"remote_balance": "114006",
"local_chan_reserve_sat": "1250",
"remote_chan_reserve_sat": "1250",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "9606",
"commitments": {
"local_txid": "642fd81aae1dc459c62a87feb47fa0c2ac965e3a7471fcc31fec15aa7fb199ce",
"remote_txid": "657f776996e3d50946b5c00efa1224debcac90cbbfb7185d398681d1bbf963d2",
"remote_pending_txid": "",
"local_commit_fee_sat": "727",
"remote_commit_fee_sat": "727",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "028cb0e85745efb47d1d3ec7dd0bf5ace7e14d07240c4e9f60ae28cab3548a7625",
"channel_point": "067715a289ea28a4dfd1a3bb5af1311b15de99529192bfa33ce3df375514ff12:1",
"capacity": "1000000",
"local_balance": "222032",
"remote_balance": "776332",
"local_chan_reserve_sat": "10000",
"remote_chan_reserve_sat": "10000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": 3
},
"limbo_balance": "222032",
"commitments": {
"local_txid": "b9181cbce1d3af707ebbe60c9cfe5f641ce324212f41b8d6717573a9b9477eee",
"remote_txid": "2a67084afe1958ddb5ba192c29838289f33ce4619882a62efb0e15684c40fb67",
"remote_pending_txid": "",
"local_commit_fee_sat": "975",
"remote_commit_fee_sat": "975",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f",
"channel_point": "16d2bfdee5b1a80ddf237c63d993bbe134451469fa386c6e0f266abeaf2055bb:1",
"capacity": "4000000",
"local_balance": "3162987",
"remote_balance": "834124",
"local_chan_reserve_sat": "40000",
"remote_chan_reserve_sat": "40000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": "ANCHORS"
},
"limbo_balance": "3162987",
"commitments": {
"local_txid": "0b47b36823dfb4395222299e09f00febcf45071580939a6f6772472f6f796fc4",
"remote_txid": "f56dedb46270d89cc3e6c22f8665d14038bc48c6dde08d709462367b4b291143",
"remote_pending_txid": "ccffe7882c10705a64d0410637124870ec161770361edba133b18ae7f715f9cc",
"local_commit_fee_sat": "2888",
"remote_commit_fee_sat": "3575",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "0299797daa21faa6e4151517b286c41d4c7a8cb21fbfba59ca3bcd21a9f8392bd4",
"channel_point": "ea68a978aa3774f7164fd0b5e45e0df8052754f6c281d1a3b25e201da6dc958f:0",
"capacity": "1000000",
"local_balance": "22369",
"remote_balance": "974259",
"local_chan_reserve_sat": "10000",
"remote_chan_reserve_sat": "10000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "22369",
"commitments": {
"local_txid": "6b4f7793089631dafb0153f118750febc5a038878fba2bcf4caddf4865a5f91e",
"remote_txid": "49cc3a8a3cd9f1e8d1efb8b27458e774a2ae5c699ca9439bdfdc98fdefbf8f48",
"remote_pending_txid": "",
"local_commit_fee_sat": "2711",
"remote_commit_fee_sat": "2711",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "0299797daa21faa6e4151517b286c41d4c7a8cb21fbfba59ca3bcd21a9f8392bd4",
"channel_point": "dd63512ed22e21abf9b2804c84b86f4a3d41c70f52f4db63b49455f2bf137ac2:1",
"capacity": "4000000",
"local_balance": "3852538",
"remote_balance": "112311",
"local_chan_reserve_sat": "40000",
"remote_chan_reserve_sat": "40000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "3852538",
"commitments": {
"local_txid": "beb2a6201db505b9f827ea37484d0467c4bdd26c3cf60c8cb76bd4bcf9352ba7",
"remote_txid": "b769749683700fccf23be5fbd07f10f94c7cb51954af9ad17e123cd5f1185dc2",
"remote_pending_txid": "",
"local_commit_fee_sat": "3240",
"remote_commit_fee_sat": "3240",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "029c5fd64570f2d40dfc2f683865c81e93269dc032416142de5e2e2461359ae34d",
"channel_point": "4a269b21adb8084788e6b316d51234c6d2e7d3ca27b8905b4d3e0e7ace9bb2ca:1",
"capacity": "4000000",
"local_balance": "3981523",
"remote_balance": "15006",
"local_chan_reserve_sat": "40000",
"remote_chan_reserve_sat": "40000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": 3
},
"limbo_balance": "3981523",
"commitments": {
"local_txid": "9ed8876542e1cbd969c49a3cfe482a9c1e0e69b618e0ac49d0009587697e6e11",
"remote_txid": "564ace0d64c1aedec7d5c88dc46484ea669bd9ce8df44ce5e5c056a4ad0506ca",
"remote_pending_txid": "7407a281c8e50714aeb2de46b238a394ea84ef735de2ec94b6e54ff96bfefa31",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69",
"channel_point": "355923a7ffb50ceab012b3d3359384de1e83db76eec4264c5242098173b12423:1",
"capacity": "10000000",
"local_balance": "117802",
"remote_balance": "9879189",
"local_chan_reserve_sat": "100000",
"remote_chan_reserve_sat": "100000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": 3
},
"limbo_balance": "117802",
"commitments": {
"local_txid": "98c02e4de657e46574e52ba60acf521b32092cbf79be6ce2e931b94da1970ad8",
"remote_txid": "599645fca7e4afa70be4946006ad3bee77a406a735a1ed8cc1f09e3915e4f72a",
"remote_pending_txid": "",
"local_commit_fee_sat": "2348",
"remote_commit_fee_sat": "2348",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "031ef2c0eab86352e7dbc4d22bb7459c99dc613ac23d791657dc8a2d2296e8ac42",
"channel_point": "79502fbf351bebff8f8f7e464ffd24d6ecda6b84db70fc8c487e147e6f54971f:0",
"capacity": "250000",
"local_balance": "0",
"remote_balance": "246530",
"local_chan_reserve_sat": "2500",
"remote_chan_reserve_sat": "2500",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "0",
"commitments": {
"local_txid": "39731f5329da7f1b7e76e693e809ed67380489dae2ce98f80613cde732fb5fed",
"remote_txid": "81921918c86c524f79714868aea8b8cb162ad9bcf062d81416753b79ff6b8039",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "0325b28ceb0b964d645a8767b95be8d25c4588c5b6ef3d15cb95f7408ed955ea40",
"channel_point": "9f27ec8494b70df4929b2d8a3fe5238d98e92a8fed1f336a95eca496aa28a031:1",
"capacity": "500000",
"local_balance": "464397",
"remote_balance": "32919",
"local_chan_reserve_sat": "5000",
"remote_chan_reserve_sat": "5000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "464397",
"commitments": {
"local_txid": "93c357f68dea86b9bbe5ea194d0f766169e1381d10d6aad3d79ae8600c70c3b7",
"remote_txid": "9b4a2948d84fba9edcdfe8f73552234d73a990497223952a6d3e72f95e3f1bef",
"remote_pending_txid": "",
"local_commit_fee_sat": "2023",
"remote_commit_fee_sat": "2023",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "035e4ff418fc8b5554c5d9eea66396c227bd429a3251c8cbc711002ba215bfc226",
"channel_point": "6db5a7d4f2812676f84928fbc96a97bc78f7e8466b77069d80914cc375b9c968:0",
"capacity": "500000",
"local_balance": "133719",
"remote_balance": "362810",
"local_chan_reserve_sat": "5000",
"remote_chan_reserve_sat": "5000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": 3
},
"limbo_balance": "133719",
"commitments": {
"local_txid": "5a1fc8ea17a3d5af7480f5fa221238ad218f28ac9058ba1fbbf26d05254370e9",
"remote_txid": "d62b08351ae84a8ac19d95c95d3fbaca0eda3abe5569f3e81a1858b5f813a1f4",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03627ebe50fc6eb80b0caab0c3714958c701eda735e3c29588e83150d6d4a93976",
"channel_point": "b4625a324bc56359c065b91bc0b523162d06ada7e3b4204a15413a834270ab1d:0",
"capacity": "109353",
"local_balance": "104262",
"remote_balance": "4907",
"local_chan_reserve_sat": "1093",
"remote_chan_reserve_sat": "1093",
"initiator": "INITIATOR_REMOTE",
"commitment_type": "ANCHORS"
},
"limbo_balance": "104262",
"commitments": {
"local_txid": "9c861e33c0c8bb9e365e55437ac45bcfb270d5738d657519739aecb023cd376f",
"remote_txid": "4e3d6356014e4987c3c957cf5573e9475cbd42d06eb3b5a67f9c5f4465abf557",
"remote_pending_txid": "",
"local_commit_fee_sat": "183",
"remote_commit_fee_sat": "183",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "037659a0ac8eb3b8d0a720114efc861d3a940382dcfa1403746b4f8f6b2e8810ba",
"channel_point": "252f9180f6da1d5e2f76049e06829044b3c1a69277f3360220a28a1d3afd1d30:0",
"capacity": "2000000",
"local_balance": "1027503",
"remote_balance": "937345",
"local_chan_reserve_sat": "20000",
"remote_chan_reserve_sat": "20000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": 3
},
"limbo_balance": "1027503",
"commitments": {
"local_txid": "33faef8013bfe0f1ab7059afc95a50aff600c23eac1cca65d89304156fa9dabb",
"remote_txid": "ebb5ec651c12a3e7484354fcb81664bad7b4fff3305a37b02c7844e222cf932a",
"remote_pending_txid": "ae893cb708dfac34c6fc35adf4e3199732a5fbf6ef14e1d24be585be614d4da1",
"local_commit_fee_sat": "3240",
"remote_commit_fee_sat": "2864",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "038fe1bd966b5cb0545963490c631eaa1924e2c4c0ea4e7dcb5d4582a1e7f2f1a5",
"channel_point": "7a53a47fa2ee146530ac2b86945e0c8bf99f4da1b8efd28798fd2124eb29f42c:1",
"capacity": "20000000",
"local_balance": "565483",
"remote_balance": "19402637",
"local_chan_reserve_sat": "200000",
"remote_chan_reserve_sat": "200000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": "ANCHORS"
},
"limbo_balance": "565483",
"commitments": {
"local_txid": "7dd1cd701869eb8c80129988b86d1b3a23172b55ed59233a3c0f616421a8d8c6",
"remote_txid": "48260032763ea196b289a612a9a9fc0c24f9c4cea2cbb771cb24e63048198f02",
"remote_pending_txid": "",
"local_commit_fee_sat": "1864",
"remote_commit_fee_sat": "1864",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03c8eaed66353ca6678ab1b32b5f59d149dda22f951ae61beacb0afd02af12b6be",
"channel_point": "5f8ec275c9d5b9b8416c1a8c0ad4b7b24e623f237af9091046cb0b66ac1432e1:0",
"capacity": "4000000",
"local_balance": "3772570",
"remote_balance": "223489",
"local_chan_reserve_sat": "40000",
"remote_chan_reserve_sat": "40000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "3772570",
"commitments": {
"local_txid": "9a4f01e976a7b182f8705be004b47603df83663ef02acadb58965370738f2cd8",
"remote_txid": "ac810d8037623b199e0ce93e803cce034e0664da38692e62771ad77a99a74557",
"remote_pending_txid": "",
"local_commit_fee_sat": "3280",
"remote_commit_fee_sat": "3280",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03cdb27fbfdc7f42e1ce2747906189b1105c7c30b99ecb96cb82afd2414eb8afb2",
"channel_point": "75436e6a4fb9d07c93eb971c7e3c640f4422d7d7a9ebb10b8c1059c6122cef24:1",
"capacity": "1000000",
"local_balance": "20579",
"remote_balance": "977012",
"local_chan_reserve_sat": "10000",
"remote_chan_reserve_sat": "10000",
"initiator": "INITIATOR_LOCAL",
"commitment_type": 3
},
"limbo_balance": "20579",
"commitments": {
"local_txid": "d4a2865e09814e7bd94e2ed7c89c9cba094e6b95dd38ac248e723303cb4ec8d5",
"remote_txid": "7851eeab428f5fd71452d4ca3179452dfba54b26e0d500c4157ad15403739ef8",
"remote_pending_txid": "",
"local_commit_fee_sat": "1748",
"remote_commit_fee_sat": "1748",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03d76e15709c57225560af75576b71c4d72dbac0dffaaba5ab8cc94e17ff8e708f",
"channel_point": "5cdbeceb8efb73bf20b73ba0118be6129dfe942b14e961285767553189c70af8:1",
"capacity": "4000000",
"local_balance": "3951373",
"remote_balance": "46003",
"local_chan_reserve_sat": "40000",
"remote_chan_reserve_sat": "40000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "3951373",
"commitments": {
"local_txid": "917ca379fbee606b91dd8757f1f6cfd46d26b4d2f37b06f39af14b246db5b148",
"remote_txid": "7e88bd040a51786d57dd968912b474300b7bd177212113179f85f99d9879917d",
"remote_pending_txid": "",
"local_commit_fee_sat": "1963",
"remote_commit_fee_sat": "1963",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03df3f0a2fd6bea5429a596461ce784c922b2981ada1af89cfefcd9ccfb16c16a7",
"channel_point": "e67b47f9bb9cefe8167ec310628761f8f8a28d53fe10b05605328b64e68a42f0:0",
"capacity": "623972",
"local_balance": "7018",
"remote_balance": "608713",
"local_chan_reserve_sat": "6239",
"remote_chan_reserve_sat": "6239",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "7018",
"commitments": {
"local_txid": "b93158ace1898c5442c22887fbbacd62c3d0bb28b606c9ef3fabc7a256d29044",
"remote_txid": "902f49af77656a335ef939421457625e7694e14b898cf8a8f9fe390f8d6ac36a",
"remote_pending_txid": "",
"local_commit_fee_sat": "7580",
"remote_commit_fee_sat": "7580",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03f29fee4eaca86e88049c0e630d96c715a5d9a0048b3b14946d45eca745784cef",
"channel_point": "f0d03a840a2e082f6ed02bb194211f82cb3788653b8482f2c75d90c511ff1b51:1",
"capacity": "5000000",
"local_balance": "3273600",
"remote_balance": "1722929",
"local_chan_reserve_sat": "50000",
"remote_chan_reserve_sat": "50000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "3273600",
"commitments": {
"local_txid": "6136027e43eb22ebed1c1486a51f50e07c8edfd0dc395ff86c39b946c3df1f13",
"remote_txid": "5e1a1967d20880135ef09d6f7db8da75072c19538a9fdc605f1caeceff791d5d",
"remote_pending_txid": "",
"local_commit_fee_sat": "2810",
"remote_commit_fee_sat": "2810",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03f3297397c8f5f685a562847611e20d15f56d6aaabc4d808a6e04e631dea6e612",
"channel_point": "628ec461b48e1ca56f0400a10a2d3ae22abf39da6faf62eaa9f46b5e85fc623e:1",
"capacity": "10000000",
"local_balance": "9077459",
"remote_balance": "813314",
"local_chan_reserve_sat": "100000",
"remote_chan_reserve_sat": "100000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "9077459",
"commitments": {
"local_txid": "d5812ec30949c7ca63d72de59dcc1c6c319c3831b70d9bef00217d3f878d16e7",
"remote_txid": "362db0b496d9daa10f069c793dbac1c70185af67be8aea0ec85a3625c62efc64",
"remote_pending_txid": "",
"local_commit_fee_sat": "3527",
"remote_commit_fee_sat": "3527",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03fc8aaec61b42da62e36f5473623ca9b91285d7af62c6189d14bcf84c549d473b",
"channel_point": "7f5120d28e4ac79082229536e28a66729e905b6a9aec831536a8878641742ffd:1",
"capacity": "3000000",
"local_balance": "1548240",
"remote_balance": "1448745",
"local_chan_reserve_sat": "30000",
"remote_chan_reserve_sat": "30000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "1548240",
"commitments": {
"local_txid": "ac2a994fe1fb9339957fc187db2dfc6bf3ec4d9ced99d1f6d4b4eabad8ba69ee",
"remote_txid": "849093bb1a56a53ff930c0c2f68b004c06da8f6c613b6367c563d5ac3bbd6786",
"remote_pending_txid": "",
"local_commit_fee_sat": "2352",
"remote_commit_fee_sat": "2352",
"remote_pending_commit_fee_sat": "0"
}
},
{
"channel": {
"remote_node_pub": "03fcf0e4e02498354e44adc3e35df94eaf6aab151b48eb931ccb6df551c1047b8d",
"channel_point": "7bc047ae8bea0d9e1041bf335f59588b11aeef6943697d69b61fbb2994fce4a4:1",
"capacity": "500000",
"local_balance": "447748",
"remote_balance": "50407",
"local_chan_reserve_sat": "5000",
"remote_chan_reserve_sat": "5000",
"initiator": "INITIATOR_REMOTE",
"commitment_type": 3
},
"limbo_balance": "447748",
"commitments": {
"local_txid": "4462ef043315bdb8b7ec71d212814b848875bf953dddd741d04c7d52f765e5a9",
"remote_txid": "791f740b0fbdbe2d03208013678b3e3d34fafcb04d95e5353dfd7dace6a3b019",
"remote_pending_txid": "",
"local_commit_fee_sat": "1184",
"remote_commit_fee_sat": "1184",
"remote_pending_commit_fee_sat": "0"
}
}
]
}

[guggero]

Thanks. Still that many, huh? What is the hardware you are running this on? Could it be that the disk is super slow? Or are you running a pruned bitcoind node?
Something definitely isn't right (or ready) with the chain sync. A lot (just did a few random samples) of these channels are already resolved on-chain. So for some reason lnd either is taking suuuuper long to re-scan (which could mean bitcoind is slow to run the scan) or something else is going on. Can you please (as well as answering the questions above) attach the latest lnd logs?

[ziggie1984]

Could relate to this #8028, need to investigate why the rescan for the output is not finding the corresponding transaction 🤔

[sanjay-shah]

lnd: 2 GB Memory / 1 Intel vCPU / 50 GB Disk SSD / SFO3 - Ubuntu 20.04 (LTS) x64. It's Ubuntu VM on cloud.

sudo lshw -short
H/W path Device Class Description

                      system         Droplet

/0 bus Motherboard
/0/0 memory 96KiB BIOS
/0/401 processor DO-Premium-Intel
/0/1000 memory 2GiB System Memory
/0/1000/0 memory 2GiB DIMM RAM
/0/100 bridge 440FX - 82441FX PMC [Natoma]
/0/100/1 bridge 82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1 storage 82371SB PIIX3 IDE [Natoma/Triton II]
/0/100/1.2 bus 82371SB PIIX3 USB [Natoma/Triton II]
/0/100/1.2/1 usb1 bus UHCI Host Controller
/0/100/1.3 bridge 82371AB/EB/MB PIIX4 ACPI
/0/100/2 display QXL paravirtual graphic card
/0/100/3 network Virtio network device
/0/100/3/0 eth0 network Ethernet interface
/0/100/4 network Virtio network device
/0/100/4/0 eth1 network Ethernet interface
/0/100/5 storage Virtio SCSI
/0/100/5/0 generic Virtual I/O device
/0/100/6 storage Virtio block device
/0/100/6/0 /dev/vda disk 53GB Virtual I/O device
/0/100/6/0/1 /dev/vda1 volume 49GiB EXT4 volume
/0/100/6/0/e /dev/vda14 volume 4095KiB BIOS Boot partition
/0/100/6/0/f /dev/vda15 volume 105MiB Windows FAT volume
/0/100/7 generic Virtio memory balloon
/0/100/7/0 generic Virtual I/O device
/0/1 system PnP device PNP0b00
/0/2 input PnP device PNP0303
/0/3 input PnP device PNP0f13
/0/4 storage PnP device PNP0700
/0/5 communication PnP device PNP0501

sudo cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : DO-Premium-Intel
stepping : 7
microcode : 0x1
cpu MHz : 2494.140
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 pku ospke avx512_vnni arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit mmio_stale_data retbleed eibrs_pbrsb
bogomips : 4988.28
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

=================

bitcoind full node (not pruned): 2 GB Memory / 50 GB Disk + 700 GB / SFO3 - Ubuntu 20.04 (LTS) x64. On a separate VM.

sudo lshw -short
H/W path Device Class Description

                      system         Droplet

/0 bus Motherboard
/0/0 memory 96KiB BIOS
/0/401 processor DO-Regular
/0/1000 memory 2GiB System Memory
/0/1000/0 memory 2GiB DIMM RAM
/0/100 bridge 440FX - 82441FX PMC [Natoma]
/0/100/1 bridge 82371SB PIIX3 ISA [Natoma/Triton II]
/0/100/1.1 storage 82371SB PIIX3 IDE [Natoma/Triton II]
/0/100/1.2 bus 82371SB PIIX3 USB [Natoma/Triton II]
/0/100/1.2/1 usb1 bus UHCI Host Controller
/0/100/1.3 bridge 82371AB/EB/MB PIIX4 ACPI
/0/100/2 display QXL paravirtual graphic card
/0/100/3 network Virtio network device
/0/100/3/0 eth0 network Ethernet interface
/0/100/4 network Virtio network device
/0/100/4/0 eth1 network Ethernet interface
/0/100/5 storage Virtio SCSI
/0/100/5/0 generic Virtual I/O device
/0/100/6 storage Virtio block device
/0/100/6/0 /dev/vda disk 53GB Virtual I/O device
/0/100/6/0/1 /dev/vda1 volume 49GiB EXT4 volume
/0/100/6/0/e /dev/vda14 volume 4095KiB BIOS Boot partition
/0/100/6/0/f /dev/vda15 volume 105MiB Windows FAT volume
/0/100/7 generic Virtio memory balloon
/0/100/7/0 generic Virtual I/O device
/0/1 system PnP device PNP0b00
/0/2 input PnP device PNP0303
/0/3 input PnP device PNP0f13
/0/4 storage PnP device PNP0700
/0/5 communication PnP device PNP0501
/0/6 scsi2 storage
/0/6/0.0.1 /dev/sda volume 700GiB Volume

sudo cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : DO-Regular
stepping : 2
microcode : 0x1
cpu MHz : 2294.608
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon rep_good nopl cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt md_clear
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data
bogomips : 4589.21
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

lnd and bitcoind are connected using private networking, they are both on the same data center.
Latency between lnd and bitcoind:
ping 10.124.0.10
PING 10.124.0.10 (10.124.0.10) 56(84) bytes of data.
64 bytes from 10.124.0.10: icmp_seq=1 ttl=64 time=2.41 ms
64 bytes from 10.124.0.10: icmp_seq=2 ttl=64 time=0.777 ms
64 bytes from 10.124.0.10: icmp_seq=3 ttl=64 time=1.11 ms
64 bytes from 10.124.0.10: icmp_seq=4 ttl=64 time=0.853 ms
^C
--- 10.124.0.10 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3033ms
rtt min/avg/max/mdev = 0.777/1.288/2.411/0.660 ms

Here's the lnd log:

lnd.log

[guggero]

Thank you, that is very helpful. I added some comments to #8028 which might indeed be related.