Memory Leak?

[wobloz]

Hi, I've got a memory issue with LND. I run a 100+ channel node. After a restart, lnd has about 350MB of memory. However, it keeps steadily rising. After a day or so the consumption is about 1GB. After a few days 2GB until the machine crashes. Any idea, please? Is this a known memory leak? Thanks, Pavel

• version of lnd : v0.15.5-beta
• which operating system (uname -a on *Nix) : Linux myNode 5.10.103-v8+ 1529 SMP PREEMPT Tue Mar 8 12:26:46 GMT 2022 aarch64 GNU/Linux
• version of btcd, bitcoind, or other backend: bitcoind 24.0.1,
• any other relevant environment details: mynode 0.3.06

[ellemouton]

Hi @wobloz! Would you be willing to do a couple of pprof scans of your lnd node as the memory consumption increases? That should give us an indication of where the problem area is. Here is a description for how to do this: https://github.com/lightningnetwork/lnd/blob/master/docs/debugging_lnd.md#capturing-pprof-data-with-lnd

[wobloz]

I am sorry, but I don't know how to perform a heap dump for lnd. Could you help please?

[wobloz]

And a second question. Does a heap dump contain sensitive information? Is it safe to post it?

[ellemouton]

you can do it like this: curl -sK -v http://localhost:9736/debug/pprof/heap > heap.out

Does a heap dump contain sensitive information? Is it safe to post it?

No it should be fine to post. To nicely see the info it provides, you can do this: go tool pprof -http=':8082' heap.out
It just gives info about what objects are holding memory but not about the contents of that memory. If you want to rather DM me the logs on slack, then feel free to do so.

[wobloz]

Here is the dump. 1420 MB today.
heap.out.log

[wobloz]

One more heap dump at 1835 MB. Now I have to restart.
heap2.out.log

[ellemouton]

@wobloz , Wondering if this is perhaps linked to #5983

Do you know if your watchtower servers are currently active? ie, it could be that you dont currently have an active tower server and so these updates just queue in memory. Definitely something we are wanting to fix soon

[wobloz]

I have found this in my lnd.conf:

[Wtclient]
wtclient.active=1

But I haven't configured it manually. It has to be there from the default installation. Can it be related? I have no watchtower connection AFAIK:

admin@myNode:~ $ lncli wtclient towers
{
"towers": [
]
}

[ellemouton]

aha! yeah if wtclient is active but there is no configured tower then this will happen.

I think for the meantime, if you dont want to add a tower, just remove the wtclient.active=1 line from the config. Otherwise, configure a tower.

[wobloz]

OK. Let me reconfigure it like this and I will let you know after a few days if it has helped. Thank you.