unlimit RSS memory occupy about Rocksdb(maybe) in Kvrocks

[xiaofan8421]

Background:
our expected rss mem occupy is 1G(metadata) + 1G(subkey) + 4*64(all memtables) + others =(almost) 3G(maybe below it, but if >3G, unexpected)
But actually, rss mem in our machine is almost 6.6G and slow-growth is still happening in Debian-11-Server. it exceeds the understanding of our team expected.
But the other thing, we run the same Kvrocks version in our Centos-7-Server is ok, almost 3G rss mem. mem slow-growth does not happen anymore.
Our Reproduce Benchmark Test: our inner bin-stress-test which uses PYSNC to test this confusing problem.

our cfg about Rocksdb cache in Kvrocks:
rocksdb.metadata_block_cache_size 1024
rocksdb.subkey_block_cache_size 1024
rocksdb.share_metadata_and_subkey_block_cache yes
rocksdb.block_size 16384
rocksdb.cache_index_and_filter_blocks yes
rocksdb.max_write_buffer_number 4
rocksdb.write_buffer_size 64

reproduce machine:
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/

Here is Kvrcocks rss mem which is collected by pmap mem tool

Firstly, we doubt that Kvrocks oom out of mem leak by itself, but I compiled Google ASan into it, running a few days, ok now.

Secondly, I did some more research for further. Except above pmap, I dumped some Kvrocks address mem segment content. for example:

Thirdly, I find the monitor system unexpected phenomenon: rss memory outgrows fast when Rocksdb compaction midnight.

Based on the above findings, I think that it is Rocksdb uses lots of rss memory, but why it is unlimited in the above cfg? I have no idea now. After all, we have already made block_cache shared (metadata/subkey/index/filter).

[mapleFU]

Would you mind provide the version of kvrocks? Since different version of RocksDB is used, and config might changed in our system.

[xiaofan8421]

Kvrocks: v2.4.0
Rocksdb: v8.1.1 (which is shown in Kvrocks-dev2.4.0 rocksdb.cmake)

[git-hulk]

@xiaofan8421 Thanks for your detailed analysis. Could you show the Y-axis number of this picture? I'm not sure if it's a memory fragment issue caused by compaction.

[xiaofan8421]

emm, I both tested glibc(if ASan enabled, Maybe Kvrocks doesn't use jemalloc? When I tested if jemalloc enabled, ASan is not working) and jemalloc on Debian, and all happened.


As is shown in the above picture, jeprof shows correctly, 2.xG is as expected, but other RSS memory couldn't explain.

Let me have a look at max-open-files. Do you mean rocksdb.max_open_files? I use the default value: 8096

Do you think that rocksdb.max_open_files will have a great influence on high memory?

[git-hulk]

Do you think that rocksdb.max_open_files will have a great influence on high memory?

No, unless it's set to 0, 8096 won't consume so much memory.

[PragmaTwice]

(if ASan enabled, Maybe Kvrocks doesn't use jemalloc? When I tested if jemalloc enabled, ASan is not working)

If ASan is enabled, it is quite normal that a large amount of memory is consumed.

[xiaofan8421]

What a confusing problem that makes me confused. Actually, I do many PoCs, but currently only proves that Rocksdb uses high mem but the root cause is still unknown(I mean how to make it in my cfg control). I searched Rocksdb official issues, unlimited memory also raised many discussions in these years. It needs time to analyze.

I found this weird problem almost a month ago when I was doing stress-test work on our project. I did have not too many machines to do some more PoCs because I am in a small company(everything is limited, including my time). But I will follow up with the issue in my spare time.

[xiaofan8421]

Here 71% means about 5.7G RSS mem usage.
One test machine mem usage for last 7 days, there is no connection except redis-sentinel.
But Obviously, slow-growth never stops every day, higher and higher in the overall trend.

[mapleFU]

facebook/rocksdb#3216 (comment)

I think this is like the problem we meet. But I catch a cold and didn't have bandwitdh dive into it, maybe you can take a look

[xiaofan8421]

I'm sorry to hear that. Thank your 302 redirection. I have already taken a look at it carefully. Helpful but not explain everything.
Differently in my test machine, not only /proc/sys/vm/swappiness is zero, but also its cgroup.memory.swap is zero(namely no diverged).
Last but no least, I also think that this slow-growth memory is related with SST compaction when I collect some proofs before.

[kinoute]

I'm interested by this as well. We mainly switched from Redis to Kvrocks to reduce the memory usage and more importantly, if possible, to constraint it, no matter the decrease of throughput/performance.

With the default configuration (8 workers, block_cache_size to 4096, cache_index_and_filter_blocks to yes) and the compaction enabled, we can see that the memory usage keeps increasing. For around 48M keys, our container/pod in Kubernetes is up to 9.27Gi after two weeks with no sign of decrease.

Since we have limited the container to use 12Gi at best, we are a little concerned about it.

[kinoute]

@git-hulk We use 2.8.0 (the official Docker image) on Kubernetes 1.27. What's funny is that running htop in the container reports a RSS memory usage that seems "correct", same in the Grafana Kvrocks Dashboard (the RSS memory usage is around 5.2-5.6Gb max, it fluctuates but never goes more than that).

But the overall container uses more RAM:

'used_memory_rss_human': '5.18G',

(it does decrease sometimes, sorry about that). We use the compaction cron at 3am. I saw this issue about memory limit for connections: #2284. Not sure if it is related. Our number of clients is quite low, around 50 at the moment.

[git-hulk]

@kinoute Thanks for your detailed feedback. It seems like an expected value if the RSS(physical memory) value was around 5GiB. I'm unsure if it's consumed by the compaction behavior, can have a look at the LOGs file to see if any compaction happens during the high memory period.

[kinoute]

@git-hulk Not sure if it can help but here is the log from June 13th when the container had the highest memory usage and suddenly it dropped at around 3/4am. At the time we are using the bgsave-cron, which we are about to disable:

https://gist.github.com/kinoute/ad7cf83574b30c535b1d058422ac8dba

[git-hulk]

@kinoute Thank you! Based on your pictures, the RSS memory is in the expected range. And for the workig_set_bytes, this value includes the buffer/page cache, so I'm unsure if the memory reduce is caused by OS reclaiming the cache after compaction. It will be helpful if can monitor the cache size in OS, but it should be fine to high value of memory_working_set since it won't lead to OOM issue.

[kinoute]

@git-hulk Not sure about this. It seems Kubernetes is watching the container_memory_working_set_bytes metric to decide if it needs to OOMkill the process. (https://stackoverflow.com/a/66851497)

We can only hope that the amount of RAM used in total, related to the RSS usage, is indeed buffer/page cache that could be evicted when the memory pressure gets important (close to our limit). But we won't know until it reaches it.

[dbrkv]

I am experiencing the same issue. Kubernetes is reporting memory usage of 4.5Gi-5.5Gi for the container, which is growing by approximately 1Gi every day.

Container details:

• Image: apache/kvrocks:2.9.0
• CPU architecture: ARM (AWS Graviton)

Configuration used:

rocksdb.block_cache_size 256
rocksdb.metadata_block_cache_size 256
rocksdb.subkey_block_cache_size 256
rocksdb.share_metadata_and_subkey_block_cache yes

[dbrkv]

@git-hulk

I also configured lower k8s pod memory limits, let's see if it gets OOM killed or not.
I will post the status later.

result of cat /sys/fs/cgroup/memory.stat

anon 468709376
file 580464640
kernel 18677760
kernel_stack 311296
pagetables 1880064
sec_pagetables 0
percpu 0
sock 28672
vmalloc 0
shmem 0
zswap 0
zswapped 0
file_mapped 528384
file_dirty 499712
file_writeback 0
swapcached 0
anon_thp 0
file_thp 0
shmem_thp 0
inactive_anon 468660224
active_anon 16384
inactive_file 490569728
active_file 89894912
unevictable 0
slab_reclaimable 15986608
slab_unreclaimable 374080
slab 16360688
workingset_refault_anon 0
workingset_refault_file 15294
workingset_activate_anon 0
workingset_activate_file 5362
workingset_restore_anon 0
workingset_restore_file 5
workingset_nodereclaim 0
pgscan 17427
pgsteal 17417
pgscan_kswapd 667
pgscan_direct 16760
pgsteal_kswapd 657
pgsteal_direct 16760
pgfault 474521
pgmajfault 210
pgrefill 1219
pgactivate 17852
pgdeactivate 1219
pglazyfree 0
pglazyfreed 0
zswpin 0
zswpout 0
thp_fault_alloc 0
thp_collapse_alloc 0

[git-hulk]

@dbrkv It looks good from your information since Kvrocks only occupies 480mib anon memory.

[dbrkv]

I believe I've figured out what's happening with KVRocks' memory usage:
KVRocks appears to use maximum available memory, approaching 80-90% of pod resource limits. However, it doesn't trigger OOM kills. The system aggressively releases cached memory when nearing configured limits, preventing OOM situations.

[kinoute]

Yeah, I have seen the same behavior on my clusters after my previous messages, which is a good thing!

[ltagliamonte-dd]

i was looking at the same issue, but i haven't figured out how the system understand that's near the limit.
i don't see neither kvrocks nor rocksdb readying the cgroup limits.
Anyone any idea?