Compacting DB hangs indefinitely when done manually

[eksperimental]

Hi @lucaong
I notice compacting the DB with CubDB.compact/1 hangs indefinitely.
You can see how the .cub file changes over time and the .compact file reached a size an never moves from there.

$ ls -al
-rw-r--r-- 1 xxxx xxxx 78604311 Aug  3 06:11 12.cub
-rw-r--r-- 1 xxxx xxxx  1965076 Aug  3 06:06 15.compact

$ ls -al
-rw-r--r-- 1 xxxx xxxx 85419031 Aug  3 06:14 12.cub
-rw-r--r-- 1 xxxx xxxx  1965076 Aug  3 06:06 15.compact

iex> CubDB.compacting?(:the_db)
true

Let me know if you need any more information. I could create a sample project if needed.

Cheers

[eksperimental]

This is actually happening, when set_auto_compact/2 is used.

[lucaong]

Thanks for reporting @eksperimental , let me look into this.

If you do have a sample project or a test to reproduce it, that’s definitely very helpful, as I have been unable to reproduce this so far.

[lucaong]

One thing that strikes me as odd: the database file is called 12.cub, but the compacting file is called 15.compact. It should be called 13.compact instead. Are you sure it isn’t a leftover from a previous database that was using the same folder?

[lucaong]

Never mind what I wrote above about the file name, it simply means that the previous compactions didn’t complete by the time the main process was stopped, or crashed.

[lucaong]

Hi @eksperimental , I did not manage to reproduce this. Do you maybe have more info on how to trigger this issue? Possibly a specific sequence of actions? Thank you

[eksperimental]

Hi @lucaong
I was doing using CubDB to log some information, putting maybe 500+ entries to the DB, and I could see it was struggling to insert those records and they were being delayed.
The use case is withe entries being inserted at times, but them being 500+

[lucaong]

I tried reproducing it with tens of thousands of concurrent writes, but no luck. On what OS and CubDB version did you experience this issue? Do you use auto compaction or only manual?