No-Lock database backup and restore

[lvca]

SUMMARY

Status: Completed
Progress: Design 100% -> Development & Unit Tests 100% -> Integration Tests 100%
Currently Assigned: @lvca
ETA: 5 days
From: Ideas []
To: Issues [#70]

---

ArcadeDB stored the database on the file system using one directory per database. If it's running as a server, the database directories are under the databases directory of the ArcadeDB package.

Before the more sophisticated Delta Backup (a new discussion about this soon), we need a full backup and restore that avoid blocking the execution of transactions.

Requirements:

• ArcadeDB uses "txlog" files for transactions. No need to save in the backup
• ArcadeDB writes the changes to "txlog" first and then updates the actual database pages in an asynchronous way. During the backup, those pending pages in RAM must be all flushed to disk
• A new lock must be acquired to the database preventing concurrent changes to the schema/indexes. Right now the EmbeddedDatabase has an RW lock, but we need something that supports different operations:
• write lock: no concurrent access to the instance. This is typically used when the schema is changed
• read lock: this is acquired by pretty much every operation, even a transaction commit
• backup lock: this must block acquiring a write lock, and at the same time should still allow acquiring read locks
• During the backup, the flushing of the pages to disk must be suspended. This doesn't impact the database reliability, because all the changes are still in the txlog
• Transactions can still run in parallel

The actual backup operation will be creating a zip file with the content of the files on the filesystems (all but txlogs that can be modified during the backup).

[timw]

IMO Cassandra does a lot right in this space (although being completely log oriented it has advantages) by utilising platform features, and perhaps ArcadeDB could do something similar.

For transaction log backup, if enabled, each segment can simply be hard linked into a backup folder when it is "complete" - an external backup shipping process can then copy these into persistent storage and remove the hard links.

For a full database backup, the backup process could be a simple as "safepoint for backup", volume snapshot, "restore full operation", meaning the period where the database isn't fully operational is minimal. An external process can then compress, offload and remove the snapshot as desired. The upside of this approach is it can work with whatever the local platform tech is - e.g. LVM , BTRFS/ZFS, Amazon EBS etc. could all work with this approach.

Just some ideas to consider, and mesh well with common operational approaches/tooling - I'm not up to speed enough with the ArcadeDB data store to know what the best overall approach is.

[lvca]

Thanks, @timw for pointing that out. We have LSM indexes where we could totally follow the same strategy, but database files are mutable on disk, so it won't work the same approach.

[lvca]

This has been implemented and released in v21.9.1: #70