Delete not deletable known-broken backups

[vt-alt]

We got few backups failed, but they are not marked 'deletable'. I guess they could stay in retention process as reliable monthly/weekly backups. It would be a pity to have valid backups discarded, but to keep known failed backups.

How would you suggest to delete them?
Maybe you could add means to forcefully delete backups or to mark backups as broken so they are deleted when backups are rotated.

ps. JFYI failure is like this:

2021-09-09 17:45:18 +0300: burp[1543183] Begin phase2 (receive file data)
2021-09-10 06:20:10 +0300: burp[1543183] could not mkdir
/home/burp/nicevt.office.basealt.ru/working/data.tmp/t/root/tmp-stress-ng-dirdeep-6726-0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/
0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0
/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/
0/0/0/0/0/0/0/0/0/2021-09-10 06:20:10 +0300: burp[1543183] build path failed2021-09-10 06:20:10 +0300: burp[1543183] error in start_to_receive_new_file
2021-09-10 06:20:10 +0300: burp[1543183]   last tried file:
f:005B:/usr/include/c++/9/ext/pb_ds/detail/left_child_next_sibling_heap_/policy_access_fn_imps.hpp
2021-09-10 06:20:11 +0300: burp[1543183] End phase2 (receive file data)
2021-09-10 06:20:11 +0300: burp[1543183] error in backup phase 2
2021-09-10 06:20:11 +0300: burp[1543183] Backup failed

pps. This backup also quite slow, we have few users who report they consistently cannot backup, because burp performing very very slowly on their box.

[grke]

Hello,

The log you pasted seems to indicate that the problem is not old backups, it's a new backup that failed in phase2 because the client has some very long path that is exceeding some limit on the server side.
(incidentally, I thought this kind of long path issue was fixed in burp at some point, what burp versions are you using?)

For interrupted backups like that, you should probably be looking at the working_dir_recovery_method option.
From the man page:

       working_dir_recovery_method=[resume|delete]
              This option tells the server what to do when it finds the  work‐
              ing  directory of an interrupted backup (perhaps somebody pulled
              the plug on the server, or something). This can be overridden by
              the  client configurations files in clientconfdir on the server.
              Options are...

       delete: Just delete the old working directory.

       resume: Continue the previous backup from the point at  which  it  left
       off.  NOTE: If the client has changed its include/exclude configuration
       since the backup was interrupted, the recovery  method  will  automati‐
       cally switch to 'delete'. See also the 'resume attempts' option.

If I have something like this from 'burp -a l', where 4 has been interrupted, and you have 'working_dir_recovery_method=delete', it will be deleted the next time the client tries to backup, and a new 4 will be created. You can also try 'burp -a delete -b 4' to do it immediately.

Backup: 0000001 2021-08-10 07:46:47 +1000 (deletable)
Backup: 0000002 2021-08-29 09:49:19 +1000
Backup: 0000003 2021-09-12 19:01:25 +1000
Backup: 0000004 2021-09-12 19:03:15 +1000 (working)

If you really do want to delete old, complete backups, then this page describes what makes a backup deletable: https://burp.grke.org/docs/retention.html

A backup is 'deletable' if it is the oldest backup, or comes immediately after
a 'hardlinked' backup. This is explained further in the section about multiple
'keep' values, below.

If the backup you want to delete does not meet those conditions, you would need to delete the intervening deletable backups until the one that you want to remove meets the conditions.
eg, if it is the second oldest backup, you need to delete the oldest backup first

[vt-alt]

Thanks for your detailed answer. We always use working_dir_recovery_method = delete but this seems not worked in our case as failed backups hanged forever in working state (if looked in burp -as). For example we have failed backup 6 7 8 (9 is good) but they are still visible in backups list.

Backup: 0000001 2019-12-09 03:16:27 +0300 (deletable)
Backup: 0000003 2020-02-15 19:17:04 +0300 (deletable)
Backup: 0000004 2020-03-12 20:40:05 +0300
Backup: 0000005 2020-03-16 17:36:31 +0300
Backup: 0000006 2021-01-10 14:12:33 +0300
Backup: 0000007 2021-06-02 17:34:09 +0300
Backup: 0000008 2021-09-09 17:39:16 +0300
Backup: 0000009 2021-09-10 15:23:33 +0300

As I understand your idea I should just sequentially delete all backups, this is almost equivalent to starting backups afresh for this box. Perhaps burp would benefit from option of making some particular backup deletable (for recovery or cleaning purposes).

[vt-alt]

Well, I deleted everything (except 1 and 9) and started new backup. It's failed with

2021-09-15 21:42:37 +0300: burp[452241] Backup started
2021-09-15 21:42:37 +0300: burp[452241] Client version: 2.3.38
2021-09-15 21:42:37 +0300: burp[452241] Protocol: 1
2021-09-15 21:42:37 +0300: burp[452241] Using librsync hash md4
2021-09-15 21:42:37 +0300: burp[452241] WARNING: Client 'nicevt.office.basealt.ru' version '2.3.38' does not match server version '2.3.30'. An upgrade is
recommended.
2021-09-15 21:42:37 +0300: burp[452241] Begin phase1 (file system scan)
2021-09-15 21:43:17 +0300: burp[452241] WARNING: Will not descend: file system change not allowed /proc: No such file or directory
2021-09-15 21:44:08 +0300: burp[452241] End phase1 (file system scan)
2021-09-15 21:44:08 +0300: burp[452241] Begin phase2 (receive file data)
2021-09-15 21:45:03 +0300: burp[452241] End phase2 (receive file data)
2021-09-15 21:45:03 +0300: burp[452241] Backup ending - disconnect from client.
2021-09-15 21:45:03 +0300: burp[452241] Begin phase3 (merge manifests)
2021-09-15 21:46:07 +0300: burp[452241] End phase3 (merge manifests)
2021-09-15 21:46:07 +0300: burp[452241] Begin phase4 (shuffle files)
2021-09-15 21:46:07 +0300: burp[452241] Previous backup is not a hardlinked_archive
2021-09-15 21:46:07 +0300: burp[452241]  will generate reverse deltas
2021-09-15 21:46:07 +0300: burp[452241] Duplicating current backup.
2021-09-15 21:57:29 +0300: burp[452241] First keep value: 7, backup: 10 (10-1=9)
2021-09-15 21:57:29 +0300: burp[452241] New backup is not a hardlinked_archive (9%7=2)
2021-09-15 21:57:29 +0300: burp[452241] Doing the atomic data jiggle...
2021-09-15 22:01:57 +0300: burp[452241] opendir /home/burp/nicevt.office.basealt.ru/deleteme/nicevt.office.basealt.ru-0000007 2021-06-02 17:34:09
+0300/data.tmp/t/root/tmp-stress-ng-dirdeep-6726-0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/
0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0
/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/2021-09-15 22:01:57 +0300: burp[452241] error in backup phase 4
2021-09-15 22:01:57 +0300: burp[452241] Backup failed

Before backup start there was no /root/tmp-* files (which I deleted long ago). Now I wonder that backup state. burp -al:

Backup: 0000001 2019-12-09 03:16:27 +0300 (deletable)
Backup: 0000009 2021-09-10 15:23:33 +0300 (deletable)
Backup: 0000010 2021-09-15 21:42:37 +0300 (finishing)

It says finishing but there is nothing happening on the sever with it. What to do?

[grke]

Hello,

It seems like that very long path is causing problems.
Is that path in backup 9?
Is that path in backup 10?

I would try:
Excluding that data.tmp directory from the backup configuration for that client.
Editing the manifest in backup 9 to remove that long entry, if it exists.
Editing the manifest in backup 10 to remove that long entry, if it exists.

If the manifest is gzipped, you will need to gunzip it first:
gunzip manifest.gz
Then edit it, and when you are done:
gzip manifest

When you edit the manifest, each file entry will look something like this, with 4 lines. The entry starts with a line beginning with 't' and ends with a line beginning with 'x':

t0018t/home/graham/testdirx/c
r003AgB pAWd IGk B A A A P BAA I BhEaE6 BhEaE6 BhEaE6 A A J A A
f0017/home/graham/testdirx/c
x002315:e1a858bfab68b83ca1cbbdd8b8f8ffd6

Directories have 2 lines, like this, starting with an 'r' line, ending with a 'd' line:

r003CgB pB6/ EHt D A A A BAA BAA I BhNz5Q BhKstz BhKstz A A J A A
d0015/home/graham/testdirx