allow to exclude previews from backup upon instance backup

Follow up of: #5550

Containers/borgbackup/backupscript.sh

@@ -193,6 +193,13 @@ if [ "$BORG_MODE" = backup ]; then
     # Exclude the nextcloud log and audit log for GDPR reasons
     BORG_EXCLUDE=(--exclude "/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/nextcloud.log*" --exclude "/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/audit.log")
 
+    # Exclude previews from backup if selected to speed up process
+    ADDITIONAL_BORG_EXCLUDES=()
+    if [ -n "$BACKUP_EXCLUDE_PREVIEWS" ]; then
+        ADDITIONAL_BORG_EXCLUDES=(--exclude "sh:nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/appdata_*/preview/**")
+        echo "Excluding previews from backup"
+    fi
+
     # Make sure that there is always a borg.config file before creating a new backup
     if ! [ -f "/nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/borg.config" ]; then
         echo "Did not find borg.config file in the mastercontainer volume."
@@ -203,7 +210,7 @@ if [ "$BORG_MODE" = backup ]; then
     # Create the backup
     echo "Starting the backup..."
     get_start_time
-    if ! borg create "${BORG_OPTS[@]}" "${BORG_EXCLUDE[@]}" "::$CURRENT_DATE-nextcloud-aio" "/nextcloud_aio_volumes/" --exclude-from /borg_excludes; then
+    if ! borg create "${BORG_OPTS[@]}" "${BORG_EXCLUDE[@]}" "::$CURRENT_DATE-nextcloud-aio" "/nextcloud_aio_volumes/" --exclude-from /borg_excludes "${ADDITIONAL_BORG_EXCLUDES[@]}"; then
         echo "Deleting the failed backup archive..."
         borg delete --stats "::$CURRENT_DATE-nextcloud-aio"
         echo "Backup failed!"

compose.yaml

@@ -18,6 +18,7 @@ services:
       # APACHE_PORT: 11000 # Is needed when running behind a web server or reverse proxy (like Apache, Nginx, Caddy, Cloudflare Tunnel and else). See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
       # APACHE_IP_BINDING: 127.0.0.1 # Should be set when running behind a web server or reverse proxy (like Apache, Nginx, Caddy, Cloudflare Tunnel and else) that is running on the same host. See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
       # APACHE_ADDITIONAL_NETWORK: frontend_net # (Optional) Connect the apache container to an additional docker network. Needed when behind a web server or reverse proxy (like Apache, Nginx, Caddy, Cloudflare Tunnel and else) running in a different docker network on same server. See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
+      # BACKUP_EXCLUDE_PREVIEWS: false # Exclude previews when performing backups in order to speed up the backup process
       # BORG_RETENTION_POLICY: --keep-within=7d --keep-weekly=4 --keep-monthly=6 # Allows to adjust borgs retention policy. See https://github.com/nextcloud/all-in-one#how-to-adjust-borgs-retention-policy
       # COLLABORA_SECCOMP_DISABLED: false # Setting this to true allows to disable Collabora's Seccomp feature. See https://github.com/nextcloud/all-in-one#how-to-disable-collaboras-seccomp-feature
       # NEXTCLOUD_DATADIR: /mnt/ncdata # Allows to set the host directory for Nextcloud's datadir. ⚠️⚠️⚠️ Warning: do not set or adjust this value after the initial Nextcloud installation is done! See https://github.com/nextcloud/all-in-one#how-to-change-the-default-location-of-nextclouds-datadir

php/containers.json

@@ -524,6 +524,7 @@
         "BORG_MODE=%BORGBACKUP_MODE%",
         "SELECTED_RESTORE_TIME=%SELECTED_RESTORE_TIME%",
         "RESTORE_EXCLUDE_PREVIEWS=%RESTORE_EXCLUDE_PREVIEWS%",
+        "BACKUP_EXCLUDE_PREVIEWS=%BACKUP_EXCLUDE_PREVIEWS%",
         "BACKUP_RESTORE_PASSWORD=%BACKUP_RESTORE_PASSWORD%",
         "ADDITIONAL_DIRECTORIES_BACKUP=%ADDITIONAL_DIRECTORIES_BACKUP%",
         "BORGBACKUP_HOST_LOCATION=%BORGBACKUP_HOST_LOCATION%",

php/public/index.php

@@ -110,6 +110,7 @@
         'borg_restore_password' => $configurationManager->GetBorgRestorePassword(),
         'daily_backup_time' => $configurationManager->GetDailyBackupTime(),
         'is_daily_backup_running' => $configurationManager->isDailyBackupRunning(),
+        'are_previews_excluded_from_backups' => $configurationManager->arePreviewsExcludedFromBackups(),
         'timezone' => $configurationManager->GetTimezone(),
         'skip_domain_validation' => $configurationManager->shouldDomainValidationBeSkipped(),
         'talk_port' => $configurationManager->GetTalkPort(),

php/src/Controller/ConfigurationController.php

@@ -41,6 +41,12 @@ public function SetConfig(Request $request, Response $response, array $args) : R
                 $this->configurationManager->SetBorgRestoreLocationVarsAndPassword($restoreLocation, $borgRemoteRepo, $borgPassword);
             }
 
+            if (isset($request->getParsedBody()['backup_exclude_previews'])) {
+                $this->configurationManager->SetBackupExcludePreviewsEnabledState('true');
+            } else {
+                $this->configurationManager->SetBackupExcludePreviewsEnabledState('false');
+            }
+
             if (isset($request->getParsedBody()['daily_backup_time'])) {
                 if (isset($request->getParsedBody()['automatic_updates'])) {
                     $enableAutomaticUpdates = true;

php/src/Controller/DockerController.php

@@ -79,13 +79,18 @@ public function GetLogs(Request $request, Response $response, array $args) : Res
     }
 
     public function StartBackupContainerBackup(Request $request, Response $response, array $args) : Response {
-        $this->startBackup();
+        $this->startBackup($request);
         return $response->withStatus(201)->withHeader('Location', '/');
     }
 
-    public function startBackup() : void {
+    public function startBackup($request) : void {
         $config = $this->configurationManager->GetConfig();
         $config['backup-mode'] = 'backup';
+        if (isset($request->getParsedBody()['backup_exclude_previews'])) {
+            $config['backup_exclude_previews'] = true;
+        } else {
+            $config['backup_exclude_previews'] = false;
+        }
         $this->configurationManager->WriteConfig($config);
 
         $id = self::TOP_CONTAINER;

php/src/Data/ConfigurationManager.php

@@ -120,6 +120,23 @@ public function GetBackupTimes() : array {
         return $backupTimes;
     }
 
+    public function GetBackupExcludePreviews() : string {
+        $envVariableName = 'BACKUP_EXCLUDE_PREVIEWS';
+        $configName = 'backup_exclude_previews';
+        $defaultValue = 'false';
+        return $this->GetEnvironmentalVariableOrConfig($envVariableName, $configName, $defaultValue);
+    }
+
+    public function arePreviewsExcludedFromBackups() : bool {
+        return $this->GetBackupExcludePreviews() === 'true';
+    }
+
+    public function SetBackupExcludePreviewsEnabledState(string $value) : void {
+        $config = $this->GetConfig();
+        $config['backup_exclude_previews'] = $value;
+        $this->WriteConfig($config);
+    }
+
     public function wasStartButtonClicked() : bool {
         if (isset($this->GetConfig()['wasStartButtonClicked'])) {
             return true;

php/src/Docker/DockerActionManager.php

@@ -275,6 +275,8 @@ public function CreateContainer(Container $container) : void {
                     $replacements[1] = $this->configurationManager->GetSelectedRestoreTime();
                 } elseif ($out[1] === 'RESTORE_EXCLUDE_PREVIEWS') {
                     $replacements[1] = $this->configurationManager->GetRestoreExcludePreviews();
+                } elseif ($out[1] === 'BACKUP_EXCLUDE_PREVIEWS') {
+                    $replacements[1] = $this->configurationManager->GetBackupExcludePreviews();
                 } elseif ($out[1] === 'APACHE_PORT') {
                     $replacements[1] = $this->configurationManager->GetApachePort();
                 } elseif ($out[1] === 'TALK_PORT') {

php/templates/containers.twig

@@ -169,7 +169,19 @@
                                                     <option value="{{ restore_time }}">{{ restore_time }} UTC</option>
                                                 {% endfor %}
                                             </select><br>
-                                            <input type="checkbox" id="restore-exclude-previews" name="restore-exclude-previews"><label for="restore-exclude-previews">Exclude previews from restore which will speed up the restore process but will trigger a scan of the preview folder as soon as the Nextcloud container starts the next time</label><br>
+                                            <input
+                                                type="checkbox"
+                                                id="restore-exclude-previews"
+                                                name="restore-exclude-previews"
+                                                {% if are_previews_excluded_from_backups == true %}
+                                                    {# if previews are excluded from daily backups then the user will probably want to exclude them from the restore too #}
+                                                    checked="checked"
+                                                    data-initial-state="true"
+                                                {% else %}
+                                                    data-initial-state="false"
+                                                {% endif %}
+                                            >
+                                            <label for="restore-exclude-previews">Exclude previews from restore which will speed up the restore process but will trigger a scan of the preview folder as soon as the Nextcloud container starts the next time</label><br>
                                             <input type="submit" value="Restore selected backup"/>
                                         </form>
                                     {% endif %}
@@ -483,6 +495,18 @@
                                         <form method="POST" action="/api/docker/backup" class="xhr">
                                             <input type="hidden" name="{{csrf.keys.name}}" value="{{csrf.name}}">
                                             <input type="hidden" name="{{csrf.keys.value}}" value="{{csrf.value}}">
+                                            <input
+                                                type="checkbox"
+                                                id="backup_exclude_previews"
+                                                name="backup_exclude_previews"
+                                                {% if are_previews_excluded_from_backups == true %}
+                                                    checked="checked"
+                                                    data-initial-state="true"
+                                                {% else %}
+                                                    data-initial-state="false"
+                                                {% endif %}
+                                            >
+                                            <label for="backup_exclude_previews">Exclude previews from backup which will speed up the backup process</label><br>
                                             <input type="submit" value="Create backup" onclick="return confirm('Create backup? Are you sure that you want to create a backup? This will stop all running containers and create the backup.')" />
                                         </form>
 
@@ -533,6 +557,18 @@
                                                     <input type="hidden" name="{{csrf.keys.name}}" value="{{csrf.name}}">
                                                     <input type="hidden" name="{{csrf.keys.value}}" value="{{csrf.value}}">
                                                     <input type="submit" value="Submit backup time" /><br>
+                                                    <input
+                                                        type="checkbox"
+                                                        id="daily_backup_exclude_previews"
+                                                        name="backup_exclude_previews"
+                                                        {% if are_previews_excluded_from_backups == true %}
+                                                            checked="checked"
+                                                            data-initial-state="true"
+                                                        {% else %}
+                                                            data-initial-state="false"
+                                                        {% endif %}
+                                                    >
+                                                    <label for="daily_backup_exclude_previews">Exclude previews from daily backups which will speed up the backup process</label><br>
                                                     <input type="checkbox" id="automatic_updates" name="automatic_updates" checked="checked"><label for="automatic_updates">Automatically update all containers, the mastercontainer and on saturdays your Nextcloud apps</label><br>
                                                     <input type="checkbox" id="success_notification" name="success_notification" checked="checked"><label for="success_notification">Send notifications about successful backups (notifications about unsuccessful backups will always be sent)</label>
                                                 </form>

readme.md

@@ -429,7 +429,9 @@ Be aware that this solution does not back up files and folders that are mounted
 ---
 
 #### What is getting backed up by AIO's backup solution?
-Backed up will get all important data of your Nextcloud AIO instance like the database, your files and configuration files of the mastercontainer and else. Files and folders that are mounted into Nextcloud using the external storage app are not getting backed up. There is currently no way to exclude the data directory because it would require hacks like running files:scan and would make the backup solution much more unreliable (since the database and your files/folders need to stay in sync). If you still don't want your datadirectory to be backed up, see https://github.com/nextcloud/all-in-one#how-to-enable-automatic-updates-without-creating-a-backup-beforehand for options (there is a hint what needs to be backed up in which order).
+Backed up will get all important data of your Nextcloud AIO instance like the database, your files and configuration files of the mastercontainer and else. Previews are backed up by default but if you want to speed up the backup process you can disable them by configuring it in AIO. Alternatively by adding `--env BACKUP_EXCLUDE_PREVIEWS=true` to the docker run command of the mastercontainer (but before the last line `nextcloud/all-in-one:latest`! If it was started already, you will need to stop the mastercontainer, remove it (no data will be lost) and recreate it using the docker run command that you initially used).
+
+Files and folders that are mounted into Nextcloud using the external storage app are not getting backed up. There is currently no way to exclude the data directory because it would require hacks like running files:scan and would make the backup solution much more unreliable (since the database and your files/folders need to stay in sync). If you still don't want your datadirectory to be backed up, see https://github.com/nextcloud/all-in-one#how-to-enable-automatic-updates-without-creating-a-backup-beforehand for options (there is a hint what needs to be backed up in which order).
 
 #### How to adjust borgs retention policy?
 The built-in borg-based backup solution has by default a retention policy of `--keep-within=7d --keep-weekly=4 --keep-monthly=6`. See https://borgbackup.readthedocs.io/en/stable/usage/prune.html for what these values mean. You can adjust the retention policy by providing `--env BORG_RETENTION_POLICY="--keep-within=7d --keep-weekly=4 --keep-monthly=6"` to the docker run command of the mastercontainer (but before the last line `nextcloud/all-in-one:latest`! If it was started already, you will need to stop the mastercontainer, remove it (no data will be lost) and recreate it using the docker run command that you initially used) and customize the value to your fitting. ⚠️ Please make sure that this value is valid, otherwise backup pruning will bug out!

tests/QA/060-environmental-variables.md

@@ -11,6 +11,7 @@
 - [ ] When starting the mastercontainer with `--env NEXTCLOUD_UPLOAD_LIMIT=11G` it should change Nextclouds upload limit to 11G. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-upload-limit-for-nextcloud for allowed values.
 - [ ] When starting the mastercontainer with `--env NEXTCLOUD_MEMORY_LIMIT=1024M` it should change Nextclouds PHP memory limit to 1024M. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud for allowed values.
 - [ ] When starting the mastercontainer with `--env NEXTCLOUD_MAX_TIME=4000` it should change Nextclouds upload max time 4000s. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-max-execution-time-for-nextcloud for allowed values.
+- [ ] When starting the mastercontainer with `--env BACKUP_EXCLUDE_PREVIEWS=true` it should exclude backing up previews when creating a backup.
 - [ ] When starting the mastercontainer with `--env BORG_RETENTION_POLICY="--keep-within=1d --keep-weekly=1 --keep-monthly=1"` it should change borgs retention policy to the defined one. This can be checked when creating a backup and looking at the logs.
 - [ ] When starting the mastercontainer with `--env WATCHTOWER_DOCKER_SOCKET_PATH="$XDG_RUNTIME_DIR/docker.sock"` it should map `$XDG_RUNTIME_DIR/docker.sock` to `/var/run/docker.sock` inside the watchtower container which allow to update the mastercontainer on docker rootless.
 - [ ] When starting the mastercontainer with `--env AIO_DISABLE_BACKUP_SECTION=true` it should hide the backup section that gets shown after AIO is set up (everything of [020-backup-and-restore](./020-backup-and-restore.md)) and simply show that the backup section is disabled.