create-needed-for-boot-dirs fails to umount properly

[collinarnett]

Problem

I am new to using impermanence and I observed the following error with the create-needed-for-boot-dirs service

Oct 22 18:50:04 azathoth systemd[1]: Starting create-needed-for-boot-dirs.service...
Oct 22 18:50:04 azathoth create-needed-for-boot-dirs-start[1340]: umount: /persist-tmp-mnt/persist: target is busy.
Oct 22 18:50:04 azathoth systemd[1]: create-needed-for-boot-dirs.service: Main process exited, code=exited, status=32/n/a
Oct 22 18:50:04 azathoth systemd[1]: create-needed-for-boot-dirs.service: Failed with result 'exit-code'.
Oct 22 18:50:04 azathoth systemd[1]: Failed to start create-needed-for-boot-dirs.service.

Configuration

I am using disko and zfs with native encryption to setup my disks. Given that I am using native encryption I cannot use postDeviceCommands with impermanence because my drives must be first unencrypted to rollback. Instead of postDeviceCommands I am using initrd.systemd. My service is defined below.

    initrd.systemd = {
      enable = true;
      services.rollback = {
        description = "Rollback ZFS datasets to a pristine state";
        serviceConfig.Type = "oneshot";
        unitConfig.DefaultDependencies = "no";
        wantedBy = ["initrd.target"];
        after = ["zfs-import-zroot.service"];
        requires = ["zfs-import-zroot.service"];
        before = ["sysroot.mount"];
        path = with pkgs; [zfs];
        script = ''
          zfs rollback -r zroot/root@empty && echo "rollback complete"
        '';
      };
      services.create-needed-for-boot-dirs = {
        after = pkgs.lib.mkForce ["rollback.service"];
        wants = pkgs.lib.mkForce ["rollback.service"];
      };
    };

As you can see I also added after and wants to create-needed-for-boot-dirs in case the rollback service was conflicting with create-needed-for-boot-dirs's ability to umount.

Here is my impermanence configuration:

{
  environment.persistence."/persist" = {
    directories = [
      "/var/log"
      "/var/lib/libvirt"
      "/var/lib/nixos"
      "/var/lib/systemd/coredump"
    ];
  };
  users.groups.multimedia = {};
  environment.persistence."/persist/save" = {
    directories = [
      {
        directory = "/media";
        group = "multimedia";
        mode = "0770";
      }
    ];
    users.collin = {
      directories = [
        "brew"
        "projects"
        "work_projects"
        "Downloads"
        "Pictures"
        "Documents"
        "Videos"
        {
          directory = ".config/sops/age/";
          mode = "0700";
        }
        {
          directory = ".ssh";
          mode = "0700";
        }
        ".local/share/direnv"
      ];
    };
  };
}

It might also be relevant to show how my zfs datasets are defined:

        datasets = {
          persist = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/persist";
            options."com.sun:auto-snapshot" = "false";
            postCreateHook = "zfs snapshot zroot/persist@empty";
          };
          # dataset where all files that should both persist and need to be backed up are stored
          # the Impermanence module is what ensures files are correctly stored here
          persistSave = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/persist/save";
            options."com.sun:auto-snapshot" = "false";
            postCreateHook = "zfs snapshot zroot/persistSave@empty";
          };
          # Nix store etc. Needs to persist, but doesn't need backed up
          nix = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            mountpoint = "/nix";
            options = {
              atime = "off";
              canmount = "on";
              "com.sun:auto-snapshot" = "false";
            };
            postCreateHook = "zfs snapshot zroot/nix@empty";
          };
          root = {
            type = "zfs_fs";
            options.mountpoint = "legacy";
            options."com.sun:auto-snapshot" = "false";
            mountpoint = "/";
            postCreateHook = ''
              zfs snapshot zroot/root@empty
            '';
          };
        };

My entire machine config can be found here https://github.com/collinarnett/brew/tree/main/hosts/azathoth

Attempted fixes

As I eluded to earlier, I tried to make sure that create-needed-for-boot-dirs doesn't conflict with the rollback service but I still have the same error.
I also found this commit where the author explains

The generated script for the service unmounts /persist before /persist/cache and therefore errors out with a "target is busy" error.

This seems to imply that the order of unmounting matters but I'm not enough of an expert to determine a fix.

[name-snrl]

Hi. I also started using impermanence with zfs not too long ago. When I was looking for a non-lts kernel for my system that would also be compatible with zfs, I came across https://github.com/chaotic-cx/nyx. This repo also has a great solution for your purposes:

https://github.com/chaotic-cx/nyx/blob/main/modules/nixos/zfs-impermanence-on-shutdown.nix

kernel+zfs definition in my cfg:

https://github.com/name-snrl/nixos-configuration/blob/b3d3366d6c2f446b45154ad9d614114f94be6497/modules/nixos/profiles/desktop/default.nix#L9
https://github.com/name-snrl/nixos-configuration/blob/b3d3366d6c2f446b45154ad9d614114f94be6497/modules/nixos/profiles/zfs.nix#L81