Formatting to mount existing DDN exascaler?

[Tristan-Kosciuch]

I find the documentation unclear on how to use the pre-existing-network-storage module with the DDN exascaler. @heyealex an example would likely help me. When I deploy my application the wrong lustre IP address is being used and the lustre filesystem is not mounted. This is what I am including in my yaml blueprint:

vars:
  lustre_mgs_ip: 10.0.0.218

- group: primary
  modules:
  - id: lustrefs
    source: modules/file-system/pre-existing-network-storage
    settings:
      server_ip: $(vars.lustre_mgs_ip)
      remote_mount: /exacloud
      local_mount: /mnt/exacloud
      fs_type: lustre

From the variables.tf file of the pre-existing-network-storage module I see the format for server_ip should be <MGS NID>[:<MGS NID>]. The remote_mount should also be the filesystem name? Any help is appreciated.

I can manually mount the filesystem using the following code so I know the exascaler is working:

#!/bin/bash

LUSTREIP=10.0.0.218 # this is the IP address of the mgs node

cat > /etc/esc-client.cfg << EOF
{
    "Version": "2.0.0",
    "MountConfig": {
        "ClientDevice": "$LUSTREIP@tcp:/exacloud",
        "Mountpoint": "/mnt/exacloud",
        "PackageSource": "http://$LUSTREIP/client-packages"
    }
}
EOF
curl -fsSL http://$LUSTREIP/client-setup-tool -o /usr/sbin/esc-client
chmod +x /usr/sbin/esc-client
esc-client auto setup --config /etc/esc-client.cfg

[heyealex]

Hello Tristan -
I ran a quick test to determine what is produced by the DDN EXAScaler module, and got the following valid network storage output:

network_storage_ddn = {
  "client_install_runner" = # Removed for clarity
  "fs_type" = "lustre"
  "local_mount" = "/scratch"
  "mount_options" = ""
  "mount_runner" = {
    "content" = "mkdir -p /scratch && mount -t lustre 10.0.0.4@tcp:/exacloud /scratch"
    "destination" = "mount-ddn-lustre.sh"
    "type" = "shell"
  }
  "remote_mount" = "/exacloud"
  "server_ip" = "10.0.0.4@tcp"
}

I believe it's the @tcp after the IP that you are missing in your example. This should definitely be documented better, and I'll make a task for us to do that in the near future. Thanks for bringing this to our attention!

I hope this helps get you unblocked! Let me know if I can clarify anything else or if this does not work for you.

[Tristan-Kosciuch]

I terraform destroy then clear the directory and recreate it with ghpc whenever I make a change to my yaml. Just noticed the overwrite flag, that's less work but not the issue here.

terraform.tfvars The correct lustre IP is here.

project          = "spherical-berm-323321"
region           = "us-central1"
lustre_mgs_ip    = "10.0.0.218@tcp"
subnetwork_name  = "slurm-gcp-v5-primary-subnet"
project_id       = "spherical-berm-323321"
startup_timeouts = 300
network_name     = "slurm-gcp-v5-net"
family           = "dvmdostem-lustre-slurm-22-05-6-ubuntu"
zone             = "us-central1-c"
deployment_name  = "slurm-lustre-dvmdostem-v5"
labels = {
  ghpc_blueprint  = "slurm-lustre-dvmdostem-v5"
  ghpc_deployment = "slurm-lustre-dvmdostem-v5"
}

/etc/fstab Aha..

LABEL=cloudimg-rootfs   /        ext4   defaults        0 1
LABEL=UEFI      /boot/efi       vfat    umask=0077      0 1
10.0.0.214@tcp0:/exacloud       /mnt/exacloud   lustre  auto,_netdev,user_xattr 0       0

10.0.0.218@tcp:/exacloud    /mnt/exacloud     lustre      defaults,_netdev  0 0

Could deleting /etc/fstab and recreating it correctly as part of my packer build solve the problem?

[heyealex]

I ran into the same thing when recreating. I believe the duplicate mount line for /mnt/exacloud was created when you created the image with the DDN network storage attached.

Deleting the line from /etc/fstab in the image should work, which can be done in a couple of ways:

1. Update the image creation blueprint to include a startup script runner (or additional script) that removes the line.
2. (Short term fix) Create a temporary VM from the image, delete the line, shut down the VM and create a new image in the same family from the VMs disk.

I'll try out (1) and send over an updated blueprint that includes the added runner when I get a chance to test it out. In the meantime, you can try on your end or test this out using (2) to unblock you.

[heyealex]

I updated the startup script module in your create image blueprint, just adding one line to remove the existing /mnt/exascaler line:

- id: mount-at-startup
    source: modules/scripts/startup-script
    settings:
      runners:
      - $(tmplustrefs.install_ddn_lustre_client_runner)
      - type: shell
        destination: "clean_up.sh"
        content: |
          #!/bin/bash
          esc-client unmount -m "/mnt/exacloud"
          rm /etc/esc-client.conf
          sed -i '/.*\/mnt\/exacloud.*/d' /etc/fstab
    outputs: [startup_script]

With an image created using that startup script, I was able to deploy the slurm cluster with DDN EXAscaler mounted (some content removed for brevity):

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute      up   infinite     20  idle~ slurmlustr-compute-ghpc-[0-19]
debug*       up   infinite      4  idle~ slurmlustr-debug-ghpc-[0-3]

$ df
Filesystem                       1K-blocks     Used Available Use% Mounted on
.
10.0.0.10@tcp:/exacloud          517014716     1252 511754200   1% /mnt/exacloud
slurmlustr-controller:/opt/apps   50620416 13930496  36673536  28% /opt/apps
slurmlustr-controller:/home       50620416 13930496  36673536  28% /home
slurmlustr-controller:/etc/munge  50620416 13930496  36673536  28% /etc/munge
.

Let me know if that works for you.

[Tristan-Kosciuch]

This worked, thank you for quick turnaround!

[heyealex]

Good to hear! I'm glad we could help.