Mounting existing GCP bucket fails

[Tristan-Kosciuch]

An existing bucket is not being properly mounted.

blueprint.yaml line

  - id: datafs
    source: modules/file-system/pre-existing-network-storage
    settings:
      remote_mount: dvm-dos-tem-outputs
      local_mount: /mnt/data
      fs_type: gcsfuse
      mount_options: defaults,_netdev,implicit_dirs

/etc/fstab

LABEL=cloudimg-rootfs   /        ext4   defaults        0 1
LABEL=UEFI      /boot/efi       vfat    umask=0077      0 1

10.0.0.218@tcp:/exacloud    /mnt/exacloud     lustre      defaults,_netdev  0 0
dvm-dos-tem-outputs   /mnt/data     gcsfuse     defaults,_netdev,implicit_dirs     0 0

sudo journalctl -o cat -u google-startup-scripts. relevant lines:

startup-script: INFO: Setting up mount (gcsfuse) dvm-dos-tem-outputs to /mnt/data
startup-script: INFO: Mount point '/mnt/data' was mounted.

ll /mnt

ls: cannot access '/mnt/data': Permission denied
total 16
drwxr-xr-x  5 root root 4096 Jan 18 21:08 ./
drwxr-xr-x 24 root root 4096 Jan 18 21:09 ../
d?????????  ? ?    ?       ?            ? data/
drwxr-xr-x  3 root root 4096 Jan 18 21:08 disks/
drwxr-xr-x  9 root root 4096 Jan 18 20:48 exacloud/

full blueprint.yaml

blueprint_name: slurm-lustre-dvmdostem-v5

vars:
  project_id:  ## GCP project ID ##

  deployment_name: slurm-lustre-dvmdostem-v5
  region: us-central1
  zone: us-central1-c
  lustre_mgs_ip: 10.0.0.218@tcp
  startup_timeouts: 300
  network_name: slurm-gcp-v5-net
  subnetwork_name: slurm-gcp-v5-primary-subnet
  family: dvmdostem-lustre-slurm-22-05-6-ubuntu
  project: spherical-berm-323321
  data_bucket: dvm-dos-tem-outputs
  
terraform_backend_defaults:
  type: gcs
  configuration:
    bucket: wcrc-tfstate-9486302

deployment_groups:
- group: primary
  modules:
  - id: network1
    source: modules/network/pre-existing-vpc

  - id: lustrefs
    source: modules/file-system/pre-existing-network-storage
    settings:
      server_ip: $(vars.lustre_mgs_ip)
      remote_mount: /exacloud
      local_mount: /mnt/exacloud
      fs_type: lustre

  - id: datafs
    source: modules/file-system/pre-existing-network-storage
    settings:
      remote_mount: dvm-dos-tem-outputs
      local_mount: /mnt/data
      fs_type: gcsfuse
      mount_options: defaults,_netdev,implicit_dirs

  - id: debug_node_group
    source: community/modules/compute/schedmd-slurm-gcp-v5-node-group
    settings:
      node_count_dynamic_max: 4
      machine_type: n2-highcpu-2
      instance_image:
        family: $(vars.family)
        project: $(vars.project)

  - id: debug_partition
    source: community/modules/compute/schedmd-slurm-gcp-v5-partition
    use:
    - network1
    - lustrefs
    - datafs
    - debug_node_group
    settings:
      partition_name: debug
      enable_placement: false
      is_default: true

  - id: compute_node_group
    source: community/modules/compute/schedmd-slurm-gcp-v5-node-group
    settings:
      node_count_dynamic_max: 20
      machine_type: "n2-highcpu-96"
      instance_image:
        family: $(vars.family)
        project: $(vars.project)
      # set the two following options to true to use spot VMs that are much less expensive but may be stopped at any time
      # preemptible: false
      # enable_spot_vm: false
      # spot_instance_config:
      #   termination_action: "STOP"

  - id: compute_partition
    source: community/modules/compute/schedmd-slurm-gcp-v5-partition
    use:
    - network1
    - lustrefs
    - datafs
    - compute_node_group
    settings:
      partition_name: compute

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v5-controller
    use:
    - network1
    - lustrefs
    - datafs
    - debug_partition
    - compute_partition
    settings:
      machine_type: n2-standard-2
      disk_type: pd-standard
      source_image_family: $(vars.family)
      source_image_project: $(vars.project)
      login_startup_scripts_timeout: $(vars.startup_timeouts)
      controller_startup_scripts_timeout: $(vars.startup_timeouts)
      compute_startup_scripts_timeout: $(vars.startup_timeouts)

  - id: slurm_login
    source: community/modules/scheduler/schedmd-slurm-gcp-v5-login
    use:
    - network1
    - slurm_controller
    settings:
      machine_type: n2-standard-2
      disk_type: pd-standard
      disable_login_public_ips: false
      source_image_family: $(vars.family)
      source_image_project: $(vars.project)

  - id: hpc_dashboard
    source: modules/monitoring/dashboard
    outputs: [instructions]

[cboneti]

You may be running into permissions issue. By default, the startup scripts run as root, and therefore gcsfuse is also invoked as root, which is not great. There are ways around this, but we chose not to implement them as the gcs fuse functionality targeted a few simple workflows, such as automated data staging. Are you running ls as root?
gcs fuse itself imposes several restrictions, for more information, please take a look at: https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/mounting.md#access-permissions

[Tristan-Kosciuch]

Thanks, this is very low priority for me so I'm just manually mounting the bucket