Containers running in AKS can't write to lustre if the user is not root.

[pinduzera]

Is your feature request related to a problem?/Why is this needed
Can't write to lustre filesystem in AKS with containers running in any other user that is not root.

Describe the solution you'd like in detail
Allow write options for containers running under distinct users.

Describe alternatives you've considered
Tried all the squash permissions with distinct PID/GID and IP considerations, nothing have worked. The containers were able to read, but no write

Additional context
The documentation has an example but that assumes the container is running as the root user, which some enterprises can't use as security policy. The documentation is also not clear if there is a way how to handle the permissions for the filesystem

[dabradley]

@pinduzera Thanks for bringing this up. I was able to see similar behavior in some circumstances. I will explain what is going on with the driver and what can be done to improve this behavior.

The ownership/permissions that are exposed to a pod running as a non-root user is, by default, the same as it would be if you mounted the Lustre cluster manually with the client. So, if the permissions on the files in the Lustre cluster are correct for your user, it will work as expected.

However, it does appear that an AMLFS (Azure Managed Lustre Filesystem) cluster is first created upon deployment with an empty root:root owned top-level of the filesystem. So, one way to avoid this being an issue for your pods is to ensure that the AMLFS cluster has been changed to have the uid and/or gid that you are expecting for the pod user. This can be done manually with chown or similar command after any type of mounting, whether manually from a VM within Azure, or via an init container or single run of an AKS job.

This will work for cases where the user is already manually creating the AMLFS cluster and performing other management tasks with it, but I understand that this may not be as user-friendly for more automated approaches or for multiple clusters.

So, one option using the AKS/Kubernetes built-in functionality is to set fsGroup and potentially fsGroupChangePolicy in a pod configuration, as in this example:

kind: Pod
apiVersion: v1
metadata:
  name: lustre-echo-date
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    fsGroupChangePolicy: "Always" # Or "OnRootMismatch"
  nodeSelector:
... 

This would allow AKS to change the gid of all the files/directories within that Lustre cluster mount to the specified fsGroup (in this case gid 1000). I realized when testing this that our driver does need an additional change for this to work, and I will be submitting a PR to improve this. Until then, you can add fsGroupPolicy: File to the deploy/csi-azurelustre-driver.yaml before uninstalling and re-installing the driver, or to the existing csidrivers.storage.k8s.io azurelustre.csi.azure.com in your install, so that it looks like this:

---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: azurelustre.csi.azure.com
spec:
  fsGroupPolicy: File
  attachRequired: false
  podInfoOnMount: true

This changes the default from ReadWriteOnceWithFSType to allow AKS to change the group id for the files/directories in the AMLFS cluster any time any pod is created with a given fsGroup.

Again, I will be making changes for the next release so the driver will 'just work' for the end user with these options, but I hope that these suggestions can allow you to make progress until it is officially supported in the driver.

Some notes:

1. Be aware that this will update the ownership ACLs for files and directories in the actual AMLFS cluster for every pod that is created with an fsGroup option. Be sure not to configure your deployment in such a way that these pods will be fighting to change group ids in an inconsistent manner.
2. If you are using root squash, make sure it is configured so that AKS can manage the AMLFS cluster from its IP addresses.
3. This driver is only currently supported with AKS and AMLFS (Azure Managed Lustre Filesystem). Since you mentioned AKS, I assume you are also using AMLFS, but I want to be clear there is no expectation of this driver working by default with other types of Lustre deployments.