Volumes and Volume Mounts

[ThorstenHans]

Using Kubernetes volumes from within a Spin App would unlock plenty scenarios. Unfortunately, this is currently not possible, because trying to read files from guest code (although access has been granted) results in file not found errors.

For repro I created the following repository: https://github.com/ThorstenHans/tmp--spinkube-volume-mounts

Code Flow

For every incoming request the code does the following

1. Read /mount/.gitkeep and prints the content to stdout
2. Read /mount/secrets/some and prints the content to stdout
3. Returns an HTTP Response (200 if successful or 500 in case of an error)

The mount folder

• The mount folder is part of the repository (required to make spin registry push work
• The mount/.gitkeep file is part of the repository
• The mount/secrets folder is created upon deployment using a volumeMount (the underlying volume references a Kubernetes Secret

Reproducing the Error

• Clone the repo at https://github.com/ThorstenHans/tmp--spinkube-volume-mounts
• Deploy Kubernetes Secret kubectl apply -f kubernetes/secret.yaml
• Build the App and Push the OCI artifact spin oci --build ttl.sh/spinapp-volume-repro:2h
• If using a different image as the one specified in the previous command, modify kubernetes/spinapp.yaml
• Deploy the Spin App kubectl apply -f kubernetes/spinapp.yaml
• Setup port-forwarding to the service of the pod kubectl port-forward svc/repro 8080:80
• Send an HTTP request to the Spin App curl -iX GET http://localhost:8080
• Check the logs of the underlying pod kubectl logs -l core.spinoperator.dev/app-name=repro

Run setup verification Pod

The repo also contains a regular Pod to verify volume mount is working.

• You can run it via kubectl apply -f kubernetes/pod.yaml
• Jump in the pod using kubectl exec -it repro-pod /bin/bash
• Cat the content of the mounted file using cat /mount/secrets/some

Investigations on the underling Kubernetes Node

# Get ContainerID and Uid of your Pod
POD_NAME=<your_pod_name>

# ContainerID
kubectl get po -n sample $POD_NAME -o json | jq '.status.containerStatuses[0].containerID'

# uid 
kubectl get po -n sample $POD_NAME -o json | jq '.metadata.uid' 

# Create a node debug pod
kubectl get no
NODE_NAME=<desired_node_here>
kubectl debug node/$NODE_NAME -it --image ubuntu     

chroot /host /bin/bash

# Find Spin App folder using it's container ID
cd /run/containerd/io.containerd.runtime.v2.task/k8s.io/<PUT_YOUR_CONTAINER_ID_HERE>/rootfs

# Find the mounted volumes from kubelet
cd /var/lib/kubelet/pods/<PUT_YOUR_POD_UID_HERE>

[rajatjindal]

I spent some time debugging this and found a simple workaround. if we mount volume under /assets/<component-id>/<dir>, then this works.

this is because spin-oci-loader copies the static assets from oci artifact to the dir /assets/<component-id> and creates a spin.lock file with updated location.

but this also means we need to check if oci artifacts overrides the volume if they have same files/folders.

[rajatjindal]

I further tried with volume mount a file which already exist in spin-app oci artifact, and got following error:

time="2024-05-23T07:35:55.824024042Z" level=error msg="run_wasi ERROR >>>  failed: failed to resolve content for component "repro"

Caused by:
    0: failed to write inline content to "/assets/repro/mount/.gitkeep"
    1: Read-only file system (os error 30)"

time="2024-05-23T07:35:55.8245285Z" level=info msg="error running start function: failed to resolve content for component "repro""

maybe we can update oci-loader to skip writing the path if it already exist? (but what will happen if its a directory with different content).

I think we need to define what should be the expected behavior here and take it from there.

[jsturtevant]

I believe what might be happening here is that we have two different file systems: the container file system and the wasm filesystem. Kuberentes/containerd are handling the volume mounts into the container but the spin shim isn't taking those volume mounts and mapping them into the wasm file system.

In runwasi's wasmtime, we we've kind of skirted around this by mounting the entire container filesystem into the wasm file system: https://github.com/containerd/runwasi/blob/0f7817ab3e0c5d4f6b1bd3182ddc65bdf957f315/crates/containerd-shim-wasmtime/src/instance.rs#L304

I don't think this is what we necessarily want, we probably want to take specific volume mounts and copy them into the wasmfile system