Kubernetes on AWS templates do not produce a functional cluster

[scottslowe]

What happened?

As of a few EKS revisions ago, EKS clusters require the AWS EBS CSI driver/add-on to be installed in order to host workloads that use volumes. Our current templates do not have the AWS EBS CSI driver/add-on installed, and therefore cannot host any workloads that use volumes. They are, essentially, not functional.

To complicate the matter, the EKS component used in the templates does not currently support managing EKS add-ons (see pulumi/pulumi-eks#587 and pulumi/pulumi-eks#592). It is unclear to me at this point whether this means we need to stop using the EKS component or not. The AWS v6 upgrade also complicates this matter.

Example

1. Run pulumi new kubernetes-aws-go. Complete the new project wizard.
2. Run pulumi up. Wait for the cluster to be provisioned.
3. Obtain the Kubeconfig for the cluster (typically using pulumi stack output kubeconfig > file).
4. Using the Kubeconfig obtained in step 3, deploy the Bitnami WordPress Helm chart to the cluster. (Alternately, you may use any workload, Helm chart or otherwise, that requires a volume.)
5. Observe the Pods stuck in a Pending state. Eventually, a SchedulingFailed event will appear in the output of kubectl describe pod.

Output of pulumi about

This is not specific to a particular version of Pulumi.

Additional context

This problem was uncovered while trying to resolve issues with pulumi/examples#1472

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[cnunciato]

Verified today this is still an issue. Steps taken after deploying the template with pulumi up:

$ pulumi stack output kubeconfig > kubeconfig
$ helm repo add bitnami https://charts.bitnami.com/bitnami --kubeconfig ./kubeconfig
$ helm repo update
$ helm install wp-test bitnami/wordpress --kubeconfig ./kubeconfig
$ kubectl get all --kubeconfig ./kubeconfig

NAME                                     READY   STATUS    RESTARTS   AGE
pod/wp-test-mariadb-0                    0/1     Pending   0          27m
pod/wp-test-wordpress-644b87db76-9xd55   0/1     Pending   0          27m

NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                      AGE
service/kubernetes          ClusterIP      172.20.0.1       <none>                                                                    443/TCP                      36m
service/wp-test-mariadb     ClusterIP      172.20.107.81    <none>                                                                    3306/TCP                     27m
service/wp-test-wordpress   LoadBalancer   172.20.220.251   adb69b344391a49c99566465e37fe460-1702088664.us-west-2.elb.amazonaws.com   80:32649/TCP,443:30442/TCP   27m

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/wp-test-wordpress   0/1     1            0           27m

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/wp-test-wordpress-644b87db76   1         1         0       27m

NAME                               READY   AGE
statefulset.apps/wp-test-mariadb   0/1     27m

Looks like there's a potential workaround in this issue: pulumi/pulumi-eks#833 (comment)

[thoward]

@justinvp this was previously assigned to @cnunciato and needs to be picked up by someone to get over the finish line.