Unabled to create a pod with qemu-sev on CoCo 0.5

[wainersm]

I got a single-node Kubernetes 1.24.0 cluster on an AMD SEV machine. Recently I got it installed CoCo 0.5.0 (previously using 0.3.0) but I am not being able to start a simple pod from an encrypted image (same image used to work with CoCo 0.3.0).

First question: should I rebuild the image for 0.5.0?

Assuming the old image should work with 0.5.0, here goes more information.

The output of kubectl describe indicates the image_rs didn't get the key:

$ kubectl describe pod/coco-custom-nginx
Name:         coco-custom-nginx
Namespace:    default
Priority:     0
Node:         virtlab1012/10.8.0.194
Start Time:   Tue, 02 May 2023 14:59:35 -0400
Labels:       <none>
Annotations:  <none>
Status:       Pending
IP:           10.244.0.14
IPs:
  IP:  10.244.0.14
Containers:
  nginx:
    Container ID:   
    Image:          wainersm/coco-custom-nginx:encrypted
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rmvnv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-rmvnv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              node-role.kubernetes.io/worker=
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  9m37s                   default-scheduler  Successfully assigned default/coco-custom-nginx to virtlab1012
  Normal   Pulling    7m56s (x4 over 9m35s)   kubelet            Pulling image "wainersm/coco-custom-nginx:encrypted"
  Warning  Failed     7m55s (x4 over 9m34s)   kubelet            Failed to pull image "wainersm/coco-custom-nginx:encrypted": rpc error: code = Internal desc = failed to handle layer: failed to get decrypt key missing private key needed for decryption
  Warning  Failed     7m55s (x4 over 9m34s)   kubelet            Error: ErrImagePull
  Warning  Failed     7m45s (x6 over 9m34s)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m26s (x20 over 9m34s)  kubelet            Back-off pulling image "wainersm/coco-custom-nginx:encrypted

simple-kbs got the request, meaning the attestation-agent can talk with KBS, and validated the policy:

[2023-05-02T18:59:35Z INFO  simple_kbs::grpc] Launch Bundle Requested
[2023-05-02T18:59:36Z INFO  simple_kbs::grpc] Secret Requested
[2023-05-02T18:59:36Z INFO  simple_kbs::grpc] Policy validated succesfully. Connection: Connection { policy: 3, fw_api_major: 1, fw_api_minor: 42, fw_build_id: 42, launch_description: "shim launch", fw_digest: "X9w+htu6bEOGKwFVhWtt2q/UfmL94Vg4L+nchiC+RBU=" }

Should simple-kbs log a message to confirm the key set was released? Anyway...

The MySQL datase seems to have the correct data:

$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} -e "SELECT * FROM keysets;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+----------+-----------+-------+
| id | keysetid | kskeys    | polid |
+----+----------+-----------+-------+
| 10 | KEYSET-1 | [key_id1] |  NULL |
+----+----------+-----------+-------+
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} -e "SELECT * FROM secrets;"
mysql: [Warning] Using a password on the command line interface can be insecure.
+----+-----------+----------------------------------------------+-------+
| id | secret_id | secret                                       | polid |
+----+-----------+----------------------------------------------+-------+
| 10 | key_id1   | 7JOQd6TASVf9xL6h9AbmBz6Cn/RozUfl/VBD/QPEsCk= |  NULL |
+----+-----------+----------------------------------------------+-------+

More details about the docker.io/wainersm/coco-custom-nginx:encrypted image:

$ ./bin/skopeo inspect docker://wainersm/coco-custom-nginx:encrypted
{
    "Name": "docker.io/wainersm/coco-custom-nginx",
    "Digest": "sha256:dcf6a2af5c67c421d300084d3e5fed6fa04158c6a42a453f2e71f0a92dd41ccf",
    "RepoTags": [
        "encrypted"
    ],
    "Created": "2023-01-31T17:13:36.297119928Z",
    "DockerVersion": "",
    "Labels": {
        "maintainer": "NGINX Docker Maintainers \[email protected]\u003e"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:1807346aac17f623ba65b0fa1b80887f9c6d1ee3100cddba83d43584cb102dd9",
        "sha256:55d02335427addf4f254ea39b22a3af69864a52e3c247d54126628dde659e695",
        "sha256:d98bc5d0272d71a61bf67356bee34f0c2307b08f924ce0c0ac156bd750f77b12",
        "sha256:fda146edbdb158c956245f576ff20e2bbf3d1ec487c3f38536512ead078d699d",
        "sha256:3056b68c6a22ac6bf38da22424772c5163c08501f2030f795b6276ffe547e21a",
        "sha256:3bc84d163b66fa670125133f660d9a13a72466f0d06e995ac0f143cd9b71e052",
        "sha256:0f724d5ee0dd8f98180c846e2aebcbe9bdc8a07a0e74af6879f23a55378ab430"
    ],
    "LayersData": [
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:1807346aac17f623ba65b0fa1b80887f9c6d1ee3100cddba83d43584cb102dd9",
            "Size": 32727602,
            "Annotations": {
                "org.opencontainers.image.enc.keys.provider.attestation-agent": "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoieWZya2czUGw0VlMwcUV4eUlJMEM3dWFnV0Q3eVFCL2hXeG44b0FNWGp6NkRPZjM1aVJXcnhVYXFweVpVSklzYXZGWnBLaTcwSkpaanVUQnNWcmdSMFh0MTFmSmovU3Z0WVRrdzhZZVhQUGQyVmpyRTZrMHo5MldCQ2hwYVA3SDFHbmdKTjh4Y1JUeURZdkxtelczMXNaMHNKT1hoYU5IVFlNVS9xMmY3dmljOTdzYjhmMjBiMGxvUFpxMjI3dG4rYnJlQ0FpR2k0RE0yTVVxS0NIUWUwamNpS3lBSDRIVE5BSmpMdVFrcWZtam83Ym5JTGdVMEV0bzYxMnJUV3c5eVBBPT0iLCJpdiI6ImtpUndKWGJQTDhCdjAwN21yc1A3N1E9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJpS0dHZDZwaUFybVFqRGFqOHpoSldGbFRGNzFtK0h2SkUvZ0t6QlFhTVhZPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:55d02335427addf4f254ea39b22a3af69864a52e3c247d54126628dde659e695",
            "Size": 26378173,
            "Annotations": {
                "org.opencontainers.image.enc.keys.provider.attestation-agent": "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoic3BzWmRyU21wNDJ5MC9uWjVOS0lMdGI4aXIxVEFyUlY2a0VCdnhqRDNaWStja041STBqMm5KSTBmWjE2WmdKdFZvN0JocGJSL3ZNWWluMVNRV2R4YnB2Sm5kR0YrcEdMcWc5Nit2czBSWVhrYU94SXNtNEJkV1B5d0xiUFpacjY1Wk54U29PS3pLNkRDVFFrcU42bWdCdmZUZzZjTVo3RW1MQkMyb1M0blgzZnFkVFNRZm5XSG9oNEEraXhmc2NHSVpJS2I2T3pYZytWNmdDYVpXb3AyY1ZwV25uOHNEWW9UVmJjRTg1UUQzVVZuWW5jVk9DSSswZzkxUDF0cmltOXRRPT0iLCJpdiI6IlU3Sm1ib3Q0djBGWjVtMFRDd3ExbkE9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJNMFhvcFIrSFN6Yk9QVG9VSmtFZlM2ZThJa0IxQzNTd3g5RWJnblFxKzZBPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:d98bc5d0272d71a61bf67356bee34f0c2307b08f924ce0c0ac156bd750f77b12",
            "Size": 632,
            "Annotations": {
                "org.opencontainers.image.enc.keys.provider.attestation-agent": "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoiVmVVaGk4bjk0aEp0UmZ0MXpicDk2eDFKWXJKeVNKOTc0YklKZkxrZHpQTW1qWS9McTlUTXRvT1I2c2Nmdi83SFdZQTc5M2FmalhIZEJ5TDUrN20vZktVUGNEUzh0UnJwV054NGlvZ0VYa3pIRERhU2NUQTVjL0YxOGNPVlN6NldQcjBtcGJLSWhqNk9EclRiSVdRSlBmdG9hU3BnRTVFMXdGWkJveEFxd2lvMGQzT3Vuc0FuT3l6Ty9xSENhL1JKL0FPUFowbGRLRERKSWJkTFB1czhHbm10U2FVUzJxOFFRV3pmQnpHQys3ZTZDL3FXWFp6ajMycURaQVpJTk5xbGVRPT0iLCJpdiI6IkM1YU4zNCtGSEZDUkEwRjB4U3JMVVE9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiI2bXJZR293TnpPbklBbkdlS0dEb3BrSDZIWHpKU3dlS3hRSy94OHJ2cHA0PSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:fda146edbdb158c956245f576ff20e2bbf3d1ec487c3f38536512ead078d699d",
            "Size": 975,
            "Annotations": {
                "org.opencontainers.image.enc.keys.provider.attestation-agent": "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoiVHhYUDJYUzd6Mzk4RUJPNmRia3VDYkljNEZNN29aT2lNd2dnT3g4QlhrclJuQnRoNUd6UFdFcENOU21jYStpNGZMbUczcVJlN3dhcWROTzg3cU9sWjltRlNISkxxZS84TWpROXRnZkV5MDgxTVJSVTlxMjhKbHhreUQzUGdUQzh0ZGZtb2JwbEpNWUkxcmkyVjZ0Z1ZJcTNHZE9ObXhrME9SNGlwVEJiZFVZdXJGUzJvdXFpMlliK21VOHdiMzBBSzdzYnVjcHlGODlDQWttMGJEcHVoa3RYMEJHczFNS0FYckNCcGZLWDhNdnM4SXRCem9CNmxvU01DRVBLZjkrQmt3PT0iLCJpdiI6IlNOWXBQRUh1V2Z3aTNDRTRTMmRlSWc9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJaZWRWTThveGpjVCtKVTRxTkthcVI4ZWlGdGk4ZTRDaVZKK3dSTlA2TTVBPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:3056b68c6a22ac6bf38da22424772c5163c08501f2030f795b6276ffe547e21a",
            "Size": 787,
            "Annotations": {
                "org.opencontainers.image.enc.keys.provider.attestation-agent": "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoidXlaZUk1MTFYZElsb1c4UXpZZEFMdzB4WUJtNzhsdis5aE1QY0RkOUhhWUNYZUZlN3ZQRmJRQjlWcVQ3Qmo3Si9IUXV0ZHpSQ0NacVJhTUt4Wk5CR1RqZUZzekVYRS92MlJZODl0SDBaalpyckw1RVE1dDUrREloVXc0WmZFVElKU1ZtQkkrUzYzTmxVUExYd0RpYWlzTm9tYVg3QlY5ODJ2MzNhR0VIcG9BZ0FvR2dBWUdJWWRHVWlVaGRyeWQ0VWVkZjdBRW03STVrS0xQa0RhY0VUTGxBdmtFWmlRSTNnY1R4UmkrbkxIclFSNVRiaEI2QTZaYzNNWHBDOCtnUmdRPT0iLCJpdiI6ImpDbmhtTDFxNHZ5d0s1WkpDRUZ3YWc9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJIZEFRQVl1eVMySnlJbTdya1RTdXI3ay9LeXYzS3hkWlc4ZUh3WUcxNHZRPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:3bc84d163b66fa670125133f660d9a13a72466f0d06e995ac0f143cd9b71e052",
            "Size": 1437,
            "Annotations": {
                "org.opencontainers.image.enc.keys.provider.attestation-agent": "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoiRU93R3RWV1d5c2N0K3lkTmNSa255bWVmM2dHNFRtdjVRUnVTT3IyU1g1RVZDOFBCMkNHcW8rb0lxTW1CRXBFWXJWaHYxc0E2Q1ZNSjYrWkZEM0hSVTkxcVJMbVhScmROOWRNVDR2YlRqZEZoOHlsR2JwdnFBeUY5Wjg4UytFRTl2MFNiSHU1ZlBaKzA4YUduQllBSWQxVTdBTWp2RmtzQjkzMUZsMVV6NXl5OXVTUUVBZ0Q3c3lTdEdNS1VlVmJ5b0pDbGFUZ2hqRThMUUIwMTN3Mlp4R1dxL0wrZjNlV0lua1V6dWVyUXBqN210a2NqUm54ZjUxRTZzNjFmNDhSSnFRPT0iLCJpdiI6InF0TFIrcGJQUWVic3YrUmgwTmJzZEE9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJWaTJ6SmoyR3BaWHVJdmxDUU04MndVVmw2dy84c29GVVpac2prTTBrWkVzPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        },
        {
            "MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip+encrypted",
            "Digest": "sha256:0f724d5ee0dd8f98180c846e2aebcbe9bdc8a07a0e74af6879f23a55378ab430",
            "Size": 245,
            "Annotations": {
                "org.opencontainers.image.enc.keys.provider.attestation-agent": "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoiK20vTEdhOVFtS3hlQlZwaU1yOTNvakk3TXV5TmlRVjNKd2JnL1p2b0dpR2hSQTBEUWVKZHozcEVnTHlnZDEremdZQyszVmpNTiszbWpRK1dPeDUvaVlWOXZYOWJmS3J0ZGVtbGFXMDY1QjBYd3lUc2FWMytkRXlmNHg1R0w5VzdvaE5IVkRBS1g1TmhmdGIwbUxHakxEc1pKaWRYazdKeUpGRzhzZUgrRXNUcGdPQjdtSzlwRk9vbDZWMHZTaVd6ZHBYRGNUeEp2QnB3Q1l4blVxVFk3YUprL0pOK1pNUnhoZ1hBTndGdTYxcFZFUTNobzFGbXZYUGNCZHFCemRGUUFRPT0iLCJpdiI6ImhXWWJiRjVwUUxIR25WMXVONmxTYWc9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=",
                "org.opencontainers.image.enc.pubopts": "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJicGFIY1d2M29MazQyTUJxWFZiMDJsSVQvWjV6WVBpdk12MXp2ZS9CVnNvPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ=="
            }
        }
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "NGINX_VERSION=1.23.3",
        "NJS_VERSION=0.7.9",
        "PKG_RELEASE=1~bullseye"
    ]
}
$ echo "eyJraWQiOiJrZXlfaWQxIiwid3JhcHBlZF9kYXRhIjoieWZya2czUGw0VlMwcUV4eUlJMEM3dWFnV0Q3eVFCL2hXeG44b0FNWGp6NkRPZjM1aVJXcnhVYXFweVpVSklzYXZGWnBLaTcwSkpaanVUQnNWcmdSMFh0MTFmSmovU3Z0WVRrdzhZZVhQUGQyVmpyRTZrMHo5MldCQ2hwYVA3SDFHbmdKTjh4Y1JUeURZdkxtelczMXNaMHNKT1hoYU5IVFlNVS9xMmY3dmljOTdzYjhmMjBiMGxvUFpxMjI3dG4rYnJlQ0FpR2k0RE0yTVVxS0NIUWUwamNpS3lBSDRIVE5BSmpMdVFrcWZtam83Ym5JTGdVMEV0bzYxMnJUV3c5eVBBPT0iLCJpdiI6ImtpUndKWGJQTDhCdjAwN21yc1A3N1E9PSIsIndyYXBfdHlwZSI6ImFlc18yNTZfY3RyIn0=" | base64 -d
{"kid":"key_id1","wrapped_data":"yfrkg3Pl4VS0qExyII0C7uagWD7yQB/hWxn8oAMXjz6DOf35iRWrxUaqpyZUJIsavFZpKi70JJZjuTBsVrgR0Xt11fJj/SvtYTkw8YeXPPd2VjrE6k0z92WBChpaP7H1GngJN8xcRTyDYvLmzW31sZ0sJOXhaNHTYMU/q2f7vic97sb8f20b0loPZq227tn+breCAiGi4DM2MUqKCHQe0jciKyAH4HTNAJjLuQkqfmjo7bnILgU0Eto612rTWw9yPA==","iv":"kiRwJXbPL8Bv007mrsP77Q==","wrap_type":"aes_256_ctr"}
$ echo "eyJjaXBoZXIiOiJBRVNfMjU2X0NUUl9ITUFDX1NIQTI1NiIsImhtYWMiOiJpS0dHZDZwaUFybVFqRGFqOHpoSldGbFRGNzFtK0h2SkUvZ0t6QlFhTVhZPSIsImNpcGhlcm9wdGlvbnMiOnt9fQ==" | base64 -d
{"cipher":"AES_256_CTR_HMAC_SHA256","hmac":"iKGGd6piArmQjDaj8zhJWFlTF71m+HvJE/gKzBQaMXY=","cipheroptions":{}}

[fitzthum]

If you decode one of those base64 values in the org.opencontainers.image.enc.keys.provider.attestation-agent annotation, you will see

{
  "kid": "key_id1",
  "wrapped_data": "spsZdrSmp42y0/nZ5NKILtb8ir1TArRV6kEBvxjD3ZY+ckN5I0j2nJI0fZ16ZgJtVo7BhpbR/vMYin1SQWdxbpvJndGF+pGLqg96+vs0RYXkaOxIsm4BdWPywLbPZZr65ZNxSoOKzK6DCTQkqN6mgBvfTg6cMZ7EmLBC2oS4nX3fqdTSQfnWHoh4A+ixfscGIZIKb6OzXg+V6gCaZWop2cVpWnn8sDYoTVbcE85QD3UVnYncVOCI+0g91P1trim9tQ==",
  "iv": "U7Jmbot4v0FZ5m0TCwq1nA==",
  "wrap_type": "aes_256_ctr"
}

Notice that kid value is just key_id1. This is an old image. After release 0.5.0 this key id is replaced by a more standardized resource id. The format of the resource id is checked somewhere before requesting a key from the KBS.

In the log you see that a request is made to the KBS, but this is the request for the launch bundle, rather than the key itself. The launch bundle just helps to setup the communication channel between the KBS and KBC. When using a new image, you should see another request for the key.

To generate a new image, you should check out the coco_keyprovider guide. Note that once you make this new image, you will need to provision the db of simple-kbs with keys with a secret_id that matches the new resource uri.

[Amulyam24]

Hi @fitzthum, @wainersm. I'm facing a similar issue with v0.6.0 installed. I built a new image using the coco_keyprovider by following this guide - https://www.redhat.com/en/blog/confidential-containers-amd-sev

Pod creation fails with: Failed to pull image "amulyam24/coco-custom-nginx:encrypted": rpc error: code = Internal desc = failed to handle layer: failed to get decrypt key missing private key needed for decryption

I can see that the policy is valid successfully in simple-kbs logs and the secret is also injected as per kata logs.

The error from logs

vmconsole="\x1b[0m\x1b[38;5;8m[\x1b[0m2023-08-18T10:54:39Z \x1b[0m\x1b[1m\x1b[31mERROR\x1b[0m attestation_agent::rpc::getresource::ttrpc\x1b[0m\x1b[38;5;8m]\x1b[0m Call AA-KBC to get resource failed: status: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", details: [], metadata: MetadataMap { headers: {} }”

Any idea what's going wrong here or how to debug this issue further?

[fitzthum]

Hm. A bit hard to know without more info, but one thing to keep in mind is that there will actually be two connections to the KBS. First, the shim will get the launch bundle from the simple-kbs. This is probably what you are seeing as occurring successfully in the log of the KBS. Then the AA will try to fetch secrets from the KBS from inside the guest. This second connection seems to be failing. You should double check the KBS_URI parameter and make sure that it is reachable from inside the guest (don't set it to localhost, for instance).

[Amulyam24]

You should double check the KBS_URI parameter and make sure that it is reachable from inside the guest

Thanks @fitzthum, this was the issue. I specified the KBS_URI as 0.0.0.0:{port} and it worked when I specified the right IP.