kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: #659

naveenspen14 · 2021-04-21T18:34:42Z

Hi Team,

We are installing cf-for-k8s on vmware environment. We are getting the below error through kapp.

$ kapp deploy -a cf -f cf4k8s_setup/cf-for-k8s-rendered.yml -y
Target cluster 'https://:6443' (nodes:master-0-cf4k8s01, 4+)

Changes

Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri
(cluster) bionic-stack ClusterStack 0/1 t 5h - - reconcile ok -
^ cf-buildpack-store ClusterStore 0/1 t 5h - - reconcile ok -
^ defaults.webhook.kpack.io MutatingWebhookConfiguration - 5h update - reconcile ok -
^ istiod-istio-system ValidatingWebhookConfiguration - 5h update - reconcile ok -
^ validation.webhook.kpack.io ValidatingWebhookConfiguration - 5h update - reconcile ok -
cf-workloads-staging cc-kpack-registry-service-account ServiceAccount - 5h update - reconcile ok -
^ cf-default-builder Builder - - create - reconcile - -
kpack webhook-certs Secret - 5h update - reconcile ok -

Op: 1 create, 0 delete, 5 update, 2 noop
Wait to: 8 reconcile, 0 delete, 0 noop

11:40:25PM: ---- applying 1 changes [7/8 done] ----
11:41:26PM: create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging

kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging:
Saving record of last applied resource:
Updating resource builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: admission webhook "validation.webhook.kpack.io" denied the request: validation failed: invalid value: “cf4k8s”/cf-default-builder: spec.tag (reason: BadRequest)

The interesting part is, cf api <> is a success and but couldn't able to push any apps.

Also, we not seeing any build packs after installation. Do we need to install build packs separately through packeto.

cf-gitbot · 2021-04-21T18:34:44Z

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/177859822

The labels on this github issue will be updated when the story is started.

jimconner · 2021-04-22T09:46:06Z

The cf-default-builder container image is uploaded to your specified registry as part of the deployment. From the error message you've got, I'm guessing that the registry didn't like the request that it got sent due to quoting within the spec.tag... invalid value: “cf4k8s”/cf-default-builder: - I'm guessing here, but it doesn't look right ot me that 'cf4k8s' is quoted, but the bits after the slash is not. Maybe something quoted in the manifest that shoudn't be?

naveenspen14 · 2021-04-22T14:22:36Z

Thanks, Jimconner.

I'm still getting the same issue after removing the quotes to repository_prefix: cf4k8s.

7:45:30PM: ---- waiting on 2 changes [305/308 done] ----
7:45:30PM: ongoing: reconcile clusterstack/bionic-stack (kpack.io/v1alpha1) cluster
7:45:30PM: ^ No failing or successful conditions found
7:46:28PM: ongoing: reconcile clusterstore/cf-buildpack-store (kpack.io/v1alpha1) cluster
7:46:28PM: ^ No failing or successful conditions found
7:46:30PM: ---- waiting on 2 changes [305/308 done] ----
7:46:30PM: ongoing: reconcile clusterstack/bionic-stack (kpack.io/v1alpha1) cluster
7:46:30PM: ^ No failing or successful conditions found

kapp: Error: Timed out waiting after 15m0s

one of the observations is, the pod ccdb-migrate-6tkjr is in a completed state but an error with the volume mount.

Events:
Type Reason Age From Message

Normal Scheduled 11m default-scheduler Successfully assigned cf-system/ccdb-migrate-6tkjr to k8s-worker-2-cf4k8s03
Normal Created 10m kubelet, k8s-worker-2-cf4k8s03 Created container istio-init
Normal Started 10m kubelet, k8s-worker-2-cf4k8s03 Started container istio-init
Normal Pulling 10m kubelet, k8s-worker-2-cf4k8s03 Pulling image "index.docker.io/istio/proxyv2:1.7.3"
Normal Pulled 10m kubelet, k8s-worker-2-cf4k8s03 Successfully pulled image "index.docker.io/istio/proxyv2:1.7.3"
Normal Started 10m kubelet, k8s-worker-2-cf4k8s03 Started container istio-proxy
Normal Created 10m kubelet, k8s-worker-2-cf4k8s03 Created container istio-proxy
Normal Pulling 10m kubelet, k8s-worker-2-cf4k8s03 Pulling image "cloudfoundry/cloud-controller-ng@sha256:5ee75f427b8859eb35e7c9449992ccd4fb4c3dbd37db95d1ffac02a35db12553"
Normal Pulled 10m kubelet, k8s-worker-2-cf4k8s03 Successfully pulled image "cloudfoundry/cloud-controller-ng@sha256:5ee75f427b8859eb35e7c9449992ccd4fb4c3dbd37db95d1ffac02a35db12553"
Normal Created 10m kubelet, k8s-worker-2-cf4k8s03 Created container run-migrations
Normal Started 10m kubelet, k8s-worker-2-cf4k8s03 Started container run-migrations
Normal SandboxChanged 10m kubelet, k8s-worker-2-cf4k8s03 Pod sandbox changed, it will be killed and re-created.
Normal Pulling 10m (x2 over 11m) kubelet, k8s-worker-2-cf4k8s03 Pulling image "index.docker.io/istio/proxyv2:1.7.3"
Normal Pulled 10m (x2 over 10m) kubelet, k8s-worker-2-cf4k8s03 Successfully pulled image "index.docker.io/istio/proxyv2:1.7.3"
Warning Failed 10m kubelet, k8s-worker-2-cf4k8s03 Error: cannot find volume "default-token-zh9k6" to mount into container "istio-init"

kubectl get pods -A
NAMESPACE NAME cf-blobstore cf-blobstore-minio-6d9d86dff5-wtljx cf-db cf-db-postgresql-0 cf-system ccdb-migrate-6tkjr cf-system cf-api-clock-dc89dfc98-9c7q5 cf-system cf-api-controllers-6464964cc7-966zr cf-system cf-api-deployment-updater-78cf895cc-bfjnj cf-system cf-api-server-5c58f95fb7-hs7rn cf-system cf-api-worker-64957dc6d4-fglrd cf-system eirini-api-59c8f57956-mllgc cf-system eirini-app-migration-f6t5t cf-system eirini-event-reporter-595b7fd969-8djhm cf-system eirini-event-reporter-595b7fd969-9nhpk cf-system eirini-task-reporter-54d4b685d4-58bwl cf-system eirini-task-reporter-54d4b685d4-vgwcc cf-system fluentd-7kbcl cf-system fluentd-l5fkt cf-system fluentd-ppv8x cf-system fluentd-rqc25 cf-system fluentd-sk4pq cf-system instance-index-env-injector-5 cf-system log-cache-backend-759d9b7797-mp8rf cf-system log-cache-frontend-c68f7f45f-gd2tc cf-system metric-proxy-5b48fbcb56-795jm cf-system routecontroller-69586ffd46-w8msq cf-system uaa-7bbdbff88f-gw596 cf-workloads restart-workloads-for-istio1-7-3-6qbk5 istio-system istio-ingressgateway-4ls6r istio-system istio-ingressgateway-5vs4v istio-system istio-ingressgateway-jp49t istio-system istio-ingressgateway-lrk92 istio-system istiod-89bc798f5-vd7zd kpack kpack-controller-5c9b8fcc97-c6pjw kpack kpack-webhook-555bf54bc4-ccp7h READY STATUS RESTARTS AGE
2/2 Running 0 14m
2/2 Running 0 14m
0/2 Completed 0 11m
2/2 Running 0 11m
3/3 Running 0 14m
2/2 Running 0 14m
6/6 Running 0 11m
3/3 Running 0 11m
2/2 Running 0 14m
0/1 Completed 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
fff98685b-b2rd5 1/1 Running 0 14m
3/3 Running 0 14m
3/3 Running 0 14m
2/2 Running 0 14m
2/2 Running 0 14m
3/3 Running 0 14m
0/1 Completed 0 14m
2/2 Running 0 15m
2/2 Running 0 15m
2/2 Running 0 15m
2/2 Running 0 15m
1/1 Running 0 15m
2/2 Running 0 14m
2/2 Running 0 14m

kubernetes version: v1.18.4
cf-for-k8s version: v3.0.0
os: centos 7
kernel version: 5.9.1

cf buildpacks
Getting buildpacks...

buildpack position enabled locked filename stack
No buildpacks found

Here I'm using artifactory docker repository as an app registry.

Kindly help me.

naveenspen14 · 2021-04-26T08:52:56Z

Request you to help me on this.
2:11:26PM: fail: reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging
2:11:26PM: ^ Encountered failure condition Ready == False: (message: stack bionic-stack is not ready)

kapp: Error: waiting on reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging:
Finished unsuccessfully (Encountered failure condition Ready == False: (message: stack bionic-stack is not ready))

naveenspen14 · 2021-04-26T19:27:24Z

Looks like issue is with connectivity with docker hub.

Namespace cf-workloads-staging
Name cf-default-builder
Kind Builder
Status conditions:
- lastTransitionTime: "2021-04-26T18:54:20Z"
message: stack bionic-stack is not ready
status: "False"
type: Ready
observedGeneration: 1
stack: {}

Namespace (cluster)
Name bionic-stack
Kind ClusterStack
Status buildImage: {}
conditions:
- lastTransitionTime: "2021-04-26T19:01:58Z"
message: 'Get "https://index.docker.io/v2/": read tcp 10.244.181.113:54198->52.55.43.248:443:
read: connection reset by peer'
status: "False"
type: Ready
observedGeneration: 1
runImage: {}

Now I have changed app_register to docker hub.

app_registry:
hostname: https://hub.docker.com/
repository_prefix: "hub.docker.com/cf4k8s"
username: "*"
password: ""

12:58:02AM: ongoing: reconcile clusterstack/bionic-stack (kpack.io/v1alpha1) cluster
12:58:02AM: ^ No failing or successful conditions found
12:58:02AM: fail: reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging
12:58:02AM: ^ Encountered failure condition Ready == False: (message: stack bionic-stack is not ready)
12:58:02AM: debug: CommandRun: end (10.519357212s)

kapp: Error: waiting on reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging:
Finished unsuccessfully (Encountered failure condition Ready == False: (message: stack bionic-stack is not ready))

Appreciate your help.

jimconner · 2021-04-27T07:49:15Z

message: 'Get "https://index.docker.io/v2/": read tcp 10.244.181.113:54198->52.55.43.248:443:
read: connection reset by peer'

Do you have a firewall or proxy blocking your access to Dockerhub or something like that? cf-for-k8s needs to push the cf-default-builder image up to the registry that you defined, and it would appear that it can't get a connection to Dockerhub.

Hope that helps.

naveenspen14 · 2021-04-28T19:54:44Z

Thanks, Jim for the details.
Docker registry details are working fine while tested through hack scripts. But the same is failing through kapp.

cat registry.yaml

app_registry:
hostname: https://index.docker.io/v1/
repository_prefix: XXXXXX
username: ""
password: "***"

[myhome@k8s hack]$ bash validate-registry-access.sh registry.yaml
WARNING: The hack scripts are intended for development of cf-for-k8s.
They are not officially supported product bits. Their interface and behavior may change at any time without notice.
registry_host -> https://index.docker.io/v1/
username -> ******
repo -> nvn4u81
docker_tag -> XXXXXX/cfk8s-test-delete-me
password -> **********
logging into dockerhub with username and password
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/myhome/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
building tiny test docker image...
docker push-ing XXXXXX/cfk8s-test-delete-me to test push access...
Confirmed push access to dockerhub registry

any other alternative, instead of pushing the cf-default-builder image to the registry. Install only on the K8s environment.

jimconner · 2021-04-29T06:31:12Z

Hi Naveen. When I was trying out cf-for-k8s a few months ago (haven't touched it for a while), I used harbor as my registry. I did this because my broadband speed is limited and because I don't have a paid-for dockerhub account and therefore was hitting the container pull limits that they introduced back in November. My deployments of cf-for-k8s and harbor were running on top of minikube.

Here's the notes I made from when I set harbor up... Maybe there'll be something in here that helps you.

Generate trusted cert for Harbor using LetsEncrypt Certbot:
certbot certonly --manual --preferred-challenges dns \
    -m [email protected] -vvv --agree-tos -d harbor.mydomain.org.uk \
    --work-dir ~/tmp/cert/ --logs-dir ~/tmp/cert/logs --config-dir ~/tmp/cert/config
# do the DNS TXT record dance to make LetsEncrypt trust us and generate the cert
cd ~/tmp/cert/config/live/harbor.mydomain.org.uk
cp fullchain.pem tls.crt
cp key.pem tls.key
kubectl create secret tls harbor --cert tls.crt --key tls.key # Store the secret in kubes as tls and call it 'harbor'. We use the secret for deploying harbor via helm

Install Harbor on top of minikube:
# https://github.com/goharbor/harbor-helm/
helm repo add harbor https://helm.goharbor.io
helm install helm-harbor harbor/harbor \
    --set expose.type=loadBalancer \
    --set expose.tls.auto.commonName=harbor.mydomain.org.uk \
    --set domain=harbor.mydomain.org.uk \
    --set externalURL=https://harbor.mydomain.org.uk \
    --set expose.tls.certSource=secret \
    --set expose.tls.secret.secretName=harbor

Default User/Pass: admin/Harbor12345

Log in at https://harbor.mydomain.org.uk # Yay for valid certs
Create user 'cf-for-k8s' ... Password: OhNoTheInternetKnowsMyPasswords
Create project for 'cf-images'. We'll use this for the kubes images. Add cf-for-k8s user as a member. 'Maintainer' permissions seems to work.
Create a project for 'cf-for-k8s'. We'll use this for apps. Add cf-for-k8s as a mebmber as before.

cf-for-k8s:
Follow steps for creating cf-values file : https://github.com/cloudfoundry/cf-for-k8s/blob/develop/docs/getting-started-tutorial.md

In my cf-values.yml I had the following for app_registry

app_registry:
  hostname: https://harbor.mydomain.org.uk/v2/
  repository_prefix: "harbor.mydomain.org.uk/cf-for-k8s"
  username: "cf-for-k8s"
  password: "OhNoTheInternetKnowsMyPasswords"

naveenspen14 · 2021-04-29T08:08:12Z

Thanks, Jim. I will try the same.

Birdrock · 2021-05-10T22:23:19Z

@naveenspen14 Were you able to resolve your issue?

naveenspen14 · 2021-05-27T05:20:55Z

Hi Jim & Birdrock,

I couldn't resolve this issue in VMware proxy environment. Except for docker-registry connectivity, others are working fine. Due to this issue, we couldn't be able to push any apps.
But it worked seamlessly on Tencent cloud. Currently trying on AWS.

drpdishant · 2021-05-27T12:57:26Z

@jimconner
I have installed Harbor using helm on Kubernetes, but somehow the blob access is not working with it, I checked pushing images, its successful but its show error while accessing blob

docker push core.registry.openxcell.dev/cf-for-k8s/alpine
Using default tag: latest
The push refers to repository [core.registry.openxcell.dev/cf-for-k8s/alpine]
b2d5eeeaba3a: Layer already exists 
received unexpected HTTP status: 500 Internal Server Error

Due to this I am getting error in kapp deploy

3:42:59PM:  ^ Encountered failure condition Ready == False:  (message: POST https://core.registry.openxcell.dev/v2/cloudfoundry/cf-default-builder/blobs/uploads/: UNKNOWN: unknown error; map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir Path:/storage/docker/registry/v2/repositories/cloudfoundry/cf-default-builder]])

kapp: Error: waiting on reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging:
  Finished unsuccessfully (Encountered failure condition Ready == False:  (message: POST https://core.registry.openxcell.dev/v2/cloudfoundry/cf-default-builder/blobs/uploads/: UNKNOWN: unknown error; map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir Path:/storage/docker/registry/v2/repositories/cloudfoundry/cf-default-builder]]))

cf-gitbot added the unscheduled label Apr 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: #659

kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: #659

naveenspen14 commented Apr 21, 2021

cf-gitbot commented Apr 21, 2021

jimconner commented Apr 22, 2021

naveenspen14 commented Apr 22, 2021 •

edited

Loading

naveenspen14 commented Apr 26, 2021

naveenspen14 commented Apr 26, 2021 •

edited

Loading

jimconner commented Apr 27, 2021

naveenspen14 commented Apr 28, 2021 •

edited

Loading

jimconner commented Apr 29, 2021 •

edited

Loading

naveenspen14 commented Apr 29, 2021

Birdrock commented May 10, 2021

naveenspen14 commented May 27, 2021

drpdishant commented May 27, 2021

kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: #659

kapp: Error: Applying create builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging: #659

Comments

naveenspen14 commented Apr 21, 2021

cf-gitbot commented Apr 21, 2021

jimconner commented Apr 22, 2021

naveenspen14 commented Apr 22, 2021 • edited Loading

naveenspen14 commented Apr 26, 2021

naveenspen14 commented Apr 26, 2021 • edited Loading

jimconner commented Apr 27, 2021

naveenspen14 commented Apr 28, 2021 • edited Loading

cat registry.yaml

jimconner commented Apr 29, 2021 • edited Loading

naveenspen14 commented Apr 29, 2021

Birdrock commented May 10, 2021

naveenspen14 commented May 27, 2021

drpdishant commented May 27, 2021

naveenspen14 commented Apr 22, 2021 •

edited

Loading

naveenspen14 commented Apr 26, 2021 •

edited

Loading

naveenspen14 commented Apr 28, 2021 •

edited

Loading

jimconner commented Apr 29, 2021 •

edited

Loading