Backup PartiallyFailed

[jravetch]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
Backup PartiallyFailed, failing to backup secrets.

What did you expect to happen:
Full backup

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• velero backup logs velero-schedule-config-20201117120023 | grep -v 'level=info'

time="2020-11-17T12:01:42Z" level=error msg="Error listing items" backup=velero/velero-schedule-config-20201117120023 error="stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 44029; INTERNAL_ERROR" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/backup/item_collector.go:294" error.function="github.com/vmware-tanzu/velero/pkg/backup.(*itemCollector).getResourceItems" group=v1 logSource="pkg/backup/item_collector.go:294" namespace= resource=secrets

Environment:

• Velero version (use velero version): 1.5.1
• Velero features (use velero client config get features): gcp plugin 1.1.0
• Kubernetes version (use kubectl version): 1.16.13
• Kubernetes installer & version: GKE
• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release): COS

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[ashish-amarnath]

@jravetch The error is coming from the API server when Velero is trying to list all the secrets.
Can you please confirm that your Kubernetes cluster is functional and the API server is running and is available?

[jravetch]

Yes this cluster is functioning normally. It is a large cluster with over 100 nodes. Velero and previously ark was running in this cluster without issue. Velero was also upgraded from version 1.0 to 1.5.

[ashish-amarnath]

@jravetch What is the version of Kubernetes that you are running? Also, I found this issue googleapis/google-cloud-go#784 reporting a similar error and it is coming from the API server in GKE cluster. Is this happening consistently?

[jravetch]

We are running 1.16.13-gke.401. Not sure how that issue is related as I don't see GKE mentioned and was closed over 2yrs ago. We are not seeing this error outside of Velero backup logs.

[jravetch]

I am also able list all secrets via kubectl

kubectl get secrets --all-namespaces | wc -l
4184

[jravetch]

After cleaning up 3k+ unused secrets in the affected cluster, velero was able to list and backup the secrets without issue. @ashish-amarnath are you aware of any limitation when backing up large number of secrets?

[dsu-igeek]

It looks like the code is not handling paging across large numbers of resources. This is a known issue, will raise this in priority, thanks. #262

[jravetch]

We had to cleanup secrets again, as velero timed out while trying to list 3k secrets.