Descriptive restore error on timeout due to terminating namespace #7424

kaovilai · 2024-02-13T19:49:11Z

Signed-off-by: Tiger Kaovilai [email protected]

Thank you for contributing to Velero!

Please add a summary of your change

Make restore error descriptive when namespace being restored is in terminating state.
Any user seeing this error should know that velero does not force a namespace to disappear by removing finalizers because that could be destructive to some workloads.

User should make namespace be in a state other than terminating for Velero to continue restore process.

Does your change fix a particular issue?

Fixes #5697

Please indicate you've done the following:

Accepted the DCO. Commits without the DCO will delay acceptance.
Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
Updated the corresponding documentation in site/content/docs/main.

codecov · 2024-02-13T19:57:16Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.71%. Comparing base (2518824) to head (e7ffa62).
Report is 403 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7424   +/-   ##
=======================================
  Coverage   61.71%   61.71%           
=======================================
  Files         263      263           
  Lines       28869    28873    +4     
=======================================
+ Hits        17816    17819    +3     
- Misses       9793     9794    +1     
  Partials     1260     1260

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pkg/util/kube/utils.go

… Descriptive restore error on terminating namespace. Signed-off-by: Tiger Kaovilai <[email protected]> revert utils_test.go Signed-off-by: Tiger Kaovilai <[email protected]> address https://github.com/vmware-tanzu/velero/pull/7424/files/c7b189dd6035839c9eb8ce3dab4ead574de77adb#r1494194484 Signed-off-by: Tiger Kaovilai <[email protected]> Update pkg/util/kube/utils.go Signed-off-by: Tiger Kaovilai <[email protected]> Signed-off-by: Tiger Kaovilai <[email protected]>

mayankagg9722 · 2024-03-27T10:46:10Z

HI @kaovilai @blackpiglet we are seeing the Issue #7516. Please look into it once in the same method.

EnsureNamespaceExistsAndIsReady

During restore for every resource within the namespace, we are calling the check to await on if namespace exists (wait for 10 min polling). For instance, if we have 100 resources in a namespace that itself is in terminating state for very long then it impacts the restore flow to get stuck/halt and it too increases the time to restore as it does it for each resource in the same namespace.

mayankagg9722 · 2024-03-28T04:56:56Z

pkg/util/kube/utils.go

@@ -97,6 +103,9 @@ func EnsureNamespaceExistsAndIsReady(namespace *corev1api.Namespace, client core

 	// err will be set if we timed out or encountered issues retrieving the namespace,
 	if err != nil {
+		if terminatingNamespace {
+			return false, nsCreated, errors.Wrapf(err, "timed out waiting for terminating namespace %s to disappear before restoring", namespace.Name)
+		}


as this call is used during restore as well, and if such error occur for the resource during restore, can we also add this to restore warnings list?

Mention #7516

Is warning preferred over restore error?

Also per PR title, the intent is that the error are visible in restore error.

What changes do you see needed to be made to the PR?

Sorry on the late reply, I was awk.
Yes, overall its good to have this error that you added and its visible during restore

For reducing the time to address the issue #7516 we need to look at other options.

anshulahuja98 · 2024-08-23T09:05:19Z

@kaovilai , feel free to merge this now, approved it.

kaovilai · 2024-08-23T10:57:20Z

@anshulahuja98 I don't have access to merge. Not a maintainer.

github-actions bot requested review from anshulahuja98 and blackpiglet February 13, 2024 19:49

github-actions bot assigned kaovilai Feb 13, 2024

kaovilai changed the title ~~Descriptive restore error on terminating namespace~~ Descriptive restore error when restoring into a terminating namespace Feb 13, 2024

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch from 73b25fd to bbd910c Compare February 13, 2024 19:50

github-actions bot added the has-changelog label Feb 13, 2024

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch 2 times, most recently from cf72b8a to fb25a55 Compare February 13, 2024 21:06

kaovilai changed the title ~~Descriptive restore error when restoring into a terminating namespace~~ Prevent restoring namespaces from backup as Terminating, descriptive restore error on timeout due to terminating namespace Feb 13, 2024

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch from fb25a55 to 6a72c2f Compare February 13, 2024 22:57

github-actions bot added the has-unit-tests label Feb 13, 2024

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch 6 times, most recently from dfe96a5 to 18bb3a0 Compare February 14, 2024 00:24

mateusoliveira43 reviewed Feb 14, 2024

View reviewed changes

pkg/util/kube/utils.go Show resolved Hide resolved

kaovilai commented Feb 14, 2024

View reviewed changes

pkg/util/kube/utils.go Show resolved Hide resolved

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch 2 times, most recently from 7177214 to c7b189d Compare February 14, 2024 18:51

ywk253100 requested a review from reasonerjt February 19, 2024 06:56

blackpiglet reviewed Feb 19, 2024

View reviewed changes

pkg/util/kube/utils.go Outdated Show resolved Hide resolved

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch 3 times, most recently from 98ee21a to 28b5d26 Compare February 20, 2024 06:21

github-actions bot removed the has-unit-tests label Feb 20, 2024

kaovilai changed the title ~~Prevent restoring namespaces from backup as Terminating, descriptive restore error on timeout due to terminating namespace~~ Descriptive restore error on timeout due to terminating namespace Feb 20, 2024

kaovilai requested a review from blackpiglet February 20, 2024 06:22

kaovilai requested a review from mateusoliveira43 February 20, 2024 06:22

blackpiglet previously approved these changes Feb 20, 2024

View reviewed changes

shubham-pampattiwar previously approved these changes Feb 20, 2024

View reviewed changes

kaovilai dismissed stale reviews from shubham-pampattiwar and blackpiglet via ad879f3 March 26, 2024 19:57

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch from 28b5d26 to ad879f3 Compare March 26, 2024 19:57

kaovilai force-pushed the descriptiveRestoreErrorOnNamespaceTerminating branch from ad879f3 to e7ffa62 Compare March 26, 2024 19:58

kaovilai requested review from shubham-pampattiwar and blackpiglet March 26, 2024 19:59

blackpiglet approved these changes Mar 27, 2024

View reviewed changes

mayankagg9722 mentioned this pull request Mar 27, 2024

Restore getting stuck for 10 min at different intervals due to "Failed to discover group" and "I0308: client-side throttling" #7516

Closed

mayankagg9722 reviewed Mar 28, 2024

View reviewed changes

anshulahuja98 approved these changes Aug 23, 2024

View reviewed changes

anshulahuja98 merged commit f5671c7 into vmware-tanzu:main Aug 23, 2024
66 checks passed

shubham-pampattiwar mentioned this pull request Oct 21, 2024

Nominate Tiger Kaovilai as a maintainer #8328

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Descriptive restore error on timeout due to terminating namespace #7424

Descriptive restore error on timeout due to terminating namespace #7424

kaovilai commented Feb 13, 2024 •

edited

Loading

codecov bot commented Feb 13, 2024 •

edited

Loading

mayankagg9722 commented Mar 27, 2024 •

edited

Loading

mayankagg9722 Mar 28, 2024

mayankagg9722 Mar 28, 2024

kaovilai Mar 28, 2024

kaovilai Mar 28, 2024

kaovilai Mar 28, 2024

mayankagg9722 Apr 2, 2024

mayankagg9722 Apr 2, 2024

anshulahuja98 commented Aug 23, 2024

kaovilai commented Aug 23, 2024

Descriptive restore error on timeout due to terminating namespace #7424

Descriptive restore error on timeout due to terminating namespace #7424

Conversation

kaovilai commented Feb 13, 2024 • edited Loading

Please add a summary of your change

Does your change fix a particular issue?

Please indicate you've done the following:

codecov bot commented Feb 13, 2024 • edited Loading

Codecov Report

mayankagg9722 commented Mar 27, 2024 • edited Loading

mayankagg9722 Mar 28, 2024

Choose a reason for hiding this comment

mayankagg9722 Mar 28, 2024

Choose a reason for hiding this comment

kaovilai Mar 28, 2024

Choose a reason for hiding this comment

kaovilai Mar 28, 2024

Choose a reason for hiding this comment

kaovilai Mar 28, 2024

Choose a reason for hiding this comment

mayankagg9722 Apr 2, 2024

Choose a reason for hiding this comment

mayankagg9722 Apr 2, 2024

Choose a reason for hiding this comment

anshulahuja98 commented Aug 23, 2024

kaovilai commented Aug 23, 2024

kaovilai commented Feb 13, 2024 •

edited

Loading

codecov bot commented Feb 13, 2024 •

edited

Loading

mayankagg9722 commented Mar 27, 2024 •

edited

Loading