Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MaxVersions can error when trying to delete a version with an operation in progress #67

Open
martha opened this issue Jul 16, 2020 · 0 comments

Comments

@martha
Copy link

martha commented Jul 16, 2020

Steps to reproduce:

  1. Deploy a GAE backend service with MaxVersions set to 1
  2. Deploy the service again (maybe several times; the bug does not happen consistently). You will see an error from GCP that looks like this:
ERROR: (gcloud.app.versions.delete) Issue deleting version: [service-name/version-num]
--
[service-name/version-num]: ABORTED: Cannot operate on 
apps/project-name/services/service-name/versions/version-num 
because an operation is already in progress for 
apps/project-name/services/service-name/versions/version-num by {hash}.
error: exit status 1

After reaching out to GCP support to ask about this error, we received the following note from them:

If you are running the "remove versions" tool immediately after the version is deployed, we may still be performing operations internally (e.g. stopping the old version, performing traffic splitting for the new version, etc.). In cases where operations are still being performed on the old version, we will reject any other operations (in this case, your delete request). Whether other operations are occurring may also depend on how exactly you are using your CI/CD tool. I would recommend you to delete an old version more than 10 minutes after the new version is deployed and let me know if you still face the issue. We can do further digging in such a case.

Rather than waiting 10 minutes (which is a loong time lag to introduce to our Drone pipelines), this could be fixed by checking if an operation is already in progress and if so, waiting (or skipping) before starting a new operation on a version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant