Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail early for too long instance names #19

Merged
merged 1 commit into from
Nov 8, 2023

Conversation

praiskup
Copy link
Member

@praiskup praiskup commented Oct 24, 2023

The resource name length in IBM Cloud needs to be <= 63 characters.
If we attempt to allocate a set of resources (machine, volumes, IP, ...)
and one of the names for those resources is too long, the library fails
with pretty cryptic traceback:

Traceback (most recent call last):
  File "/usr/bin/resalloc-ibm-cloud-vm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.11/site-packages/resalloc_ibm_cloud/ibm_cloud_vm.py", line 348, in main
    create_instance(service, name, opts)
  File "/usr/lib/python3.11/site-packages/resalloc_ibm_cloud/ibm_cloud_vm.py", line 182, in create_instance
    response = service.create_instance(instance_prototype_model)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ibm_vpc/vpc_v1.py", line 4583, in create_instance
    response = self.send(request, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ibm_cloud_sdk_core/base_service.py", line 341, in send
    raise ApiException(response.status_code, http_response=response)
ibm_cloud_sdk_core.api_exception.ApiException: Error: Expected only one oneOf fields to be set: got 0, Code: 400

With this fix, we'll get the failure much faster:

ERROR: Field boot_volume_attachment.volume.name is longer than 63 characters: copr-ibm-cloud-s390x-washington-prod-13290257-20231024-201117-root

Copy link
Member

@nikromen nikromen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's really weird limitation :D

@@ -339,6 +339,11 @@ def main():
opts.instance_name = name
opts.instance = "production" if "-prod-" in name else "devel"

maxlen = 56
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it really 56? It says 63 in UI
Screenshot_20231024_203853

maxlen = 56
if len(name) > maxlen:
log.error("Name %s is too long, max %s characters", name, maxlen)
sys.exit(1)
Copy link
Member

@nikromen nikromen Oct 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we do something else than error? I mean something like shrinking the name?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for looking. Could be? But it is reproducible with:

RESALLOC_NAME=s390x_us_east /var/lib/resallocserver/resalloc_provision/ibm-cloud-vm create "copr_ibm_cloud_s390x_washington_prod_13290257_20231024_201117"
...
ibm_cloud_sdk_core.api_exception.ApiException: Error: Expected only one oneOf fields to be set: got 0, Code: 400

While it is just ok with one byte dropped:

RESALLOC_NAME=s390x_us_east /var/lib/resallocserver/resalloc_provision/ibm-cloud-vm create "copr_ibm_cloud_s390x_washington_prod_13290257_20231024_20111"
...

I'd swear I was able to reproduce it with copr-ibm-cloud-s390x-washington-dev-02817070-20231024-10x string while bisecting. See differences:

copr_ibm_cloud_s390x_washington_prod_13290257_20231024_201117  fails (61 bytes)
copr_ibm_cloud_s390x_washington_prod_13290257_20231024_20111   works (60 bytes)
copr-ibm-cloud-s390x-washington-dev-02817070-20231024-10x      works now, failed before

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, now I think the problem was the name of the volume, which is basically instance name + suffix. PTAL

The resource name length in IBM Cloud needs to be <= 63 characters.
If we attempt to allocate a set of resources (machine, volumes, IP, ...)
and one of the names for those resources is too long, the library fails
with pretty cryptic traceback:

    Traceback (most recent call last):
      File "/usr/bin/resalloc-ibm-cloud-vm", line 8, in <module>
        sys.exit(main())
                 ^^^^^^
      File "/usr/lib/python3.11/site-packages/resalloc_ibm_cloud/ibm_cloud_vm.py", line 348, in main
        create_instance(service, name, opts)
      File "/usr/lib/python3.11/site-packages/resalloc_ibm_cloud/ibm_cloud_vm.py", line 182, in create_instance
        response = service.create_instance(instance_prototype_model)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/ibm_vpc/vpc_v1.py", line 4583, in create_instance
        response = self.send(request, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/site-packages/ibm_cloud_sdk_core/base_service.py", line 341, in send
        raise ApiException(response.status_code, http_response=response)
    ibm_cloud_sdk_core.api_exception.ApiException: Error: Expected only one oneOf fields to be set: got 0, Code: 400

With this fix, we'll get the failure much faster:

    ERROR: Field boot_volume_attachment.volume.name is longer than 63 characters: copr-ibm-cloud-s390x-washington-prod-13290257-20231024-201117-root
Copy link
Member

@nikromen nikromen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! and sorry for ignoring this PR for so long, completely lost tracking of it :/

return
log.error("Field %s is longer than %s characters: %s",
".".join(itemspec), max_length, to_check)
sys.exit(1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still wondering if failing the creation of VM and trying it again will result (after some retries) in a shorter name? Could we also reconsider (e.g. in separate PR) different and shorter naming scheme for volumes?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still wondering if failing the creation of VM and trying it again will result (after some retries) in a shorter name?

No, the whole point of this PR is to make the failure 100% obvious.

Could we also reconsider (e.g. in separate PR) a different and shorter naming scheme for volumes?

We could. Listing them in the ibm-cloud-list-vms might be a bit more complicated then.

The whole story was basically a trivial mistake while creating the Resalloc pool name ... so I changed the
pool name from ..ibm_cloud.. to just ..ic.. and done. This PR will help others to spot the issue much easily.

@praiskup praiskup merged commit 22523a5 into fedora-copr:main Nov 8, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants