Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when create pv with VMWare driver #166

Closed
creativie opened this issue Jun 23, 2020 · 2 comments
Closed

Error when create pv with VMWare driver #166

creativie opened this issue Jun 23, 2020 · 2 comments

Comments

@creativie
Copy link

creativie commented Jun 23, 2020 •
edited
Loading

OKD 4.5 beta
ember-csi operator
VSphere 6.5

When I create pvc vsphere-controller reports an error:

2020-06-23 08:11:09.938 1 INFO oslo_vmware.api [-] Successfully established new session; session ID is beb44.�[00m
2020-06-23 08:11:09 vsphere INFO oslo_vmware.api [-] PBM WSDL updated to file:///usr/lib/python2.7/site-packages/oslo_vmware/wsdl/6.5/pbmService.wsdl�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.ember_csi [-] Ember CSI v0.9.0-74-g0ba96c3+21042020181212621894913 with 30 workers (CSI spec: v1.1.0, cinderlib: v1.1.0.dev12, cinder: v15.1.0.dev142)�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.ember_csi [-] Persistence module: CRDPersistence�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.ember_csi [-] Running as controller with backend VMwareVcVmdkDriver v3.4.2�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.ember_csi [-] Plugin name: vsphere.ember-csi.io�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.ember_csi [-] Debugging feature is ENABLED with ember_csi.rpdb and OFF. Toggle it with SIGUSR1.�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.ember_csi [-] Supported filesystems: cramfs, minix, btrfs, ext2, ext3, ext4, xfs�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.ember_csi [-] Now serving on unix:///csi-data/csi.sock...�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-349b594d-c568-4626-9eb2-3d573cf1f427] => GRPC Probe�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-349b594d-c568-4626-9eb2-3d573cf1f427] <= GRPC Probe served in 0s�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-d6bbf822-0d6b-4fc0-9899-24ea267f1529] => GRPC GetPluginInfo�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-d6bbf822-0d6b-4fc0-9899-24ea267f1529] <= GRPC GetPluginInfo served in 0s�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-cac42f50-eb8e-4e0f-9b30-28f07a4ca6e9] => GRPC GetPluginCapabilities�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-cac42f50-eb8e-4e0f-9b30-28f07a4ca6e9] <= GRPC GetPluginCapabilities served in 0s�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-8fea6eb9-f20a-4017-a8da-7e5f0ee5148a] => GRPC ControllerGetCapabilities�[00m
2020-06-23 08:11:10 vsphere INFO ember_csi.common [req-8fea6eb9-f20a-4017-a8da-7e5f0ee5148a] <= GRPC ControllerGetCapabilities served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-17f8305d-c095-4198-9d31-ac3def8a3f2a] => GRPC GetPluginInfo�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-17f8305d-c095-4198-9d31-ac3def8a3f2a] <= GRPC GetPluginInfo served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-527db6ca-f6eb-4e48-97d8-75a774080d75] => GRPC Probe�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-527db6ca-f6eb-4e48-97d8-75a774080d75] <= GRPC Probe served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-03e09cfe-c428-47cd-bada-a85eb1fa4e5a] => GRPC ControllerGetCapabilities�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-03e09cfe-c428-47cd-bada-a85eb1fa4e5a] <= GRPC ControllerGetCapabilities served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-0b0de34b-f79d-4f57-9fe9-26c508f7f926] => GRPC Probe�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-0b0de34b-f79d-4f57-9fe9-26c508f7f926] <= GRPC Probe served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-5f24393d-26f5-4ade-8f9f-c38859e5b77f] => GRPC GetPluginInfo�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-5f24393d-26f5-4ade-8f9f-c38859e5b77f] <= GRPC GetPluginInfo served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-3dd8ec8c-5cfd-4bb3-9a18-82ee672d8e63] => GRPC GetPluginCapabilities�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-3dd8ec8c-5cfd-4bb3-9a18-82ee672d8e63] <= GRPC GetPluginCapabilities served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-e8feb000-8bf5-4526-9d45-266a1f0fc0df] => GRPC ControllerGetCapabilities�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-e8feb000-8bf5-4526-9d45-266a1f0fc0df] <= GRPC ControllerGetCapabilities served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-724364d5-bbc3-4da8-8ddc-37516d2b3a4a] => GRPC Probe�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-724364d5-bbc3-4da8-8ddc-37516d2b3a4a] <= GRPC Probe served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-5c3e7f36-fc6b-46ba-af5d-7119de548181] => GRPC GetPluginInfo�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-5c3e7f36-fc6b-46ba-af5d-7119de548181] <= GRPC GetPluginInfo served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-34a430ce-d4c5-49fd-a269-e09e43aedbe5] => GRPC GetPluginCapabilities�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-34a430ce-d4c5-49fd-a269-e09e43aedbe5] <= GRPC GetPluginCapabilities served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-1068381a-09e1-4067-ac0e-d71ec4596e83] => GRPC ControllerGetCapabilities�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-1068381a-09e1-4067-ac0e-d71ec4596e83] <= GRPC ControllerGetCapabilities served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-b13ec8c1-f322-4069-a35f-8cbf3a9e8bed] => GRPC ControllerGetCapabilities�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-b13ec8c1-f322-4069-a35f-8cbf3a9e8bed] <= GRPC ControllerGetCapabilities served in 0s�[00m
2020-06-23 08:11:11 vsphere INFO ember_csi.common [req-12f886d9-2e52-4f79-bf93-500942fe4d73] => GRPC CreateVolume pvc-38bcded4-952f-4546-9c80-0c70e5bc3389�[00m
2020-06-23 08:11:11 vsphere ERROR ember_csi.common [req-12f886d9-2e52-4f79-bf93-500942fe4d73] !! GRPC CreateVolume failed in 0s with ABORTED (Operation pending for volume (error)): Exception�[00m
2020-06-23 08:11:12 vsphere INFO ember_csi.common [req-d8f1db8e-087f-4f24-abde-fcf3a8d33dc8] => GRPC CreateVolume pvc-38bcded4-952f-4546-9c80-0c70e5bc3389�[00m
2020-06-23 08:11:12 vsphere ERROR ember_csi.common [req-d8f1db8e-087f-4f24-abde-fcf3a8d33dc8] !! GRPC CreateVolume failed in 0s with ABORTED (Operation pending for volume (error)): Exception�[00m
2020-06-23 08:11:14 vsphere INFO ember_csi.common [req-e877649a-eb24-4fa4-923e-a88174bd113e] => GRPC CreateVolume pvc-38bcded4-952f-4546-9c80-0c70e5bc3389�[00m
2020-06-23 08:11:14 vsphere ERROR ember_csi.common [req-e877649a-eb24-4fa4-923e-a88174bd113e] !! GRPC CreateVolume failed in 0s with ABORTED (Operation pending for volume (error)): Exception�[00m
2020-06-23 08:11:18 vsphere INFO ember_csi.common [req-7d3a4474-2d13-440b-8f13-edec85ae18ec] => GRPC CreateVolume pvc-38bcded4-952f-4546-9c80-0c70e5bc3389�[00m
2020-06-23 08:11:18 vsphere ERROR ember_csi.common [req-7d3a4474-2d13-440b-8f13-edec85ae18ec] !! GRPC CreateVolume failed in 0s with ABORTED (Operation pending for volume (error)): Exception�[00m
2020-06-23 08:11:26 vsphere INFO ember_csi.common [req-b276dcf2-8f9a-4263-916b-cea7a91224d8] => GRPC CreateVolume pvc-38bcded4-952f-4546-9c80-0c70e5bc3389�[00m
2020-06-23 08:11:26 vsphere ERROR ember_csi.common [req-b276dcf2-8f9a-4263-916b-cea7a91224d8] !! GRPC CreateVolume failed in 0s with ABORTED (Operation pending for volume (error)): Exception�[00m
2020-06-23 08:11:42 vsphere INFO ember_csi.common [req-3550f723-7a36-4cb5-99ff-44b30dd440e2] => GRPC CreateVolume pvc-38bcded4-952f-4546-9c80-0c70e5bc3389�[00m
2020-06-23 08:11:42 vsphere ERROR ember_csi.common [req-3550f723-7a36-4cb5-99ff-44b30dd440e2] !! GRPC CreateVolume failed in 0s with ABORTED (Operation pending for volume (error)): Exception�[00m
2020-06-23 08:11:53 vsphere INFO ember_csi.common [req-141f7a30-5d7a-4059-8a84-9ea6c9c0cfb8] => GRPC CreateVolume pvc-45025ea0-8279-4f24-8f0e-2e4f27cd9f0c�[00m
2020-06-23 08:11:54.550 1 WARNING suds.client [-] Web service reported a SOAP processing fault using an unexpected HTTP status code 200. Reporting as an internal server error.�[00m
2020-06-23 08:11:54.665 1 WARNING suds.client [-] Web service reported a SOAP processing fault using an unexpected HTTP status code 200. Reporting as an internal server error.�[00m
2020-06-23 08:11:54.700 1 WARNING suds.client [-] Web service reported a SOAP processing fault using an unexpected HTTP status code 200. Reporting as an internal server error.�[00m
2020-06-23 08:11:54 vsphere ERROR ember_csi.common [req-141f7a30-5d7a-4059-8a84-9ea6c9c0cfb8] !! GRPC CreateVolume failed in 1s with Unexpected exception (unsupported operand type(s) for -: 'unicode' and 'float')
	Traceback (most recent call last):
	  File "/ember-csi/ember_csi/common.py", line 129, in dolog
	    result = f(self, request, context)
	  File "/ember-csi/ember_csi/common.py", line 224, in checker
	    return f(self, request, context)
	  File "/ember-csi/ember_csi/common.py", line 76, in wrapper
	    return func(self, request, context)
	  File "/ember-csi/ember_csi/base.py", line 485, in CreateVolume
	    **params)
	  File "/ember-csi/ember_csi/v1_0_0/csi_base.py", line 73, in _create_volume
	    **params)
	  File "/ember-csi/ember_csi/base.py", line 448, in _create_volume
	    vol = self.backend.create_volume(size=vol_size, name=name, **params)
	  File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line 157, in create_volume
	    vol.create()
	  File "/usr/lib/python2.7/site-packages/cinderlib/objects.py", line 457, in create
	    self._raise_with_resource()
	  File "/usr/lib/python2.7/site-packages/cinderlib/objects.py", line 224, in _raise_with_resource
	    six.reraise(*exc_info)
	  File "/usr/lib/python2.7/site-packages/cinderlib/objects.py", line 450, in create
	    model_update = self.backend.driver.create_volume(self._ovo)
	  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/vmdk.py", line 378, in create_volume
	    self._create_backing(volume)
	  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/vmdk.py", line 516, in _create_backing
	    extra_config=extra_config)
	  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/volumeops.py", line 846, in create_backing
	    create_spec)
	  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/volumeops.py", line 807, in _create_backing_int
	    task_info = self._session.wait_for_task(task)
	  File "/usr/lib/python2.7/site-packages/oslo_vmware/api.py", line 399, in wait_for_task
	    return evt.wait()
	  File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
	    result = hub.switch()
	  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 298, in switch
	    return self.greenlet.switch()
	  File "/usr/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 79, in _inner
	    delay = interval - timeutils.delta_seconds(start, end)
	TypeError: unsupported operand type(s) for -: 'unicode' and 'float'�[00m
2020-06-23 08:11:54 vsphere ERROR grpc._server [req-141f7a30-5d7a-4059-8a84-9ea6c9c0cfb8] Exception calling application: unsupported operand type(s) for -: 'unicode' and 'float': TypeError: unsupported operand type(s) for -: 'unicode' and 'float'
2020-06-23 08:11:54.844 1 ERROR grpc._server Traceback (most recent call last):
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib64/python2.7/site-packages/grpc/_server.py", line 389, in _call_behavior
2020-06-23 08:11:54.844 1 ERROR grpc._server     return behavior(argument, context), True
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/ember-csi/ember_csi/common.py", line 165, in wrapper
2020-06-23 08:11:54.844 1 ERROR grpc._server     return f(*args, **kwargs)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/ember-csi/ember_csi/common.py", line 129, in dolog
2020-06-23 08:11:54.844 1 ERROR grpc._server     result = f(self, request, context)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/ember-csi/ember_csi/common.py", line 224, in checker
2020-06-23 08:11:54.844 1 ERROR grpc._server     return f(self, request, context)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/ember-csi/ember_csi/common.py", line 76, in wrapper
2020-06-23 08:11:54.844 1 ERROR grpc._server     return func(self, request, context)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/ember-csi/ember_csi/base.py", line 485, in CreateVolume
2020-06-23 08:11:54.844 1 ERROR grpc._server     **params)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/ember-csi/ember_csi/v1_0_0/csi_base.py", line 73, in _create_volume
2020-06-23 08:11:54.844 1 ERROR grpc._server     **params)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/ember-csi/ember_csi/base.py", line 448, in _create_volume
2020-06-23 08:11:54.844 1 ERROR grpc._server     vol = self.backend.create_volume(size=vol_size, name=name, **params)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinderlib/cinderlib.py", line 157, in create_volume
2020-06-23 08:11:54.844 1 ERROR grpc._server     vol.create()
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinderlib/objects.py", line 457, in create
2020-06-23 08:11:54.844 1 ERROR grpc._server     self._raise_with_resource()
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinderlib/objects.py", line 224, in _raise_with_resource
2020-06-23 08:11:54.844 1 ERROR grpc._server     six.reraise(*exc_info)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinderlib/objects.py", line 450, in create
2020-06-23 08:11:54.844 1 ERROR grpc._server     model_update = self.backend.driver.create_volume(self._ovo)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/vmdk.py", line 378, in create_volume
2020-06-23 08:11:54.844 1 ERROR grpc._server     self._create_backing(volume)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/vmdk.py", line 516, in _create_backing
2020-06-23 08:11:54.844 1 ERROR grpc._server     extra_config=extra_config)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/volumeops.py", line 846, in create_backing
2020-06-23 08:11:54.844 1 ERROR grpc._server     create_spec)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/vmware/volumeops.py", line 807, in _create_backing_int
2020-06-23 08:11:54.844 1 ERROR grpc._server     task_info = self._session.wait_for_task(task)
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/oslo_vmware/api.py", line 399, in wait_for_task
2020-06-23 08:11:54.844 1 ERROR grpc._server     return evt.wait()
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2020-06-23 08:11:54.844 1 ERROR grpc._server     result = hub.switch()
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 298, in switch
2020-06-23 08:11:54.844 1 ERROR grpc._server     return self.greenlet.switch()
2020-06-23 08:11:54.844 1 ERROR grpc._server   File "/usr/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 79, in _inner
2020-06-23 08:11:54.844 1 ERROR grpc._server     delay = interval - timeutils.delta_seconds(start, end)
2020-06-23 08:11:54.844 1 ERROR grpc._server TypeError: unsupported operand type(s) for -: 'unicode' and 'float'
2020-06-23 08:11:54.844 1 ERROR grpc._server �[00m
2020-06-23 08:11:54 vsphere INFO ember_csi.common [req-316ea7bc-acd6-4a2b-901b-8d437e9a36be] => GRPC CreateVolume pvc-45025ea0-8279-4f24-8f0e-2e4f27cd9f0c�[00m
2020-06-23 08:11:54 vsphere ERROR ember_csi.common [req-316ea7bc-acd6-4a2b-901b-8d437e9a36be] !! GRPC CreateVolume failed in 0s with ABORTED (Operation pending for volume (error)): Exception�[00m
@Akrog Akrog mentioned this issue Jun 23, 2020
Open
@Akrog
Copy link
Member

Akrog commented Jun 23, 2020

@creativie thanks for reporting the issue.

After looking at the report I have opened issue #109 on our operator since this is a problem with the forms.

The problem is that OLM forms don't have the float type, so we were passing them as text, which is wrong.

The stacktrace we see is caused by the vmware_task_poll_interval configuration option.

We'll work on the operator to fix this issue, and in the meantime there are multiple possible workarounds, one is:

  • Go into the deployed EmberStorageBackend's manifest and copy the current YAML
  • Delete the existing EmberStorageBackend (not the operator itself)
  • Create a new EmberStorageBackend and paste the copied YAML instead of using the form
  • Change all the driver__VMwareVcVmdk__vmware_task_poll_interval values from "2.0" to 2.0
  • Hit the Create button

@creativie
Copy link
Author

Works well! Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Akrog @creativie