RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/ember-csi.io/csi.sock, err: rpc error: code = Unimplemented desc = Method not found!

[Akrog]

As Reported on Ember-CSI's repository (embercsi/ember-csi#153), on Kubernetes nodes since v1.15 we've been starting to see these continual errors:

Jan 19 13:19:21 kube23.foo.com kubelet: I0119 13:19:21.844405 2107 operation_generator.go:193] parsed scheme: ""
Jan 19 13:19:21 kube23.foo.com kubelet: I0119 13:19:21.844431 2107 operation_generator.go:193] scheme "" not registered, fallback to default scheme
Jan 19 13:19:21 kube23.foo.com kubelet: I0119 13:19:21.844449 2107 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/plugins/ember-csi.io/csi.sock 0 }] }
Jan 19 13:19:21 kube23.foo.com kubelet: I0119 13:19:21.848196 2107 clientconn.go:577] ClientConn switching balancer to "pick_first"
Jan 19 13:19:21 kube23.foo.com kubelet: E0119 13:19:21.850493 2107 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins/ember-csi.io/csi.sock" failed. No retries permitted until 2020-01-19 13:21:23.850457201 -0500 EST m=+1109229.205664694 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/ember-csi.io/csi.sock, err: rpc error: code = Unimplemented desc = Method not found!"

This is on a system that currently doesn't have any pvc mounts. Rebooting the system will clear it up for a time but then it will start to happen again.

Kubernetes 1.15 introduced a change in the plugin-manager that keeps retrying till all the plugins (socket files under /var/lib/kubelet/{plugins,plugin_registry}) registrations succeed. That didn't use to be the case in previous releases, where we would only see the error once.

We need to change the location of the CSI socket used for communication between the CSI plugin and its sidecars.

The operator is creating the volume for the nodes as in Ember-CSI's examples https://github.com/embercsi/ember-csi-operator/blame/8c6a833acb97d433b6c1841318ace989b2fe0250/pkg/controller/embercsi/node.go#L156:

fmt.Sprintf("%s/%s/%s", "--kubelet-registration-path=/var/lib/kubelet/plugins", GetPluginDomainName(ecsi.Name), "csi.sock"),

And like the doc's manifest at https://kubernetes-csi.github.io/docs/deploying.html#driver-volume-mounts:

      # This volume is where the socket for kubelet->driver communication is done
      - name: socket-dir
        hostPath:
          path: /var/lib/kubelet/plugins/<driver-name>
          type: DirectoryOrCreate

Though the picture (https://kubernetes-csi.github.io/docs/images/kubelet.png) refers to a different directory: /var/lib/kubelet/<driver-name>

I believe we should be changing the nodes manifest to avoid these errors.

We can use an empty dir, or we can create the sockets under /var/lib/kubelet/.