Volume without a target node after 1 node down in a 3 node cluster

[veenadong]

2.7.0, take 1 node down from a 3 node cluster:

core@glop-nm-126-mem2:~$ kubectl mayastor get volume cc36aef4-0ac9-459b-9e56-c571d5ba2c80 -o yaml
spec:
  num_replicas: 2
  size: 32212254720
  status: Created
  uuid: cc36aef4-0ac9-459b-9e56-c571d5ba2c80
  topology:
    node_topology: !labelled
      exclusion: {}
      inclusion: {}
      affinitykey: []
    pool_topology: !labelled
      exclusion: {}
      inclusion:
        openebs.io/created-by: operator-diskpool
      affinitykey: []
  policy:
    self_heal: true
  thin: true
  num_snapshots: 0
state:
  size: 32212254720
  status: Online
  uuid: cc36aef4-0ac9-459b-9e56-c571d5ba2c80
  replica_topology:
    c51a4d5d-c3b7-48cf-8674-07e7f51321fe:
      node: glop-nm-126-mem3.glcpdev.cloud.hpe.com
      pool: glop-nm-126-mem3.glcpdev.cloud.hpe.com-disk
      state: Online
      usage:
        capacity: 32212254720
        allocated: 926941184
        allocated_snapshots: 0
        allocated_all_snapshots: 0
    85e49541-ff6b-404c-b0aa-eb0e747b1a48:
      node: glop-nm-126-mem1.glcpdev.cloud.hpe.com
      pool: glop-nm-126-mem1.glcpdev.cloud.hpe.com-disk
      state: Unknown
  usage:
    capacity: 32212254720
    allocated: 926941184
    allocated_replica: 926941184
    allocated_snapshots: 0
    allocated_all_snapshots: 0
    total_allocated: 926941184
    total_allocated_replicas: 926941184
    total_allocated_snapshots: 0

Pods are not able to attach the volume:

  Warning  FailedMount  4m12s (x52 over 94m)  kubelet  MountVolume.MountDevice failed for volume "pvc-9a3c606b-9ca2-4438-a3ad-1a07138b6b95" : rpc error: code = Internal desc = Failed to stage volume 9a3c606b-9ca2-4438-a3ad-1a07138b6b95: attach failed: IO error: Input/output error (os error 5), args: hostnqn=nqn.2019-05.io.openebs:node-name:glop-nm-126-mem2.glcpdev.cloud.hpe.com,hostid=42164807-edfc-e94a-3af3-29184e3733b2,nqn=nqn.2019-05.io.openebs:9a3c606b-9ca2-4438-a3ad-1a07138b6b95,transport=tcp,traddr=10.245.244.129,trsvcid=8420,reconnect_delay=10,ctrl_loss_tmo=1980,nr_io_queues=2

Attached is the system dump (note: the logs collection failed using the plugin, so capture the logs using a different method).
mayastor.log.gz
mayastor-2024-08-20--18-05-02-UTC.tar.gz

[dcaputo-harmoni]

I am seeing this same error (attach failed: IO error: Input/output error (os error 5)) after taking one node down and bringing it back up. When running kubectl-mayastor get volumes it lists the volumes that had that target note as having a target node of <none> and accessibility of <none> as well, but a status of online.

Just to provide some further details here, it looks like for the volumes that went down as a result of the node going down, the frontend/host_acl node differs from the target node, whereas for the volumes that remained working, this was the same.

{
  "uuid": "8388d455-d250-4706-a16b-55dfa6ef8327",
  "size": 8589934592,
  "labels": null,
  "num_replicas": 2,
  "status": {
    "Created": "Online"
  },
  "policy": {
    "self_heal": true
  },
  "topology": {
    "node": {
      "Labelled": {
        "exclusion": {},
        "inclusion": {}
      }
    },
    "pool": {
      "Labelled": {
        "exclusion": {},
        "inclusion": {
          "openebs.io/created-by": "operator-diskpool"
        }
      }
    }
  },
  "last_nexus_id": null,
  "operation": null,
  "thin": true,
  "target": {
    "node": "aks-storage-93614762-vmss000002",
    "nexus": "9e0bbe18-9b2a-4aaf-ac71-10edf18a044d",
    "protocol": "nvmf",
    "active": true,
    "config": {
      "controllerIdRange": {
        "start": 5,
        "end": 6
      },
      "reservationKey": 12425731461037558000,
      "reservationType": "ExclusiveAccess",
      "preemptPolicy": "Holder"
    },
    "frontend": {
      "host_acl": [
        {
          "node_name": "aks-storage-93614762-vmss000001",
          "node_nqn": "nqn.2019-05.io.openebs:node-name:aks-storage-93614762-vmss000001"
        }
      ]
    }
  },
  "publish_context": {},
  "affinity_group": null
}

[tiagolobocastro]

Seems we had missed the first one @veenadong, sorry about that.

@dcaputo-harmoni could you please share the volume attachments which reference this volume (if any) and also a support bundle?

[dcaputo-harmoni]

@tiagolobocastro Unfortunately I had to kill and restore the cluster right after this happened, and didn't get a chance to export the data you are looking for before I did. If it happens again I'll provide these details, thanks. I was running mayastor 2.7.0 and just upgraded to 2.7.1 when it rebuilt, and I know there are some stability improvements in there so am wondering if that might help.

[tiagolobocastro]

No problem, I guess keep an eye on it and should it happen again please let us know.
Without logs hard to say if any stability fix would help here. Could be a data-plane bug or could be some simple miscommunication between the control-plane and csi, causing the volume to not be published.

[tiagolobocastro]

I think this bug: #1747 (or a variation of this) explains what happens here. CSI and control-plane get out of sync, and the volume ends up not staying published and csi-node keeps trying to connect to a target which is not there.
A few things we can improve here:

1. fix what caused the out of sync between csi and controller
2. if csi-node can't connect to the volume target, it should check if the subsystem is listening and report this information. If it stays this way it surely means the volume target is not being created.

@Abhinandan-Purkait @dsharma-dc any other thoughts here?