Missing PVs effects creation of LVM-type datacontrol on the same node

[silentred]

description

If any remote PV is disconnected from the target node, the LVM datacontrol will be unable to setup any new VG for the new datacontrol.
This scenario is not rare because the remote target node may have hareware issues that will cause lost connection of the target volume. If this happens, the node-disk-agent will be trapped in getting LVM errors and not able to any thing else.

node-disk-agent logs:

E0425 05:44:25.063070       1 sync_datacontrol.go:213] LVM_PV_LOST stderr=  WARNING: Couldn't find device with uuid nl34r9-swzT-p8b5-vJ5p-Yn8G-GbEh-BUgGN4.
  WARNING: VG obnvmf-pvc-75057105-d76d-41e8-8552-1785d5c4102b is missing PV nl34r9-swzT-p8b5-vJ5p-Yn8G-GbEh-BUgGN4 (last written to /dev/alinvme21n1).
I0425 05:44:25.702268       1 signal.go:42] Got a signal child exited


E0425 05:44:25.702830       1 sync_datacontrol.go:302] LVM_PV_LOST stderr=  WARNING: Couldn't find device with uuid nl34r9-swzT-p8b5-vJ5p-Yn8G-GbEh-BUgGN4.
  WARNING: VG obnvmf-pvc-75057105-d76d-41e8-8552-1785d5c4102b is missing PV nl34r9-swzT-p8b5-vJ5p-Yn8G-GbEh-BUgGN4 (last written to /dev/alinvme21n1).
get vg failed
I0425 05:44:25.826288       1 signal.go:42] Got a signal child exited
I0425 05:44:25.826323       1 cmd.go:126] ListPV out:   {......}
I0425 05:44:25.826532       1 sync_datacontrol.go:331] remove pvs [/dev/alinvme60n1]
I0425 05:44:25.955883       1 signal.go:42] Got a signal child exited
E0425 05:44:25.955924       1 cmd.go:239] err cmd:pvremove [/dev/alinvme60n1 -y], ProcState=exit status 5 StdErr=  PV /dev/alinvme60n1 is used by VG obnvmf-pvc-75057105-d76d-41e8-8552-1785d5c4102b so please use vgreduce first.
  (If you are certain you need pvremove, then confirm by using --force twice.)
  /dev/alinvme60n1: physical volume label not removed.
, exit status 5, output:
E0425 05:44:25.955936       1 sync_datacontrol.go:334] cmd:pvremove [/dev/alinvme60n1 -y], ProcState=exit status 5 StdErr=  PV /dev/alinvme60n1 is used by VG obnvmf-pvc-75057105-d76d-41e8-8552-1785d5c4102b so please use vgreduce first.
  (If you are certain you need pvremove, then confirm by using --force twice.)
  /dev/alinvme60n1: physical volume label not removed.
, exit status 5rm pv failed, retry in 20 sec.