Node-Rotation failure can leave cluster with shard-rebalancing disabled

[rtyley]

We recently had node-rotation fail due to #36, which is unfortunate in itself, but even worse, this was about 20 minutes after we began having a separate outage within Ophan, and were distracted by that.

We decided to try scaling up our Elasticsearch cluster, in an attempt to better handle the load, but got no benefit from scaling up, as elasticsearch-node-rotation had left our cluster with this cluster setting when it died:

"cluster.routing.rebalance.enable": "none"

...the team lost a fair bit of time attempting to understand why the new boxes weren't taking any of the load.

elasticsearch-node-rotation/src/migrateShards.ts

Line 13 in 7d97fd5

.then(() => updateRebalancingStatus(oldInstance.id, "all"))

[jacobwinch]

Ouch - that's a nasty one, thanks for reporting this.

One solution to this would be to define a failure handler lambda (as a Catch) which re-enables this setting if the execution fails part way through.

Obviously fixing #36 would help too!