[Remote State] Optimize diff publication

[soosinha]

Is your feature request related to a problem? Please describe

Meta issue: #14419
Remote state publication was launched as an experimental feature in 2.15. The logic for diff publication works as below:

• The cluster state is split into multiple entities like index metadata, templates, cluster settings, custom metadata, routing table etc.
• Whenever there is a change in an entity, only that entity is uploaded as a new file in remote store
• The manifest file is updated to reflect which entity has been updated.
• The follower node can then read the manifest file and download only the entities which have changed.

The problem with this approach comes in case of node drops. Normally one or more shards for all the indices are present on a node. Whenever a single node drops, this can result in the change in index routing tables for all indices. This would then result in uploading the routing tables for the indices.

Describe the solution you'd like

We can optimize this to compute the diff across multiple entities so that the payload for upload and download is smaller.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response