Throttle TiKV's total region count in allocate IDs

[tonyxuqqi]

Development Task

Today it's necessary to limit the region count per TiKV, such as memory consumption.
In either raftstore v1 or v2 (partitioned-raft-kv), one tikv has a specific limit of region counts.
We probably want to fail the allocate ID when the total region count per TiKV exceeds a specific threshold. For v1, it can be 100K per 32GB, For v2, it can be 5K per 32GB.

[rleungx]

Why not directly manage it by TiKV itself since TiKV has the region count info? And it also reduces one gRPC call.

[tonyxuqqi]

TiKV itself can do that too, but there's no extra follow ups to recover from it. Essentially that TiKV will not be able to split forever if no further action happens. We need to introduce other mechanism to notify PD that this TiKV has trouble now.

In PD, once it detects region split is throttled in that tikv, then it can immediately schedule region balance to recover from it. PD has global view and thus it's easy to implement such logic.
@rleungx

[overvenus]

The concept of limiting the number of regions per TiKV can be further expanded
to include the implementation of the Dynamic Region proposal. After discussing
with @tonyxuqqi, here are our initial thoughts:

1. It is important to control the total number of regions. This number should be
   within a reasonable range, neither too many nor too few. This means that in a
   cluster with a fixed number of TiKV nodes, TiKV should split regions more
   aggressively when dealing with small data volumes, while PD should merge
   regions more aggressively when handling large data volumes.
   Eventually, we aim to deprecate the TiKV config
   coprocessor.region-max-size/keys and PD config max-merge-region-size/keys.

2. PD should determine the ideal number of regions based on various statistics
   such as the cluster's data volume, node count, table count, and so on. PD has
   a global view of the cluster, which allows it to make informed decisions on
   the optimal region count.

Considerations for implementation:

• Throttling region split may have a negative impact on performance. Therefore,
  when the number of regions reaches 60% of the upper limit, PD should be more
  aggressive in merging regions.

• TiKV's merging process should be efficient, meaning that it should be fast
  and have a minimal performance impact (although this may be challenging to
  achieve).

[bufferflies]

there are some condition that region can't be merged, such as:

1. region is new, init in split-merge-interval
2. region is too hot
3. placement rule limit
4. region in different table .
5. region has the no-merged lable

and some api may split too many peer, this api should be limit.

[bufferflies]

Btw，the region-max-size can also be controlled by pd. Region should be merge if this region has no loads on it , and be split if this region is bigger and hot. TIKV don't care about the region size limit.