Dynamic allocation for writing parallelism in flink sink

[nicochen]

Here is a case that data throughput have significant difference between day and night. A flink job has an immutable parallelism of sinks once started. Take an un-keyed table as an example, 'none' distribution policy is select which means best utilization of all parallelism, while it creates biggest number of files even through small quality of data comes.

Therefore, I‘d like to propose a new distribution policy to dynamically allocate writing parallelism in streaming writing according to number of data. In this way, we can not only relieve the pressure from small files in hdfs, but also increate the efficiency of reading and optimisation work.

[nicochen]

@YesOrNo828 maybe we can take a look at this topic.

[YesOrNo828]

@nicochen Thanks for your input. This topic is interesting. And I have some questions:

1. Could this distribution policy work for tables with and without keys? It should ideally work on two kinds of tables. So it leads to the primary key table, how to reduce part of the task not out of the data and can ensure data consistency.

a new distribution policy to dynamically allocate writing parallelism in streaming writing according to number of data.

2. How to switch the shuffle rule according to the number of data?
3. Allocating the writing parallelism, Do you mean that some subtasks are still running but no data is being processed?