[Feature][Flink] Add external configuration parameters #5480

zhilinli123 · 2023-09-13T11:54:16Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

ruanwenjun · 2023-09-14T06:05:32Z

LGTM, please add doc about this.
https://github.com/apache/seatunnel/blob/dev/docs/en/other-engine/flink.md

ruanwenjun

LGTM

zhilinli123 · 2023-09-14T07:48:12Z

There is a problem with some parameter configuration. I am working on it. Do not merge it
@ruanwenjun @TyrantLucifer

TyrantLucifer · 2023-09-14T07:50:03Z

There is a problem with some parameter configuration. I am working on it. Do not merge it @ruanwenjun @TyrantLucifer

Waiting CI/CD and offer some screenshots to verify it worked.

TyrantLucifer

Please fix dead links in your docs.

zhilinli123 · 2023-09-15T06:49:50Z

PTAL: @TyrantLucifer @ruanwenjun I added some CI

TyrantLucifer · 2023-09-17T05:07:56Z

PTAL: @TyrantLucifer @ruanwenjun I added some CI

Overall LGTM, but the test case that you added I didn't any way to see the parameter worded really. Could you please compiled the project and try verify it in a local flink env and offer the screen shots for it？

zhilinli123 · 2023-09-17T13:23:26Z

conf file

env {
  # You can set flink configuration here
  execution.parallelism = 2
  job.mode = "STREAMING"
  flink.execution.checkpointing.interval = 5000
  flink.execution.checkpointing.unaligned.enabled = true
  flink.execution.checkpointing.aligned-checkpoint-timeout = 100000
  flink.jobstore.cache-size = 52428801
  flink.state.backend.rocksdb.predefined-options = SPINNING_DISK_OPTIMIZED_HIGH_MEM
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  FakeSource {
    parallelism = 2
    result_table_name = "fake"
    row.num = 16
    schema = {
      fields {
        name = "string"
        age = "int"
      }
    }
  }

  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

transform {

  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform-v2
}

sink {
  Console {
    parallelism = 3
  }

  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

PTAL: @TyrantLucifer @ruanwenjun @EricJoy2048 I added some screenshots to demonstrate this

TyrantLucifer · 2023-10-17T07:14:22Z

conf file

env {
  # You can set flink configuration here
  execution.parallelism = 2
  job.mode = "STREAMING"
  flink.execution.checkpointing.interval = 5000
  flink.execution.checkpointing.unaligned.enabled = true
  flink.execution.checkpointing.aligned-checkpoint-timeout = 100000
  flink.jobstore.cache-size = 52428801
  flink.state.backend.rocksdb.predefined-options = SPINNING_DISK_OPTIMIZED_HIGH_MEM
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  FakeSource {
    parallelism = 2
    result_table_name = "fake"
    row.num = 16
    schema = {
      fields {
        name = "string"
        age = "int"
      }
    }
  }

  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

transform {

  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform-v2
}

sink {
  Console {
    parallelism = 3
  }

  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

PTAL: @TyrantLucifer @ruanwenjun @EricJoy2048 I added some screenshots to demonstrate this

I think you should ensure the parameters injected in real flink jobs, not only codes. You should submit a seatunnel on flink engine and make a snapshot to verify it worked.

zhilinli123 · 2023-10-17T07:47:24Z

conf file

env {
  # You can set flink configuration here
  execution.parallelism = 2
  job.mode = "STREAMING"
  flink.execution.checkpointing.interval = 5000
  flink.execution.checkpointing.unaligned.enabled = true
  flink.execution.checkpointing.aligned-checkpoint-timeout = 100000
  flink.jobstore.cache-size = 52428801
  flink.state.backend.rocksdb.predefined-options = SPINNING_DISK_OPTIMIZED_HIGH_MEM
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  FakeSource {
    parallelism = 2
    result_table_name = "fake"
    row.num = 16
    schema = {
      fields {
        name = "string"
        age = "int"
      }
    }
  }

  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

transform {

  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform-v2
}

sink {
  Console {
    parallelism = 3
  }

  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

PTAL: @TyrantLucifer @ruanwenjun @EricJoy2048 I added some screenshots to demonstrate this

I think you should ensure the parameters injected in real flink jobs, not only codes. You should submit a seatunnel on flink engine and make a snapshot to verify it worked.

I provide some screenshots of the flink run web Ui

EricJoy2048

LGTM

ruanwenjun

+1

zhilinli123 · 2023-10-25T02:43:04Z

PTAL: @Hisoka-X

Hisoka-X

Looking great! Thanks @zhilinli123 .

Hisoka-X · 2023-10-25T02:58:20Z

@TyrantLucifer Could you take a look again?

hailin0 · 2023-10-25T04:03:35Z

LGTM

passed ci

[feature][flink][job][conf] Add external configuration parameters

32afca1

ruanwenjun assigned zhilinli123 Sep 14, 2023

ruanwenjun requested a review from TyrantLucifer September 14, 2023 06:06

ruanwenjun added the flink label Sep 14, 2023

[feature][flink][job][conf] Add external configuration parameters

8ad9949

ruanwenjun previously approved these changes Sep 14, 2023

View reviewed changes

github-actions bot added approved reviewed labels Sep 14, 2023

TyrantLucifer previously requested changes Sep 14, 2023

View reviewed changes

github-actions bot removed approved reviewed labels Sep 14, 2023

[feature][flink][job][conf] Add external configuration parameters

749287f

zhilinli123 dismissed ruanwenjun’s stale review via 749287f September 14, 2023 10:45

zhilinli123 added 2 commits September 14, 2023 19:38

[feature][flink][job][conf] Add external configuration parameters

81ea12c

Merge branch 'apache:dev' into feature-flink-conf

e7667dd

zhilinli123 closed this Sep 15, 2023

Merge branch 'apache:dev' into feature-flink-conf

2c2ab00

zhilinli123 reopened this Sep 15, 2023

Merge remote-tracking branch 'origin/dev' into feature-flink-conf

741f02c

zhilinli123 added 2 commits September 18, 2023 10:08

rename jobName

6d91db0

Merge branch 'apache:dev' into feature-flink-conf

e46feb7

zhilinli123 closed this Sep 18, 2023

zhilinli123 reopened this Sep 18, 2023

zhilinli123 closed this Sep 18, 2023

zhilinli123 reopened this Sep 18, 2023

zhilinli123 closed this Sep 18, 2023

zhilinli123 reopened this Sep 18, 2023

zhilinli123 closed this Sep 18, 2023

zhilinli123 reopened this Sep 18, 2023

zhilinli123 added 2 commits September 18, 2023 22:27

add comment

97beac5

add comment

42c462e

Hisoka-X changed the title ~~[feature][flink][job][conf] Add external configuration parameters~~ [Feature][Flink] Add external configuration parameters Sep 19, 2023

Hisoka-X added the feature New feature label Sep 19, 2023

Hisoka-X added this to the 2.3.4 milestone Sep 19, 2023

zhilinli123 added 4 commits September 20, 2023 15:59

add comment

bdde9af

Merge branch 'apache:dev' into feature-flink-conf

a243f72

Merge branch 'apache:dev' into feature-flink-conf

ba9a0db

Merge branch 'apache:dev' into feature-flink-conf

9f27765

zhilinli123 closed this Sep 21, 2023

zhilinli123 reopened this Sep 21, 2023

EricJoy2048 reviewed Oct 17, 2023

View reviewed changes

zhilinli123 requested review from ruanwenjun, EricJoy2048 and TyrantLucifer October 24, 2023 09:40

ruanwenjun approved these changes Oct 24, 2023

View reviewed changes

Hisoka-X approved these changes Oct 25, 2023

View reviewed changes

Hisoka-X merged commit 90bc2fe into apache:dev Oct 26, 2023
8 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Flink] Add external configuration parameters #5480

[Feature][Flink] Add external configuration parameters #5480

zhilinli123 commented Sep 13, 2023

ruanwenjun commented Sep 14, 2023 •

edited

Loading

ruanwenjun left a comment

zhilinli123 commented Sep 14, 2023

TyrantLucifer commented Sep 14, 2023

TyrantLucifer left a comment

zhilinli123 commented Sep 15, 2023

TyrantLucifer commented Sep 17, 2023

zhilinli123 commented Sep 17, 2023 •

edited

Loading

TyrantLucifer commented Oct 17, 2023

conf file

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

zhilinli123 commented Oct 17, 2023 •

edited

Loading

conf file

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

EricJoy2048 left a comment

ruanwenjun left a comment

zhilinli123 commented Oct 25, 2023

Hisoka-X left a comment

Hisoka-X commented Oct 25, 2023

hailin0 commented Oct 25, 2023

[Feature][Flink] Add external configuration parameters #5480

[Feature][Flink] Add external configuration parameters #5480

Conversation

zhilinli123 commented Sep 13, 2023

Purpose of this pull request

Check list

ruanwenjun commented Sep 14, 2023 • edited Loading

ruanwenjun left a comment

Choose a reason for hiding this comment

zhilinli123 commented Sep 14, 2023

TyrantLucifer commented Sep 14, 2023

TyrantLucifer left a comment

Choose a reason for hiding this comment

zhilinli123 commented Sep 15, 2023

TyrantLucifer commented Sep 17, 2023

zhilinli123 commented Sep 17, 2023 • edited Loading

conf file

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

TyrantLucifer commented Oct 17, 2023

conf file

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

zhilinli123 commented Oct 17, 2023 • edited Loading

conf file

TestFlinkParameter

org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

EricJoy2048 left a comment

Choose a reason for hiding this comment

ruanwenjun left a comment

Choose a reason for hiding this comment

zhilinli123 commented Oct 25, 2023

Hisoka-X left a comment

Choose a reason for hiding this comment

Hisoka-X commented Oct 25, 2023

hailin0 commented Oct 25, 2023

ruanwenjun commented Sep 14, 2023 •

edited

Loading

zhilinli123 commented Sep 17, 2023 •

edited

Loading

zhilinli123 commented Oct 17, 2023 •

edited

Loading