Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MYSQL CDC到Paimon0.9 flink1.9版本,提交任务会报:Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype #8049

Open
2 of 3 tasks
lxxstarii opened this issue Nov 14, 2024 · 0 comments
Labels

Comments

@lxxstarii
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

MYSQL CDC到Paimon0.9 flink1.9版本,提交任务会报:Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype

SeaTunnel Version

2.3.8

SeaTunnel Config

env {
  parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
}

source {
  Mysql-CDC {
    base-url = "jdbc:mysql://192.168.0.44:3306/company"
    username = "root"
    password = "000000"
    table-names = ["company.userinfo2"]
  }
}

transform {
}

sink {
  Paimon {
    catalog_name="paimon_hive"
    warehouse="hdfs://hadoop100:8020/paimon/hive"
    database="default"
    table="test_paimon_03"
  }
}

Running Command

start-seatunnel-flink-15-connector-v2.sh

Error Exception

file:/root/Downloads/home/software/apache-seatunnel-2.3.8/starter/seatunnel-flink-15-starter.jar
2024-11-13 19:25:07,517 INFO  org.apache.seatunnel.core.starter.flink.execution.FlinkExecution [] - Inject jar to config: file:/root/Downloads/home/software/apache-seatunnel-2.3.8/lib/seatunnel-hadoop3-3.1.4-uber.jar
2024-11-13 19:25:07,517 INFO  org.apache.seatunnel.core.starter.flink.execution.FlinkExecution [] - Inject jar to config: file:/root/Downloads/home/software/apache-seatunnel-2.3.8/connectors/connector-cdc-mysql-2.3.8.jar
2024-11-13 19:25:07,518 INFO  org.apache.seatunnel.core.starter.flink.execution.FlinkExecution [] - Inject jar to config: file:/root/Downloads/home/software/apache-seatunnel-2.3.8/connectors/connector-paimon-2.3.8.jar
2024-11-13 19:25:07,644 INFO  org.apache.flink.api.java.typeutils.TypeExtractor            [] - class org.apache.seatunnel.api.table.type.SeaTunnelRow does not contain a setter for field fields
2024-11-13 19:25:07,644 INFO  org.apache.flink.api.java.typeutils.TypeExtractor            [] - Class class org.apache.seatunnel.api.table.type.SeaTunnelRow cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution.
2024-11-13 19:25:07,667 INFO  org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery [] - Load SeaTunnelSink Plugin from /root/Downloads/home/software/apache-seatunnel-2.3.8/connectors
2024-11-13 19:25:07,710 INFO  org.apache.seatunnel.connectors.seatunnel.paimon.security.PaimonSecurityContext [] - Hadoop config initialized: org.apache.seatunnel.connectors.seatunnel.paimon.config.PaimonHadoopConfiguration
2024-11-13 19:25:07,731 INFO  org.apache.seatunnel.connectors.seatunnel.paimon.security.PaimonSecurityContext [] - Hadoop config initialized: org.apache.seatunnel.connectors.seatunnel.paimon.config.PaimonHadoopConfiguration
2024-11-13 19:25:07,758 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - 

===============================================================================


2024-11-13 19:25:07,758 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Fatal Error, 

2024-11-13 19:25:07,759 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-11-13 19:25:07,759 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Reason:Flink job executed failed 

2024-11-13 19:25:07,761 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: Flink job executed failed
	at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:63)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
	at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:217)
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143)
	at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115)
	at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61)
	... 19 more
Caused by: org.apache.seatunnel.connectors.seatunnel.paimon.exception.PaimonConnectorException: ErrorCode:[PAIMON-06], ErrorDescription:[Load catalog failed] - Failed to perform SecurityContext.runSecured
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:89)
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73)
	at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64)
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214)
	... 22 more
Caused by: java.lang.RuntimeException: Unexpected error when trying to load service provider.
	at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:118)
	at org.apache.paimon.factories.FactoryUtil.discoverFactory(FactoryUtil.java:38)
	at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:73)
	at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66)
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.lambda$loadCatalog$1(PaimonCatalogLoader.java:87)
	at org.apache.paimon.security.SecurityContext.runSecured(SecurityContext.java:67)
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:86)
	... 25 more
Caused by: java.util.ServiceConfigurationError: org.apache.paimon.factories.Factory: Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype
	at java.util.ServiceLoader.fail(ServiceLoader.java:239)
	at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:109)
	... 31 more
 
2024-11-13 19:25:07,761 ERROR org.apache.seatunnel.core.starter.SeaTunnel                  [] - 
===============================================================================



2024-11-13 19:25:07,762 ERROR org.apache.flink.client.cli.CliFrontend                      [] - Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Flink job executed failed
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108) ~[flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026) ~[flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) ~[flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270) ~[flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) ~[flink-dist-1.19.1.jar:1.19.1]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
	at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_144]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) [flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar:3.1.1.7.1.1.0-565-9.0]
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) [flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) [flink-dist-1.19.1.jar:1.19.1]
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) [flink-dist-1.19.1.jar:1.19.1]
Caused by: org.apache.seatunnel.core.starter.exception.CommandExecuteException: Flink job executed failed
	at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:63) ~[?:?]
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
	... 12 more
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[API-09], ErrorDescription:[Handle save mode failed]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:217) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
	... 12 more
Caused by: org.apache.seatunnel.connectors.seatunnel.paimon.exception.PaimonConnectorException: ErrorCode:[PAIMON-06], ErrorDescription:[Load catalog failed] - Failed to perform SecurityContext.runSecured
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:89) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73) ~[?:?]
	at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
	... 12 more
Caused by: java.lang.RuntimeException: Unexpected error when trying to load service provider.
	at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:118) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.paimon.factories.FactoryUtil.discoverFactory(FactoryUtil.java:38) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:73) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.lambda$loadCatalog$1(PaimonCatalogLoader.java:87) ~[?:?]
	at org.apache.paimon.security.SecurityContext.runSecured(SecurityContext.java:67) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:86) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73) ~[?:?]
	at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
	... 12 more
Caused by: java.util.ServiceConfigurationError: org.apache.paimon.factories.Factory: Provider org.apache.paimon.flink.action.CloneActionFactory not a subtype
	at java.util.ServiceLoader.fail(ServiceLoader.java:239) ~[?:1.8.0_144]
	at java.util.ServiceLoader.access$300(ServiceLoader.java:185) ~[?:1.8.0_144]
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376) ~[?:1.8.0_144]
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
	at org.apache.paimon.factories.FactoryUtil.discoverFactories(FactoryUtil.java:109) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.paimon.factories.FactoryUtil.discoverFactory(FactoryUtil.java:38) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:73) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.lambda$loadCatalog$1(PaimonCatalogLoader.java:87) ~[?:?]
	at org.apache.paimon.security.SecurityContext.runSecured(SecurityContext.java:67) ~[paimon-flink-1.19-0.9.0.jar:0.9.0]
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalogLoader.loadCatalog(PaimonCatalogLoader.java:86) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.paimon.catalog.PaimonCatalog.open(PaimonCatalog.java:73) ~[?:?]
	at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.open(DefaultSaveModeHandler.java:64) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.handleSaveMode(SinkExecuteProcessor.java:214) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.SinkExecuteProcessor.execute(SinkExecuteProcessor.java:143) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.execute(FlinkExecution.java:115) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:61) ~[?:?]
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) ~[?:?]
	at org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
	... 12 more

Zeta or Flink or Spark Version

flink 1.9 Paimon0.9

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@lxxstarii lxxstarii added the bug label Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant