Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUPPORT]--source-ordering-field should be deprecated to honor precombine field #12002

Open
bithw1 opened this issue Sep 25, 2024 · 3 comments
Open

Comments

@bithw1
Copy link

bithw1 commented Sep 25, 2024

I am using Hudi 0.15, I find that ORDERING_FIELD has been deprecated with PRECOMBINE_FIELD_NAME,

  /** @deprecated Use {@link HoodieWriteConfig#PRECOMBINE_FIELD_NAME} and its methods instead */
  @Deprecated
  public static final ConfigProperty<String> ORDERING_FIELD = ConfigProperty
      .key(PAYLOAD_ORDERING_FIELD_PROP_KEY)
      .defaultValue("ts")
      .markAdvanced()
      .withDocumentation("Table column/field name to order records that have the same key, before "
          + "merging and writing to storage.");

but HoodieStreamer still use --source-ordering-field as parameter, I think HoodieStreamer should also deprecate --source-ordering-field and introduce a new parameter to honor precombine field to make concept consistent

    @Parameter(names = {"--source-ordering-field"}, description = "Field within source record to decide how"
        + " to break ties between records with same key in input data. Default: 'ts' holding unix timestamp of record")
    public String sourceOrderingField = "ts";

@danny0405
Copy link
Contributor

Agree, maybe you can fire a fix for it, we can firstly deprecate it.

@ad1happy2go
Copy link
Collaborator

@bithw1 Do you want to contribute on this?

@kazdy
Copy link
Contributor

kazdy commented Oct 19, 2024

IIRC source ordering field config in hudi streamer is being used to set precombine field, so this is really just a naming mismatch/confusion (?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Awaiting Triage
Development

No branches or pull requests

4 participants