Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change the merge behavior when no primary key and merge key are specified to append #1005

Closed
rudolfix opened this issue Feb 26, 2024 · 0 comments · Fixed by #1225
Closed
Assignees
Labels
community This issue came from slack community workspace enhancement New feature or request

Comments

@rudolfix
Copy link
Collaborator

Feature description

When no primary key and no merge keys are defined, the staging dataset is appended to the destination dataset. A warning is issued.

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

Current behavior, which is falling back to replace is not intuitive and may lead to a loss of data. The default dlt behavior is append so that would be expected. Also switching from append to merge and forgetting to set the keys will lead to data loss.

Also, which is more interesting case, this allows the users to do append via staging. So data is moved once all jobs for a given table chain is loaded (and for some databases - in transactional manner)

Proposed solution

  1. skip delete part in sql merge job if no keys are defined
  2. issue a warning in _verify_schema when no keys are defined

Related issues

No response

@rudolfix rudolfix added enhancement New feature or request community This issue came from slack community workspace labels Feb 26, 2024
@sh-rp sh-rp moved this from Todo to Planned in dlt core library Feb 27, 2024
@sh-rp sh-rp self-assigned this Feb 27, 2024
@sh-rp sh-rp moved this from Planned to In Progress in dlt core library Mar 4, 2024
@rudolfix rudolfix moved this from In Progress to Planned in dlt core library Mar 25, 2024
@sh-rp sh-rp moved this from Planned to In Progress in dlt core library Apr 2, 2024
@rudolfix rudolfix moved this from In Progress to Planned in dlt core library Apr 2, 2024
@sh-rp sh-rp moved this from Planned to In Progress in dlt core library Apr 17, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community This issue came from slack community workspace enhancement New feature or request
Projects
Status: Done
2 participants