You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When no primary key and no merge keys are defined, the staging dataset is appended to the destination dataset. A warning is issued.
Are you a dlt user?
Yes, I'm already a dlt user.
Use case
Current behavior, which is falling back to replace is not intuitive and may lead to a loss of data. The default dlt behavior is append so that would be expected. Also switching from append to merge and forgetting to set the keys will lead to data loss.
Also, which is more interesting case, this allows the users to do append via staging. So data is moved once all jobs for a given table chain is loaded (and for some databases - in transactional manner)
Proposed solution
skip delete part in sql merge job if no keys are defined
issue a warning in _verify_schema when no keys are defined
Related issues
No response
The text was updated successfully, but these errors were encountered:
Feature description
When no primary key and no merge keys are defined, the staging dataset is appended to the destination dataset. A warning is issued.
Are you a dlt user?
Yes, I'm already a dlt user.
Use case
Current behavior, which is falling back to replace is not intuitive and may lead to a loss of data. The default dlt behavior is append so that would be expected. Also switching from append to merge and forgetting to set the keys will lead to data loss.
Also, which is more interesting case, this allows the users to do append via staging. So data is moved once all jobs for a given table chain is loaded (and for some databases - in transactional manner)
Proposed solution
_verify_schema
when no keys are definedRelated issues
No response
The text was updated successfully, but these errors were encountered: