Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add open/closed range arguments for incremental #1991

Draft
wants to merge 1 commit into
base: devel
Choose a base branch
from

Conversation

steinitzu
Copy link
Collaborator

Description

This allows configuring whether the incremental range is open or closed on both sides (start/end value).
Translates to changing the operators between > | >= / < | <= in sql database source and in incremental filtering.

Default is start_range=closed and end_range=open, meaning the exact initial value is included and the exact end value is excluded.
With start_range=open you get WHERE cursor > last_value instead of >=.
With end_range=closed you get ... cursor <= end_value so non-overlapping chunks are still possible without the start_value deduplication logic.

Related Issues

Additional Context

Copy link

netlify bot commented Oct 25, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 762d7cf
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/671ae7855f149e0008663737

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support to customize incremental compare operators for sql_table
1 participant