Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use insert .. on conflict for bulk import operations #3301

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

elexisvenator
Copy link
Contributor

Updates the behaviour of BulkInsertMode.OverwriteExisting to use insert ... on conflict do update. This is one less db statement than before and a less prone to constraint issues.
Also updates the behaviour of BulkInsertMode.IgnoreDuplicates to use insert ... on conflict do nothing.

(Wasn't going to create this so soon but ran into an issue where spamming some seed data resulted in unique constraint violations when using OverwriteExisting)

Tests pass but I had to delete the checked in generated code files. I am not sure what the correct way to regenerate these files is - can this be added to the doco somewhere so I can fix the issue myself next time 🙇 .

@jeremydmiller
Copy link
Member

I'm going to pull this down locally and try it first. Can't see any reason why this would fail offhand

@jeremydmiller
Copy link
Member

This is a breaking change because it requires rebuilding all generated code. Hmm, think this might have to wait for Marten 8.

Which I do actually want to get to relatively soon-ish

@elexisvenator
Copy link
Contributor Author

Hi @jeremydmiller,

I have given this another pass. Bulk inserting now supports conditional overwrites - when in BulkLoadMode.OverwriteExisting it is now possible to pass in a condition that must pass before a document is overwritten. This is very useful for bulk loading from external sources as it allows for safe out-of-order imports as long as documents have some sort of versioning column. Previously the only way to do this was to first query all the documents you wanted to import and drop the ones that were out of date. This way should be a lot faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants