Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel Processing #202

Open
itdependsnetworks opened this issue Feb 9, 2023 · 2 comments
Open

Parallel Processing #202

itdependsnetworks opened this issue Feb 9, 2023 · 2 comments
Labels
status: gathering feedback Further discussion is needed to determine this issue's scope and/or implementation type: enhancement

Comments

@itdependsnetworks
Copy link
Contributor

Environment

  • DiffSync version: latest

Proposed Functionality

The ability to process parts of the sync that do not effect each other.

This likely requires a facility that developers can describe what can be parallel as well as potentially finding specific scenarios where it can automagically happen.

Use Case

Use cases that would benefit from this include:

  • Getting Side A and Side B at the same time
  • Getting all of table 1 and all of table 2 from Side A at the same time
  • Update the model where there is no relationships at the same time
  • A parent of multiple children, each of those children can run at the same time
@chadell
Copy link
Collaborator

chadell commented Feb 10, 2023

isn't this something you could infer based on the children relationships? I mean, all the object of one type, e.g site, should be able to be processed in parallel. Obviously, you can find shared children, but this conflict can be postponed for a final merge.
Don't know if assuming, by default, that the objects of parent models can run in parallel makes sense. Then, if the library offers a remediation final step using the already supported Redis memory, you could run multiprocessing.
does it make sense to you? or maybe missing some point?

@itdependsnetworks
Copy link
Contributor Author

itdependsnetworks commented Feb 10, 2023

Some can be inferred, some may not be able to be inferred, but yes, that is the point

@Kircheneer Kircheneer added status: gathering feedback Further discussion is needed to determine this issue's scope and/or implementation type: major feature labels Feb 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: gathering feedback Further discussion is needed to determine this issue's scope and/or implementation type: enhancement
Projects
None yet
Development

No branches or pull requests

3 participants