[BUG] Memory consumption during iterative migration is very high

[pschoen-itsc]

Describe the bug
We recently try to run an iterative migration, but it always was killed by the OS because it kept using to much memory. It was a collection with hundred thousands of rather large documents and after a few minutes the migration scripts used up over 10 GiBs of memory.

To Reproduce
Create a big collection (best multiple GiBs on disk) and run an iterative migration on them.

Expected behavior
Memory consumption does not grow during the migration

Additional context
Using a free fall migration works fine. From the implementation of the iterative migration it is clear where to "problem" is, because all operations are collected and only executed at the end, so every document has to be held in memory. I'm not sure about the batching logic which is implemented there, but wouldn't it be a solution to directly execute each batch instead of collecting them?

[pschoen-itsc]

I'm happy to create a PR with my proposed solution, but wanted to know first what are the reasons for the current implementation.

[github-actions]

This issue is stale because it has been open 30 days with no activity.