Experiment: Guard add_limit counter using lock #1277
Conversation
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
| @@ -343,7 +344,9 @@ def _gen_wrap(gen: TPipeStep) -> TPipeStep: | |||
| for i in gen: # type: ignore # TODO: help me fix this later | |||
| yield i | |||
| if i is not None: | |||
| count += 1 | |||
| with counter_lock: | |||
There was a problem hiding this comment.
@sultaniman I think all this code is run on the main pipeline thread anyway, so the lock will not help here, but I'm not 100% sure
There was a problem hiding this comment.
Tbh I also didn't dig deeper, my theory was that we submit the work to ThreadPoolExecutor which can potentially case data races.
There was a problem hiding this comment.
operations on integers in CPython are atomic so we are not locking anything. IMO race condition goes deep here and we are better off by making our tests to accept 10th element 🤷
sultaniman
force-pushed
the
fix/add-limit-lock
branch
from
e7e1b9a
to
1fbc3dd
Compare
sultaniman
force-pushed
the
fix/add-limit-lock
branch
from
1fbc3dd
to
5a9748c
Compare
|
@sultaniman @sh-rp this kind of things should not go into PR, let's discuss them on Slack or f2f. we just waste CPU cycles on running CI |
No description provided.