-
Notifications
You must be signed in to change notification settings - Fork 252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support to dill serializer #406
base: main
Are you sure you want to change the base?
Conversation
I can't seem to find the requirements file to add |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm thinking of compatibility between dill and pickle
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
E ModuleNotFoundError: No module named 'dill'
Yes, as I've stated in my previous comment, I've failed to locate the requirements file. I haven't been able to include |
Converting to draft until the PR is ready and working. Thank you 🙏 |
I am currently searching for the place to add the |
According to the test suite, it seems it is ready for review. Please tell me if it isn't! |
@@ -166,6 +166,42 @@ def _is_build_command(argv=sys.argv, cmds=('install', 'build', 'bdist')): | |||
return arg | |||
|
|||
|
|||
def _strip_comments(l): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are these changes a must have?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These changes are at least partially needed to include the dill
library. I used the setup.py
of the celery
project to make this. There may not be a need to this, I thought it best to make something coherent with such a closely related project. I am open to discussion though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this change brings to light a design flaw with the PR.
I understand using pickle is limiting, but it appears to be a reason why there weren't any requirement.txt files, ever (except for the tests).
Adding a dependency (for dill
in this case) might be “against the design” of billiard, which is why there wasn’t any support for new dependencies in the setup.py in the first place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it appears to be a reason why there weren't any requirement.txt files, ever (except for the tests)
What would be the reason? I must say that it isn't obvious from the outside. Would it be that it would add a lot of lines to the setup.py
? I think not for it doesn't seem a valid reason, unless I am missing something.
I think this change brings to light a design flaw with the PR
I must say I don't understand why adding a dependency would be a design flaw here, especially as it is designed to be a drop-in replacement of the currently used one (as you mentioned in the related PR), only adding in capabilities. Again, this would open up the possibility to pass many unsupported types of objects as parameters (while I have only encountered the billiard
library when using celery
, I expect there are other places it is used, and I cannot imagine a situation where it would be detrimental to add these capabilities).
Adding a dependency (for dill in this case) might be “against the design” of billiard, which is why there wasn’t any support for new dependencies in the setup.py in the first place.
If there is a philosophical issue that I am not aware of in addition to the dependency introduction, I can totally accept it, but regarding this one, it does not seem (to me) that big of a deal. I am well aware that this is ultimately not my decision to make, but is there any underlying reasons this wouldn't be possible?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this resolved?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I resolve this conversation?
@@ -441,7 +441,7 @@ def _make_recv_method(self, conn): | |||
if hasattr(conn, 'get_payload') and conn.get_payload: | |||
get_payload = conn.get_payload | |||
|
|||
def _recv(timeout, loads=pickle_loads): | |||
def _recv(timeout, loads=dill_loads): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you please update or add some tests to check the dill usage?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you specify what sort of test you would like added? Pool instanciation is already tested, as well as messages sending and receiving (which should handle the tests on whether this would introduce side effects or not). I have added some tests following this review, but I don't know if they meet your expectations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this resolved?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I resolve this conversation?
requirements/default.txt
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@auvipy notice this is a new file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it an issue? Or is this related to the conversation below @auvipy's change request? (If it is related, shouldn't we resolve this one, to avoid having multiple threads open on the same issues?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I resolve this conversation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears this PR only changes the default from pickle
to dill
, but the matching celery PR celery/celery#9100 is where the dill
is actually “requested”.
Shouldn’t the celery use-case be enough so the billiard changes won’t be needed?
This is a follow-up to my other comment: https://github.com/celery/billiard/pull/406/files#r1663972900
It is necessary to have the Line 444 in 81cc942
This is what led me to also modify the billiard deserializer.
|
Fixes #405.
Must be merged along with celery/celery#9100