always serialize the function using cloudpickle #281
Conversation
This was a suggestion of @jbweston some time ago. Using this, the leaners are picklable regardless of the serialization package chosen.
Codecov Report
@@ Coverage Diff @@
## master #281 +/- ##
==========================================
+ Coverage 80.57% 80.72% +0.15%
==========================================
Files 34 34
Lines 4591 4601 +10
Branches 828 828
==========================================
+ Hits 3699 3714 +15
+ Misses 769 764 -5
Partials 123 123
Continue to review full report at Codecov.
|
|
As noted in cloudpipe/cloudpickle#375, the performance problem when serializing large collections (list, dicts, sets) of small object is caused by the fact that prior to Python 3.8, cloudpickle had not other choice than subclass the Python implementation of the Since Python 3.8, cloudpickle can subclass the C implementation of the |
|
@ogrisel, thanks a lot for your explanation (also in cloudpipe/cloudpickle#375 (comment))! |
|
LGTM, but what about |
Description
This was a suggestion of @jbweston some time ago.
Using this, the learners are picklable regardless of the serialization package chosen.
This also means that we can e.g. use lambda functions when using a
ProcessPoolExecutorand do not needlokyper se.I am making these changes now because I am finding that in some cases it's not preferable to use
cloudpickleto serialize simple dictionaries (see cloudpipe/cloudpickle#375).Checklist
pre-commit run --all(first install usingpip install pre-commit)pytestpassedType of change
Check relevant option(s).