0
# Cloudpickle Integration
1

2
Loky provides transparent integration with cloudpickle to handle serialization of functions and objects that cannot be pickled with the standard library. This enables parallel execution of interactively defined functions, lambda expressions, and complex objects.
3

4
## Capabilities
5

6
### Wrap Non-Picklable Objects
7

8
Wraps objects that cannot be pickled with standard pickle using cloudpickle for serialization.
9

10
```python { .api }
11
def wrap_non_picklable_objects(obj, keep_wrapper=True):
12
    """
13
    Wrapper for non-picklable objects to use cloudpickle for serialization.
14
    
15
    Parameters:
16
    - obj: The object to wrap (function, class, or instance)
17
    - keep_wrapper (bool): Whether to keep the wrapper after deserialization
18
    
19
    Returns:
20
    CloudpickledObjectWrapper or CloudpickledClassWrapper: Wrapped object
21
    
22
    Note:
23
    This wrapper tends to slow down serialization as cloudpickle is typically
24
    slower than pickle. The proper solution is to avoid defining functions in
25
    main scripts and implement __reduce__ methods for complex classes.
26
    """
27
```
28

29
### Set Custom Pickler
30

31
Configure a custom pickler for loky's inter-process communication.
32

33
```python { .api }
34
def set_loky_pickler(loky_pickler=None):
35
    """
36
    Set the pickler used by loky for inter-process communication.
37
    
38
    Parameters:
39
    - loky_pickler (optional): Custom pickler class. If None, resets to default
40
    
41
    Returns:
42
    None
43
    """
44

45
def register(type_, reduce_function):
46
    """
47
    Register a reduce function for objects of the given type.
48
    
49
    Parameters:
50
    - type_: The type to register a reducer for
51
    - reduce_function: Function that reduces objects of the given type
52
    
53
    Returns:
54
    None
55
    """
56

57
def dump(obj, file, reducers=None, protocol=None):
58
    """
59
    Pickle an object to a file using loky's pickling system.
60
    
61
    Parameters:
62
    - obj: Object to pickle
63
    - file: File-like object to write to
64
    - reducers (dict, optional): Custom reducers to use
65
    - protocol (int, optional): Pickle protocol version
66
    
67
    Returns:
68
    None
69
    """
70

71
def dumps(obj, reducers=None, protocol=None):
72
    """
73
    Pickle an object to bytes using loky's pickling system.
74
    
75
    Parameters:
76
    - obj: Object to pickle  
77
    - reducers (dict, optional): Custom reducers to use
78
    - protocol (int, optional): Pickle protocol version
79
    
80
    Returns:
81
    bytes: Pickled object as bytes
82
    """
83
```
84

85
### Wrapper Classes
86

87
Internal wrapper classes for handling non-picklable objects.
88

89
```python { .api }
90
class CloudpickledObjectWrapper:
91
    """Base wrapper for objects that need cloudpickle serialization."""
92
    def __init__(self, obj, keep_wrapper=False): ...
93
    def __reduce__(self): ...
94
    def __getattr__(self, attr): ...
95

96
class CallableObjectWrapper(CloudpickledObjectWrapper):
97
    """Wrapper that preserves callable property of wrapped objects."""
98
    def __call__(self, *args, **kwargs): ...
99
```
100

101
## Usage Examples
102

103
### Wrapping Lambda Functions
104

105
```python
106
from loky import get_reusable_executor, wrap_non_picklable_objects
107

108
# Lambda functions cannot be pickled with standard pickle
109
lambda_func = lambda x: x * x + 1
110

111
# Wrap the lambda for use with loky
112
wrapped_lambda = wrap_non_picklable_objects(lambda_func)
113

114
executor = get_reusable_executor(max_workers=2)
115

116
# Use wrapped lambda in parallel execution
117
results = list(executor.map(wrapped_lambda, range(10)))
118
print(f"Lambda results: {results}")
119
```
120

121
### Wrapping Interactively Defined Functions
122

123
```python
124
from loky import get_reusable_executor, wrap_non_picklable_objects
125

126
# Function defined in interactive session or __main__ module
127
def interactive_function(x, multiplier=2):
128
    """This function is defined interactively and needs wrapping."""
129
    import math
130
    return math.pow(x * multiplier, 2)
131

132
# Wrap for parallel execution
133
wrapped_func = wrap_non_picklable_objects(interactive_function)
134

135
executor = get_reusable_executor(max_workers=3)
136
inputs = [1, 2, 3, 4, 5]
137
results = list(executor.map(wrapped_func, inputs))
138
print(f"Interactive function results: {results}")
139
```
140

141
### Wrapping Classes
142

143
```python
144
from loky import get_reusable_executor, wrap_non_picklable_objects
145

146
# Class defined in main module
147
class DataProcessor:
148
    def __init__(self, scale_factor):
149
        self.scale_factor = scale_factor
150
    
151
    def process(self, data):
152
        return [x * self.scale_factor for x in data]
153

154
# Wrap the class
155
WrappedProcessor = wrap_non_picklable_objects(DataProcessor)
156

157
def process_with_wrapped_class(data_chunk):
158
    processor = WrappedProcessor(2.5)
159
    return processor.process(data_chunk)
160

161
executor = get_reusable_executor(max_workers=2)
162
data_chunks = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
163
results = list(executor.map(process_with_wrapped_class, data_chunks))
164
print(f"Class processing results: {results}")
165
```
166

167
### Automatic Wrapping
168

169
Loky automatically detects and wraps certain types of non-picklable objects:
170

171
```python
172
from loky import get_reusable_executor
173
from functools import partial
174

175
def multiply_add(x, multiplier, addend):
176
    return x * multiplier + addend
177

178
# partial objects are automatically wrapped
179
partial_func = partial(multiply_add, multiplier=3, addend=10)
180

181
executor = get_reusable_executor(max_workers=2)
182

183
# No explicit wrapping needed - loky handles it automatically
184
results = list(executor.map(partial_func, range(5)))
185
print(f"Partial function results: {results}")
186
```
187

188
### Custom Pickler Configuration
189

190
```python
191
from loky import set_loky_pickler, get_reusable_executor
192
import pickle
193

194
class CustomPickler(pickle.Pickler):
195
    """Custom pickler with special handling for certain types."""
196
    
197
    def save_global(self, obj, name=None):
198
        # Custom handling for global objects
199
        print(f"Pickling global object: {obj}")
200
        super().save_global(obj, name)
201

202
# Set custom pickler
203
set_loky_pickler(CustomPickler)
204

205
def test_function(x):
206
    return x ** 2
207

208
executor = get_reusable_executor(max_workers=2)
209
results = list(executor.map(test_function, [1, 2, 3]))
210

211
# Reset to default pickler
212
set_loky_pickler(None)
213
```
214

215
### Custom Type Reduction
216

217
```python
218
from loky.backend.reduction import register, dump, dumps
219
from loky import get_reusable_executor
220
import io
221

222
class CustomData:
223
    """Custom class that needs special serialization."""
224
    def __init__(self, value):
225
        self.value = value
226
        self._computed_result = None
227
    
228
    def compute_expensive_result(self):
229
        # Simulate expensive computation that we don't want to serialize
230
        if self._computed_result is None:
231
            self._computed_result = self.value * 1000
232
        return self._computed_result
233

234
def reduce_custom_data(obj):
235
    """Custom reducer that only serializes the essential data."""
236
    return rebuild_custom_data, (obj.value,)
237

238
def rebuild_custom_data(value):
239
    """Rebuild CustomData from reduced form."""
240
    return CustomData(value)
241

242
# Register the custom reducer
243
register(CustomData, reduce_custom_data)
244

245
def process_custom_data(data):
246
    """Task that processes CustomData objects."""
247
    result = data.compute_expensive_result()
248
    return f"Processed {data.value} -> {result}"
249

250
# Use with custom serialization
251
executor = get_reusable_executor(max_workers=2)
252

253
# Create objects that will use custom serialization
254
custom_objects = [CustomData(i) for i in range(5)]
255
results = list(executor.map(process_custom_data, custom_objects))
256

257
for i, result in enumerate(results):
258
    print(f"Object {i}: {result}")
259
```
260

261
### Manual Pickling Operations
262

263
```python
264
from loky.backend.reduction import dump, dumps
265
import io
266

267
class ComplexObject:
268
    def __init__(self, data):
269
        self.data = data
270
    
271
    def process(self):
272
        return sum(self.data)
273

274
# Create test object
275
obj = ComplexObject([1, 2, 3, 4, 5])
276

277
# Serialize to bytes
278
serialized_bytes = dumps(obj)
279
print(f"Serialized size: {len(serialized_bytes)} bytes")
280

281
# Serialize to file
282
with io.BytesIO() as buffer:
283
    dump(obj, buffer)
284
    file_size = buffer.tell()
285
    print(f"File serialization size: {file_size} bytes")
286

287
# Use with custom reducers
288
custom_reducers = {
289
    ComplexObject: lambda obj: (ComplexObject, (obj.data,))
290
}
291

292
custom_serialized = dumps(obj, reducers=custom_reducers)
293
print(f"Custom serialized size: {len(custom_serialized)} bytes")
294
```
295

296
### Handling Closures
297

298
```python
299
from loky import get_reusable_executor, wrap_non_picklable_objects
300

301
def create_closure_function(base_value):
302
    """Create a closure that captures base_value."""
303
    def closure_func(x):
304
        return x + base_value  # References base_value from outer scope
305
    return closure_func
306

307
# Closures need wrapping due to captured variables
308
closure = create_closure_function(100)
309
wrapped_closure = wrap_non_picklable_objects(closure)
310

311
executor = get_reusable_executor(max_workers=2)
312
results = list(executor.map(wrapped_closure, range(5)))
313
print(f"Closure results: {results}")
314
```
315

316
### Nested Function Handling
317

318
```python
319
from loky import get_reusable_executor
320

321
def outer_function():
322
    """Outer function that defines a nested function."""
323
    
324
    def nested_worker(data):
325
        # Nested functions are automatically wrapped
326
        return [x * 2 for x in data]
327
    
328
    executor = get_reusable_executor(max_workers=2)
329
    
330
    # Loky automatically detects and wraps nested functions
331
    data_chunks = [[1, 2], [3, 4], [5, 6]]
332
    results = list(executor.map(nested_worker, data_chunks))
333
    return results
334

335
# Call function with nested parallel processing
336
results = outer_function()
337
print(f"Nested function results: {results}")
338
```
339

340
## Best Practices
341

342
### Performance Considerations
343

344
- **Minimize Cloudpickle Usage**: While convenient, cloudpickle is slower than standard pickle
345
- **Define Functions at Module Level**: Avoid interactive definitions when possible
346
- **Implement __reduce__ Methods**: For custom classes, implement proper serialization
347

348
### Memory Management
349

350
- **Control Wrapper Retention**: Use `keep_wrapper=False` when wrapper is not needed after deserialization
351
- **Cache Wrapped Objects**: Reuse wrapped objects instead of wrapping repeatedly
352

353
### Error Handling
354

355
```python
356
from loky import get_reusable_executor, wrap_non_picklable_objects
357
import pickle
358

359
def problematic_function(x):
360
    # Function that might have serialization issues
361
    return x
362

363
try:
364
    # Attempt standard execution
365
    executor = get_reusable_executor(max_workers=2)
366
    results = list(executor.map(problematic_function, [1, 2, 3]))
367
except (pickle.PicklingError, AttributeError) as e:
368
    print(f"Pickling failed: {e}")
369
    # Fall back to wrapped version
370
    wrapped_func = wrap_non_picklable_objects(problematic_function)
371
    results = list(executor.map(wrapped_func, [1, 2, 3]))
372
    print(f"Wrapped execution succeeded: {results}")
373
```