0
# Context and Configuration
1

2
Context management and configuration options for different process start methods and serialization behavior. Allows fine-grained control over process creation and object serialization in multiprocess applications.
3

4
## Capabilities
5

6
### Context Management
7

8
Functions for managing different process start methods and contexts.
9

10
```python { .api }
11
def get_context(method=None):
12
    """
13
    Get a context for a specific start method.
14
    
15
    Args:
16
        method: start method ('fork', 'spawn', 'forkserver', or None)
17
               None returns the current default context
18
    
19
    Returns:
20
        BaseContext: context object for the specified method
21
        
22
    Raises:
23
        ValueError: if method is not available on the platform
24
    """
25

26
def set_start_method(method, force=False):
27
    """
28
    Set the start method for creating processes.
29
    
30
    Args:
31
        method: start method ('fork', 'spawn', 'forkserver')
32
        force: if True, allow changing after context has been used
33
        
34
    Raises:
35
        RuntimeError: if context has already been set and force=False
36
        ValueError: if method is not available
37
    """
38

39
def get_start_method(allow_none=False):
40
    """
41
    Get the current start method.
42
    
43
    Args:
44
        allow_none: if True, return None if no method has been set
45
        
46
    Returns:
47
        str: current start method name
48
    """
49

50
def get_all_start_methods():
51
    """
52
    Get all available start methods for the platform.
53
    
54
    Returns:
55
        list[str]: list of available start methods, default first
56
    """
57
```
58

59
### Context Objects
60

61
Different context types for specific start methods.
62

63
```python { .api }
64
class BaseContext:
65
    """
66
    Base context class providing multiprocess functionality.
67
    """
68
    # All the standard multiprocess functions are available on context objects
69
    def Process(self, target=None, name=None, args=(), kwargs={}, daemon=None):
70
        """Create a Process using this context."""
71
    
72
    def Pool(self, processes=None, initializer=None, initargs=(), 
73
             maxtasksperchild=None):
74
        """Create a Pool using this context."""
75
    
76
    def Queue(self, maxsize=0):
77
        """Create a Queue using this context."""
78
    
79
    def Lock(self):
80
        """Create a Lock using this context."""
81
    
82
    def Manager(self):
83
        """Create a Manager using this context."""
84
    
85
    # ... all other multiprocess functions available
86
```
87

88
### Process Configuration
89

90
Functions for configuring process behavior and execution environment.
91

92
```python { .api }
93
def set_executable(executable):
94
    """
95
    Set the path to Python executable for spawned processes.
96
    
97
    Useful when embedding Python or using custom Python installations.
98
    Only affects 'spawn' start method.
99
    
100
    Args:
101
        executable: path to Python executable
102
    """
103

104
def set_forkserver_preload(module_names):
105
    """
106
    Set list of module names to preload in forkserver process.
107
    
108
    Modules are imported when the forkserver starts, potentially
109
    improving performance by avoiding repeated imports.
110
    Only affects 'forkserver' start method.
111
    
112
    Args:
113
        module_names: list of module names to preload
114
    """
115
```
116

117
### Serialization Configuration
118

119
Control over object serialization behavior.
120

121
```python { .api }
122
# Context property for controlling serialization
123
@property
124
def reducer(self):
125
    """
126
    Get the reducer used for object serialization.
127
    
128
    Returns:
129
        module: reduction module (typically dill-enhanced)
130
    """
131

132
@reducer.setter
133
def reducer(self, reduction):
134
    """
135
    Set the reducer for object serialization.
136
    
137
    Args:
138
        reduction: reduction module to use
139
    """
140
```
141

142
### Advanced Configuration
143

144
Functions for specialized configuration scenarios.
145

146
```python { .api }
147
def allow_connection_pickling():
148
    """
149
    Enable pickling of connection objects.
150
    
151
    Allows connection objects to be sent between processes.
152
    This is generally not recommended for security reasons.
153
    """
154

155
def freeze_support():
156
    """
157
    Add support for frozen executables on Windows.
158
    
159
    Should be called in the main module of programs that use
160
    multiprocess and are frozen with tools like py2exe or PyInstaller.
161
    """
162

163
def get_logger():
164
    """
165
    Get the logger used by multiprocess.
166
    
167
    Returns:
168
        Logger: The multiprocess logger instance
169
    """
170

171
def log_to_stderr(level=None):
172
    """
173
    Turn on logging and add a handler which prints to stderr.
174
    
175
    Args:
176
        level: Optional logging level to set
177
        
178
    Returns:
179
        Logger: The configured logger instance
180
    """
181
```
182

183
## Start Method Details
184

185
### Fork Method
186

187
Available on Unix-like systems (Linux, macOS). Creates child processes by forking the parent process.
188

189
**Characteristics:**
190
- Fast process creation
191
- Inherits parent's memory state
192
- Shares file descriptors and handles
193
- Can have issues with threads and certain libraries
194
- Default on Linux
195

196
**When to use:**
197
- Fast process startup needed
198
- Sharing large amounts of data
199
- Unix-only applications
200

201
### Spawn Method
202

203
Available on all platforms. Creates fresh Python interpreter processes.
204

205
**Characteristics:**
206
- Slower process creation
207
- Clean process state
208
- Better isolation
209
- Required on Windows
210
- Safer with threads
211
- Default on Windows and macOS (recent versions)
212

213
**When to use:**
214
- Cross-platform compatibility needed
215
- Using threading in parent process
216
- Need process isolation
217
- Windows applications
218

219
### Forkserver Method
220

221
Available on Unix-like systems. Uses a server process to create child processes.
222

223
**Characteristics:**
224
- Moderate startup time
225
- Clean process state after server startup
226
- Server process can preload modules
227
- Avoids fork-related issues
228
- Requires additional setup
229

230
**When to use:**
231
- Want benefits of fork with clean state
232
- Using threads that don't work well with fork
233
- Need to preload heavy modules
234

235
## Usage Examples
236

237
### Basic Context Usage
238

239
```python
240
import multiprocess as mp
241

242
# Get default context
243
ctx = mp.get_context()
244
print(f"Default method: {ctx.get_start_method()}")
245

246
# Get specific context
247
spawn_ctx = mp.get_context('spawn')
248
fork_ctx = mp.get_context('fork')  # Unix only
249

250
# Use context to create objects
251
with spawn_ctx.Pool(2) as pool:
252
    results = pool.map(lambda x: x**2, [1, 2, 3, 4])
253
    print(results)
254
```
255

256
### Setting Start Method
257

258
```python
259
import multiprocess as mp
260

261
def worker():
262
    print(f"Worker using method: {mp.get_start_method()}")
263

264
if __name__ == '__main__':
265
    # Must set start method before creating processes
266
    print(f"Available methods: {mp.get_all_start_methods()}")
267
    
268
    # Set start method
269
    mp.set_start_method('spawn')
270
    print(f"Set method to: {mp.get_start_method()}")
271
    
272
    # Create process
273
    p = mp.Process(target=worker)
274
    p.start()
275
    p.join()
276
```
277

278
### Method Comparison
279

280
```python
281
import multiprocess as mp
282
import time
283
import os
284

285
def simple_worker(method_name):
286
    """Simple worker to test process creation time"""
287
    pid = os.getpid()
288
    print(f"Worker PID {pid} using {method_name}")
289
    return pid
290

291
def benchmark_method(method_name, num_processes=4):
292
    """Benchmark process creation time for a method"""
293
    try:
294
        ctx = mp.get_context(method_name)
295
    except ValueError:
296
        print(f"Method {method_name} not available")
297
        return None
298
    
299
    start_time = time.time()
300
    
301
    with ctx.Pool(num_processes) as pool:
302
        results = pool.map(simple_worker, [method_name] * num_processes)
303
    
304
    end_time = time.time()
305
    duration = end_time - start_time
306
    
307
    print(f"{method_name}: {duration:.3f} seconds for {num_processes} processes")
308
    return duration
309

310
if __name__ == '__main__':
311
    print("Benchmarking start methods:")
312
    
313
    methods = mp.get_all_start_methods()
314
    print(f"Available methods: {methods}")
315
    
316
    for method in methods:
317
        benchmark_method(method)
318
```
319

320
### Context Isolation
321

322
```python
323
import multiprocess as mp
324

325
# Global variable to test inheritance
326
global_data = "Original data"
327

328
def show_global_data(context_method):
329
    """Show whether global data is inherited"""
330
    print(f"Method {context_method}: global_data = '{global_data}'")
331

332
def modify_and_show(context_method):
333
    """Modify global data and show it"""
334
    global global_data
335
    global_data = f"Modified by {context_method}"
336
    print(f"Method {context_method}: modified global_data = '{global_data}'")
337

338
if __name__ == '__main__':
339
    # Test different contexts
340
    methods = mp.get_all_start_methods()
341
    
342
    for method in methods:
343
        try:
344
            ctx = mp.get_context(method)
345
            print(f"\nTesting {method} method:")
346
            
347
            # Show original data
348
            p1 = ctx.Process(target=show_global_data, args=(method,))
349
            p1.start()
350
            p1.join()
351
            
352
            # Try to modify data
353
            p2 = ctx.Process(target=modify_and_show, args=(method,))
354
            p2.start()
355
            p2.join()
356
            
357
            # Show data again to see if modification persisted
358
            p3 = ctx.Process(target=show_global_data, args=(method,))
359
            p3.start()
360
            p3.join()
361
            
362
        except ValueError:
363
            print(f"Method {method} not available")
364
```
365

366
### Custom Executable Configuration
367

368
```python
369
import multiprocess as mp
370
import sys
371
import subprocess
372

373
def worker_info():
374
    """Show information about the Python executable"""
375
    print(f"Executable: {sys.executable}")
376
    print(f"Version: {sys.version}")
377
    print(f"Platform: {sys.platform}")
378

379
if __name__ == '__main__':
380
    print("Default executable:")
381
    p1 = mp.Process(target=worker_info)
382
    p1.start()
383
    p1.join()
384
    
385
    # Set custom executable (if available)
386
    # This is just an example - in practice you'd use a different Python
387
    try:
388
        # Try to find another Python executable
389
        result = subprocess.run(['which', 'python3'], 
390
                               capture_output=True, text=True)
391
        if result.returncode == 0:
392
            custom_python = result.stdout.strip()
393
            if custom_python != sys.executable:
394
                print(f"\nSetting custom executable: {custom_python}")
395
                mp.set_executable(custom_python)
396
                
397
                p2 = mp.Process(target=worker_info)
398
                p2.start()
399
                p2.join()
400
            else:
401
                print("No different Python executable found")
402
        else:
403
            print("Could not find alternative Python executable")
404
    except Exception as e:
405
        print(f"Error setting custom executable: {e}")
406
```
407

408
### Forkserver with Preloading
409

410
```python
411
import multiprocess as mp
412
import time
413

414
# Heavy module to preload
415
import numpy as np
416
import json
417

418
def worker_with_numpy(data):
419
    """Worker that uses numpy - should be faster with preloading"""
420
    start_time = time.time()
421
    
422
    # Use numpy (should be already imported in forkserver)
423
    arr = np.array(data)
424
    result = np.sum(arr ** 2)
425
    
426
    end_time = time.time()
427
    print(f"Worker processed {len(data)} items in {end_time - start_time:.3f}s")
428
    return result
429

430
if __name__ == '__main__':
431
    # Only works on systems that support forkserver
432
    if 'forkserver' in mp.get_all_start_methods():
433
        # Set forkserver method
434
        mp.set_start_method('forkserver')
435
        
436
        # Preload heavy modules in forkserver
437
        mp.set_forkserver_preload(['numpy', 'json'])
438
        
439
        print("Using forkserver with preloaded modules")
440
        
441
        # Create some work
442
        data_sets = [[i] * 1000 for i in range(1, 6)]
443
        
444
        # Process with preloaded modules
445
        start_time = time.time()
446
        with mp.Pool(2) as pool:
447
            results = pool.map(worker_with_numpy, data_sets)
448
        end_time = time.time()
449
        
450
        print(f"Results: {results}")
451
        print(f"Total time: {end_time - start_time:.3f}s")
452
    else:
453
        print("Forkserver method not available on this platform")
454
```
455

456
### Advanced Configuration Example
457

458
```python
459
import multiprocess as mp
460
import os
461
import sys
462

463
class ConfigurableWorker:
464
    def __init__(self, config):
465
        self.config = config
466
    
467
    def work(self, data):
468
        pid = os.getpid()
469
        method = mp.get_start_method()
470
        
471
        result = {
472
            'pid': pid,
473
            'method': method,
474
            'data': data,
475
            'config': self.config,
476
            'processed': data * 2
477
        }
478
        
479
        return result
480

481
def create_configured_pool(method='spawn', processes=None, preload=None):
482
    """Create a pool with specific configuration"""
483
    
484
    # Set start method
485
    mp.set_start_method(method, force=True)
486
    
487
    # Configure forkserver preloading if applicable
488
    if method == 'forkserver' and preload:
489
        mp.set_forkserver_preload(preload)
490
    
491
    # Create context
492
    ctx = mp.get_context(method)
493
    
494
    # Configuration for workers
495
    config = {
496
        'method': method,
497
        'preload': preload,
498
        'python_executable': sys.executable
499
    }
500
    
501
    # Create pool with initializer
502
    def init_worker():
503
        global worker_instance
504
        worker_instance = ConfigurableWorker(config)
505
    
506
    def worker_task(data):
507
        return worker_instance.work(data)
508
    
509
    pool = ctx.Pool(processes=processes, 
510
                   initializer=init_worker)
511
    
512
    return pool, worker_task
513

514
if __name__ == '__main__':
515
    # Test different configurations
516
    configs = [
517
        {'method': 'spawn', 'processes': 2, 'preload': None},
518
    ]
519
    
520
    # Add forkserver if available
521
    if 'forkserver' in mp.get_all_start_methods():
522
        configs.append({
523
            'method': 'forkserver', 
524
            'processes': 2, 
525
            'preload': ['json', 'os']
526
        })
527
    
528
    # Add fork if available
529
    if 'fork' in mp.get_all_start_methods():
530
        configs.append({
531
            'method': 'fork', 
532
            'processes': 2, 
533
            'preload': None
534
        })
535
    
536
    data = [1, 2, 3, 4, 5]
537
    
538
    for config in configs:
539
        print(f"\nTesting configuration: {config}")
540
        
541
        try:
542
            with create_configured_pool(**config) as (pool, task_func):
543
                results = pool.map(task_func, data)
544
                
545
                print("Results:")
546
                for result in results:
547
                    print(f"  PID {result['pid']} ({result['method']}): "
548
                          f"{result['data']} -> {result['processed']}")
549
                          
550
        except Exception as e:
551
            print(f"Configuration failed: {e}")
552
```
553

554
### Context Best Practices
555

556
```python
557
import multiprocess as mp
558
import logging
559

560
# Configure logging
561
logging.basicConfig(level=logging.INFO)
562
logger = logging.getLogger(__name__)
563

564
class ProcessManager:
565
    def __init__(self, method=None, processes=None):
566
        self.method = method or self._get_best_method()
567
        self.processes = processes or mp.cpu_count()
568
        self.context = mp.get_context(self.method)
569
        
570
        logger.info(f"Initialized ProcessManager with method '{self.method}', "
571
                   f"{self.processes} processes")
572
    
573
    def _get_best_method(self):
574
        """Choose the best start method for the platform"""
575
        available = mp.get_all_start_methods()
576
        
577
        # Platform-specific preferences
578
        if sys.platform == 'win32':
579
            return 'spawn'  # Only option on Windows
580
        elif sys.platform == 'darwin':
581
            return 'spawn'  # Recommended on macOS
582
        else:
583
            # On Linux, prefer forkserver if available, then fork
584
            if 'forkserver' in available:
585
                return 'forkserver'
586
            elif 'fork' in available:
587
                return 'fork'
588
            else:
589
                return 'spawn'
590
    
591
    def create_pool(self, **kwargs):
592
        """Create a pool using the configured context"""
593
        return self.context.Pool(processes=self.processes, **kwargs)
594
    
595
    def create_process(self, **kwargs):
596
        """Create a process using the configured context"""
597
        return self.context.Process(**kwargs)
598
    
599
    def get_queue(self, **kwargs):
600
        """Create a queue using the configured context"""
601
        return self.context.Queue(**kwargs)
602

603
def example_task(x):
604
    return x ** 2
605

606
if __name__ == '__main__':
607
    import sys
608
    
609
    # Create process manager
610
    pm = ProcessManager()
611
    
612
    # Use the manager
613
    with pm.create_pool() as pool:
614
        data = list(range(10))
615
        results = pool.map(example_task, data)
616
        logger.info(f"Results: {results}")
617
    
618
    # Create individual processes
619
    q = pm.get_queue()
620
    
621
    def worker():
622
        q.put(f"Hello from {os.getpid()}")
623
    
624
    processes = []
625
    for i in range(3):
626
        p = pm.create_process(target=worker)
627
        p.start()
628
        processes.append(p)
629
    
630
    # Collect results
631
    for _ in range(3):
632
        message = q.get()
633
        logger.info(f"Received: {message}")
634
    
635
    # Wait for processes
636
    for p in processes:
637
        p.join()
638
    
639
    logger.info("All processes completed")
640
```