0
# WorkerPool Management
1

2
Core WorkerPool class providing the main interface for creating, configuring, and managing parallel worker processes or threads. The WorkerPool acts as a replacement for multiprocessing.Pool with enhanced features and better performance.
3

4
## Capabilities
5

6
### WorkerPool Class
7

8
The main class for managing a pool of worker processes or threads.
9

10
```python { .api }
11
class WorkerPool:
12
    def __init__(self, n_jobs: Optional[int] = None, daemon: bool = True, 
13
                 cpu_ids: CPUList = None, shared_objects: Any = None, 
14
                 pass_worker_id: bool = False, use_worker_state: bool = False,
15
                 start_method: str = DEFAULT_START_METHOD, keep_alive: bool = False,
16
                 use_dill: bool = False, enable_insights: bool = False, 
17
                 order_tasks: bool = False) -> None
18
```
19

20
**Parameters:**
21
- `n_jobs` (Optional[int]): Number of workers to spawn. If None, uses `cpu_count()`
22
- `daemon` (bool): Whether to start child processes as daemon processes. Default: True
23
- `cpu_ids` (CPUList): List of CPU IDs for pinning processes to specific CPUs. None disables CPU pinning
24
- `shared_objects` (Any): Objects passed as shared objects to workers (copy-on-write with fork)
25
- `pass_worker_id` (bool): Whether to pass worker ID as first argument to functions. Default: False
26
- `use_worker_state` (bool): Whether workers maintain state between tasks. Default: False
27
- `start_method` (str): Process start method ('fork', 'spawn', 'forkserver', 'threading')
28
- `keep_alive` (bool): Keep workers alive after map calls for reuse. Default: False
29
- `use_dill` (bool): Use dill for serialization instead of pickle. Default: False
30
- `enable_insights` (bool): Enable worker performance insights. Default: False
31
- `order_tasks` (bool): Provide tasks to workers in order (worker 0 gets chunk 0, etc.). Default: False
32

33
### Context Manager Support
34

35
WorkerPool supports context manager protocol for automatic resource cleanup.
36

37
```python { .api }
38
def __enter__(self) -> 'WorkerPool'
39
def __exit__(self, *_: Any) -> None
40
```
41

42
### Worker Lifecycle Management
43

44
Functions for controlling worker lifecycle and cleanup.
45

46
```python { .api }
47
def stop_and_join(self, keep_alive: bool = False) -> None
48
def terminate(self) -> None
49
```
50

51
**stop_and_join**: Gracefully stop all workers and wait for them to finish current tasks.
52
- `keep_alive` (bool): If True, keep workers alive for future reuse
53

54
**terminate**: Immediately terminate all workers without waiting for task completion.
55

56
## Usage Examples
57

58
### Basic WorkerPool Creation
59

60
```python
61
from mpire import WorkerPool
62

63
# Simple worker pool with default settings
64
with WorkerPool() as pool:
65
    results = pool.map(lambda x: x * 2, range(10))
66

67
# Custom number of workers
68
with WorkerPool(n_jobs=8) as pool:
69
    results = pool.map(some_function, data)
70
```
71

72
### Advanced Configuration
73

74
```python
75
from mpire import WorkerPool
76

77
# Worker pool with custom configuration
78
with WorkerPool(
79
    n_jobs=4,
80
    daemon=False,
81
    cpu_ids=[0, 1, 2, 3],  # Pin to specific CPUs
82
    use_worker_state=True,
83
    enable_insights=True,
84
    start_method='spawn'
85
) as pool:
86
    results = pool.map(
87
        process_function, 
88
        data, 
89
        worker_init=init_function,
90
        worker_exit=cleanup_function
91
    )
92
```
93

94
### Worker Pool Reuse
95

96
```python
97
# Keep workers alive for multiple map operations
98
pool = WorkerPool(n_jobs=4, keep_alive=True)
99

100
# First batch
101
results1 = pool.map(function1, data1)
102

103
# Second batch reuses same workers
104
results2 = pool.map(function2, data2)
105

106
# Cleanup when done
107
pool.stop_and_join()
108
```
109

110
### Manual Worker Management
111

112
```python
113
# Manual lifecycle management
114
pool = WorkerPool(n_jobs=4)
115

116
try:
117
    results = pool.map(some_function, data)
118
except Exception:
119
    # Force termination on error
120
    pool.terminate()
121
else:
122
    # Graceful shutdown
123
    pool.stop_and_join()
124
```