0
# Distribution Scheduling
1

2
Multiple scheduling algorithms for distributing tests across workers, each optimized for different test suite characteristics and performance requirements.
3

4
## Capabilities
5

6
### Scheduling Protocol
7

8
All schedulers implement the `Scheduling` protocol, providing a consistent interface for test distribution.
9

10
```python { .api }
11
class Scheduling(Protocol):
12
    """Protocol for all test distribution schedulers."""
13
    
14
    @property
15
    def nodes(self) -> list[WorkerController]:
16
        """List of active worker nodes."""
17
    
18
    @property
19
    def collection_is_completed(self) -> bool:
20
        """True if test collection is complete on all nodes."""
21
    
22
    @property
23
    def tests_finished(self) -> bool:
24
        """True if all tests have been executed."""
25
    
26
    @property
27
    def has_pending(self) -> bool:
28
        """True if there are pending tests to be scheduled."""
29
    
30
    def add_node(self, node: WorkerController) -> None:
31
        """Add a worker node to the scheduler."""
32
    
33
    def add_node_collection(
34
        self,
35
        node: WorkerController,
36
        collection: Sequence[str],
37
    ) -> None:
38
        """Add collected test items from a worker node."""
39
    
40
    def mark_test_complete(
41
        self,
42
        node: WorkerController,
43
        item_index: int,
44
        duration: float = 0,
45
    ) -> None:
46
        """Mark a test as completed by a worker."""
47
    
48
    def mark_test_pending(self, item: str) -> None:
49
        """Mark a test as pending (needs to be rescheduled)."""
50
    
51
    def remove_pending_tests_from_node(
52
        self,
53
        node: WorkerController,
54
        indices: Sequence[int],
55
    ) -> None:
56
        """Remove pending tests from a specific worker node."""
57
    
58
    def remove_node(self, node: WorkerController) -> str | None:
59
        """Remove a worker node from the scheduler."""
60
    
61
    def schedule(self) -> None:
62
        """Execute the scheduling algorithm to distribute tests."""
63
```
64

65
### Load Balancing Scheduler
66

67
Distributes tests dynamically to available workers for optimal load balancing.
68

69
```python { .api }
70
class LoadScheduling:
71
    """
72
    Load balance by sending any pending test to any available worker.
73
    
74
    Tests are distributed one at a time to the next available worker,
75
    providing good load balancing for test suites with varying execution times.
76
    """
77
```
78

79
**Characteristics:**
80
- Dynamic distribution to available workers
81
- Good for mixed execution times
82
- Minimal worker idle time
83
- Default scheduling mode with `-n`
84

85
### Each Scheduler
86

87
Sends each test to all available workers (typically for testing across environments).
88

89
```python { .api }
90
class EachScheduling:
91
    """
92
    Send each test to all available workers.
93
    
94
    Every test is executed on every worker, useful for testing
95
    across different environments or configurations.
96
    """
97
```
98

99
**Characteristics:**
100
- Every test runs on every worker
101
- Useful for cross-platform/environment testing
102
- Significantly increases total execution time
103
- Good for validation across configurations
104

105
### Load Scope Scheduler
106

107
Groups tests by scope (class, module, etc.) for better fixture reuse.
108

109
```python { .api }
110
class LoadScopeScheduling:
111
    """
112
    Load balance by sending pending groups of tests in the same scope
113
    to any available worker.
114
    
115
    Groups tests by scope (typically class or module) to improve
116
    fixture reuse and reduce setup/teardown overhead.
117
    """
118
```
119

120
**Characteristics:**
121
- Groups tests by pytest scope
122
- Improves fixture reuse efficiency
123
- Better for test suites with expensive fixtures
124
- Supports `--loadscope-reorder` for optimization
125

126
### Load File Scheduler
127

128
Groups tests by file for locality and fixture sharing.
129

130
```python { .api }
131
class LoadFileScheduling:
132
    """
133
    Load balance by sending tests grouped by file to any available worker.
134
    
135
    All tests from the same file are sent to the same worker,
136
    optimizing for file-level fixtures and reducing I/O overhead.
137
    """
138
```
139

140
**Characteristics:**
141
- Groups tests by source file
142
- Optimizes file-level fixture usage
143
- Reduces context switching
144
- Good for file-heavy test suites
145

146
### Load Group Scheduler
147

148
Like load balancing but respects `xdist_group` markers for test grouping.
149

150
```python { .api }
151
class LoadGroupScheduling:
152
    """
153
    Like LoadScheduling, but sends tests marked with 'xdist_group' 
154
    to the same worker.
155
    
156
    Tests marked with the same xdist_group name will always run
157
    on the same worker, useful for tests that share state or resources.
158
    """
159
```
160

161
**Characteristics:**
162
- Respects `@pytest.mark.xdist_group` markers
163
- Tests with same group name run on same worker
164
- Allows controlled test grouping
165
- Good for tests with shared state requirements
166

167
### Work Stealing Scheduler
168

169
Splits tests initially, then rebalances when workers become idle.
170

171
```python { .api }
172
class WorkStealingScheduling:
173
    """
174
    Split the test suite evenly between available workers initially,
175
    then rebalance when any worker runs out of tests.
176
    
177
    Provides good initial distribution with dynamic rebalancing
178
    for optimal resource utilization.
179
    """
180
```
181

182
**Characteristics:**
183
- Initial even distribution
184
- Dynamic rebalancing when workers finish
185
- Good for large test suites
186
- Minimizes communication overhead
187

188
### Scheduler Factory
189

190
Hook for creating custom schedulers.
191

192
```python { .api }
193
def pytest_xdist_make_scheduler(
194
    config: pytest.Config, log: Producer
195
) -> Scheduling | None:
196
    """
197
    Hook for creating custom scheduler implementations.
198
    
199
    Args:
200
        config: pytest configuration object
201
        log: logging producer for scheduler messages
202
        
203
    Returns:
204
        Custom scheduler instance or None to use default
205
    """
206
```
207

208
## Usage Examples
209

210
### Command Line Usage
211

212
```bash
213
# Load balancing (default with -n)
214
pytest -n 4
215

216
# Explicit load balancing
217
pytest -n 4 --dist load
218

219
# Each test on all workers
220
pytest -n 4 --dist each
221

222
# Group by scope
223
pytest -n 4 --dist loadscope
224

225
# Group by file
226
pytest -n 4 --dist loadfile
227

228
# Respect xdist_group markers
229
pytest -n 4 --dist loadgroup
230

231
# Work stealing
232
pytest -n 4 --dist worksteal
233

234
# Control loadscope reordering
235
pytest -n 4 --dist loadscope --loadscope-reorder
236
pytest -n 4 --dist loadscope --no-loadscope-reorder
237
```
238

239
### Using xdist_group Marker
240

241
```python
242
import pytest
243

244
# Tests with same group run on same worker
245
@pytest.mark.xdist_group(name="database")
246
def test_db_setup():
247
    # Database initialization
248
    pass
249

250
@pytest.mark.xdist_group(name="database")  
251
def test_db_query():
252
    # Can rely on setup from test_db_setup
253
    pass
254

255
@pytest.mark.xdist_group(name="filesystem")
256
def test_file_operations():
257
    # Different group, may run on different worker
258
    pass
259
```
260

261
### Custom Scheduler Implementation
262

263
```python
264
# In conftest.py
265
from xdist.scheduler.protocol import Scheduling
266
from xdist.workermanage import WorkerController
267

268
class CustomScheduler:
269
    """Example custom scheduler implementation."""
270
    
271
    def __init__(self, config, log):
272
        self.config = config
273
        self.log = log
274
        self._nodes = []
275
        self._pending = []
276
        self._collection_complete = False
277
    
278
    @property
279
    def nodes(self) -> list[WorkerController]:
280
        return self._nodes
281
    
282
    @property 
283
    def collection_is_completed(self) -> bool:
284
        return self._collection_complete
285
    
286
    def schedule(self) -> None:
287
        # Custom scheduling logic
288
        if self.has_pending and self.nodes:
289
            # Implement custom distribution algorithm
290
            pass
291

292
def pytest_xdist_make_scheduler(config, log):
293
    if config.getoption("--custom-scheduler"):
294
        return CustomScheduler(config, log)
295
    return None  # Use default scheduler
296
```
297

298
### Scheduler Selection Logic
299

300
```python
301
# How pytest-xdist selects schedulers based on --dist option
302
SCHEDULER_MAP = {
303
    'each': EachScheduling,
304
    'load': LoadScheduling, 
305
    'loadscope': LoadScopeScheduling,
306
    'loadfile': LoadFileScheduling,
307
    'loadgroup': LoadGroupScheduling,
308
    'worksteal': WorkStealingScheduling,
309
}
310

311
def create_scheduler(config, log):
312
    dist_mode = config.getoption("dist")
313
    
314
    # First try custom scheduler hook
315
    custom = config.hook.pytest_xdist_make_scheduler(config=config, log=log)
316
    if custom:
317
        return custom
318
    
319
    # Use built-in scheduler
320
    scheduler_class = SCHEDULER_MAP.get(dist_mode, LoadScheduling)
321
    return scheduler_class(config, log)
322
```
323

324
### Performance Considerations
325

326
```python
327
# Choosing the right scheduler for your test suite
328

329
# For test suites with:
330
# - Varying execution times -> LoadScheduling (default)
331
# - Expensive class/module fixtures -> LoadScopeScheduling  
332
# - File-based fixtures/setup -> LoadFileScheduling
333
# - Tests that share state -> LoadGroupScheduling with markers
334
# - Large test suites -> WorkStealingScheduling
335
# - Cross-environment testing -> EachScheduling
336

337
def pytest_configure(config):
338
    """Auto-select scheduler based on test suite characteristics."""
339
    if not config.getoption("dist") or config.getoption("dist") == "no":
340
        return
341
        
342
    # Analyze test suite and suggest optimal scheduler
343
    test_count = len(config.getoption("file_or_dir") or [])
344
    
345
    if test_count > 1000:
346
        # Large test suite - work stealing might be better
347
        config.option.dist = "worksteal"
348
    elif has_expensive_fixtures():
349
        # Expensive fixtures - use scope grouping
350
        config.option.dist = "loadscope"
351
```