0
# pytest-randomly
1
2
A pytest plugin that introduces controlled randomness into test execution to improve test quality and discover hidden dependencies. It randomly shuffles test execution order and resets random seeds to ensure reproducible test runs.
3
4
## Package Information
5
6
- **Package Name**: pytest-randomly
7
- **Package Type**: pypi
8
- **Language**: Python
9
- **Installation**: `pip install pytest-randomly`
10
11
## Core Imports
12
13
```python
14
import pytest_randomly
15
```
16
17
The plugin is automatically discovered by pytest via entry points - no explicit imports needed.
18
19
## Basic Usage
20
21
```python
22
# Install the plugin
23
# pip install pytest-randomly
24
25
# Run tests with automatic randomization (default behavior)
26
pytest
27
28
# Run tests with specific seed for reproducibility
29
pytest --randomly-seed=1234
30
31
# Reuse seed from previous run
32
pytest --randomly-seed=last
33
34
# Disable test order shuffling but keep seed resetting
35
pytest --randomly-dont-reorganize
36
37
# Disable seed resetting but keep order shuffling
38
pytest --randomly-dont-reset-seed
39
40
# Completely disable the plugin
41
pytest -p no:randomly
42
```
43
44
## Architecture
45
46
The plugin provides controlled randomness through:
47
48
- **Test Order Randomization**: Shuffles tests at module, class, and function levels using deterministic seeding
49
- **Random State Management**: Resets global random seeds before each test for reproducibility
50
- **Third-party Integration**: Automatically resets random states for factory-boy, faker, model-bakery, and numpy
51
- **pytest-xdist Support**: Distributes consistent seeds across parallel worker processes
52
- **Extensibility**: Entry point system for third-party random seeder registration
53
54
## Capabilities
55
56
### Plugin Configuration
57
58
Adds command-line options and configures the plugin when pytest starts.
59
60
```python { .api }
61
def pytest_addoption(parser: Parser) -> None:
62
"""
63
Adds command-line options for controlling randomness behavior.
64
65
Options added:
66
- --randomly-seed: Set seed value (int, 'default', or 'last')
67
- --randomly-dont-reset-seed: Disable random seed reset per test
68
- --randomly-dont-reorganize: Disable test order shuffling
69
"""
70
71
def pytest_configure(config: Config) -> None:
72
"""
73
Configures the plugin when pytest starts.
74
75
Handles seed generation, xdist integration, and caching.
76
"""
77
```
78
79
### Test Execution Hooks
80
81
Controls random state during different phases of test execution.
82
83
```python { .api }
84
def pytest_runtest_setup(item: Item) -> None:
85
"""
86
Resets random state before each test setup phase.
87
88
Args:
89
item: The test item being set up
90
"""
91
92
def pytest_runtest_call(item: Item) -> None:
93
"""
94
Resets random state before each test execution phase.
95
96
Args:
97
item: The test item being executed
98
"""
99
100
def pytest_runtest_teardown(item: Item) -> None:
101
"""
102
Resets random state after each test teardown phase.
103
104
Args:
105
item: The test item being torn down
106
"""
107
```
108
109
### Test Collection and Ordering
110
111
Randomly reorders collected test items for better test isolation detection.
112
113
```python { .api }
114
@hookimpl(tryfirst=True)
115
def pytest_collection_modifyitems(config: Config, items: list[Item]) -> None:
116
"""
117
Randomly reorders collected test items by module, class, then function.
118
119
Uses deterministic shuffling based on seed value to ensure
120
reproducible test order when using the same seed.
121
122
Args:
123
config: pytest configuration object
124
items: List of collected test items to reorder
125
"""
126
```
127
128
### Reporting
129
130
Provides information about the randomization seed being used.
131
132
```python { .api }
133
def pytest_report_header(config: Config) -> str:
134
"""
135
Returns header line showing the seed being used.
136
137
Args:
138
config: pytest configuration object
139
140
Returns:
141
Header string with seed information (e.g., "Using --randomly-seed=1234")
142
"""
143
```
144
145
### Utility Functions
146
147
Helper functions for seed validation and argument parsing.
148
149
```python { .api }
150
def seed_type(string: str) -> str | int:
151
"""
152
Argument parser type converter for --randomly-seed option.
153
154
Args:
155
string: String value to convert
156
157
Returns:
158
Integer seed value, or 'default'/'last' string values
159
160
Raises:
161
argparse.ArgumentTypeError: For invalid seed values
162
"""
163
164
def reduce_list_of_lists(lists: list[list[T]]) -> list[T]:
165
"""
166
Flattens a list of lists into a single list.
167
168
Args:
169
lists: List of lists to flatten
170
171
Returns:
172
Single flattened list containing all elements
173
"""
174
```
175
176
### pytest-xdist Integration
177
178
Handles parallel test execution with consistent seeding across workers.
179
180
```python { .api }
181
class XdistHooks:
182
"""
183
Hooks container for pytest-xdist integration.
184
185
Automatically registered when xdist plugin is detected.
186
"""
187
188
def pytest_configure_node(self, node: Item) -> None:
189
"""
190
Configures worker nodes in xdist with consistent seed values.
191
192
Args:
193
node: Worker node to configure
194
"""
195
```
196
197
### faker Integration
198
199
Provides seed configuration for the faker pytest plugin when faker is installed.
200
201
**Note**: This fixture is only defined when the `faker` package is available at import time.
202
203
```python { .api }
204
# Only available when faker is installed
205
if have_faker:
206
@fixture(autouse=True)
207
def faker_seed(pytestconfig: Config) -> int:
208
"""
209
Provides faker seed configuration to faker pytest plugin.
210
211
Conditionally defined fixture that only exists when faker package
212
is available at plugin import time.
213
214
Args:
215
pytestconfig: pytest configuration object
216
217
Returns:
218
Current randomly seed value for faker integration
219
"""
220
```
221
222
## Types
223
224
```python { .api }
225
# Module-level variables
226
default_seed: int # Default random seed generated at import time
227
random_states: dict[int, tuple[Any, ...]] # Cache for random states by seed
228
np_random_states: dict[int, Any] # Cache for numpy random states by seed
229
entrypoint_reseeds: list[Callable[[int], None]] | None # Cached entry point reseeder functions
230
231
# Third-party library availability flags
232
have_factory_boy: bool # True if factory-boy is available
233
have_faker: bool # True if faker is available
234
have_model_bakery: bool # True if model-bakery is available
235
have_numpy: bool # True if numpy is available
236
237
# Optional module imports
238
xdist: ModuleType | None # pytest-xdist module if available, None otherwise
239
240
# Type variable for generic list flattening
241
T = TypeVar("T")
242
```
243
244
## Third-party Integration
245
246
The plugin automatically detects and integrates with popular Python testing libraries when they are available:
247
248
- **factory-boy** (if `have_factory_boy` is True): Resets `factory.random` state for consistent factory generation
249
- **faker** (if `have_faker` is True): Resets `faker.generator.random` state for reproducible fake data
250
- **model-bakery** (if `have_model_bakery` is True): Resets `model_bakery.random_gen.baker_random` state
251
- **numpy** (if `have_numpy` is True): Resets `numpy.random` legacy global state for consistent array generation
252
- **pytest-xdist** (if detected): Distributes seed values to parallel worker processes
253
254
## Extension API
255
256
Third-party packages can register custom random seeders via the `pytest_randomly.random_seeder` entry point group:
257
258
```python
259
# In your setup.py or pyproject.toml
260
entry_points = {
261
"pytest_randomly.random_seeder": [
262
"your_lib = your_package:reseed_function"
263
]
264
}
265
266
# Your reseed function signature
267
def reseed_function(seed: int) -> None:
268
"""Reset your random generator with the provided seed."""
269
your_random_generator.seed(seed)
270
```
271
272
## Command Line Options
273
274
- `--randomly-seed=<int|'default'|'last'>`: Control the random seed value
275
- `--randomly-dont-reset-seed`: Disable per-test seed reset while keeping order shuffling
276
- `--randomly-dont-reorganize`: Disable test order shuffling while keeping seed reset
277
- `-p no:randomly`: Completely disable the plugin