0
# Configuration
1
2
Command-line options and marker parameters for controlling benchmark behavior. pytest-codspeed provides extensive configuration through CLI flags, environment variables, and marker options.
3
4
## Capabilities
5
6
### Command Line Options
7
8
Options for controlling pytest-codspeed behavior from the command line.
9
10
```python { .api }
11
# Enable CodSpeed benchmarking
12
--codspeed
13
14
# Set measurement mode
15
--codspeed-mode {instrumentation,walltime}
16
17
# Walltime mode timing options
18
--codspeed-warmup-time FLOAT # Warmup time in seconds
19
--codspeed-max-time FLOAT # Maximum benchmark time in seconds
20
--codspeed-max-rounds INT # Maximum number of rounds
21
```
22
23
#### Usage Examples
24
25
```bash
26
# Basic benchmarking with walltime mode
27
pytest tests/ --codspeed
28
29
# Use instrumentation mode for more precise measurement
30
pytest tests/ --codspeed --codspeed-mode instrumentation
31
32
# Customize timing for walltime mode
33
pytest tests/ --codspeed --codspeed-warmup-time 2.0 --codspeed-max-time 10.0 --codspeed-max-rounds 50
34
```
35
36
### Environment Variables
37
38
Environment variables that control pytest-codspeed behavior, primarily used in CI/CD environments.
39
40
```python { .api }
41
# Core environment variables
42
CODSPEED_ENV # Enables CodSpeed when set (any value)
43
CODSPEED_RUNNER_MODE # Sets default mode: "walltime" or other
44
CODSPEED_PROFILE_FOLDER # Directory for results output
45
46
# Extension build control
47
PYTEST_CODSPEED_FORCE_EXTENSION_BUILD # Force native extension compilation
48
PYTEST_CODSPEED_SKIP_EXTENSION_BUILD # Skip native extension compilation
49
```
50
51
#### Usage Examples
52
53
```bash
54
# Enable CodSpeed in CI environment
55
export CODSPEED_ENV=1
56
pytest tests/
57
58
# Force walltime mode in CI
59
export CODSPEED_ENV=1
60
export CODSPEED_RUNNER_MODE=walltime
61
pytest tests/
62
63
# Custom results directory
64
export CODSPEED_ENV=1
65
export CODSPEED_PROFILE_FOLDER=/custom/results/path
66
pytest tests/
67
```
68
69
### Marker Options
70
71
Parameters that can be passed to benchmark markers to customize individual test behavior.
72
73
```python { .api }
74
@pytest.mark.benchmark(
75
group: str = None, # Group name for benchmark organization
76
min_time: float = None, # Minimum time per round (seconds, walltime only)
77
max_time: float = None, # Maximum total benchmark time (seconds, walltime only)
78
max_rounds: int = None # Maximum number of rounds (walltime only)
79
)
80
81
@pytest.mark.codspeed_benchmark(
82
group: str = None, # Same options as benchmark marker
83
min_time: float = None,
84
max_time: float = None,
85
max_rounds: int = None
86
)
87
```
88
89
#### Usage Examples
90
91
```python
92
import pytest
93
94
# Group related benchmarks
95
@pytest.mark.benchmark(group="string_operations")
96
def test_string_join():
97
result = "".join(str(i) for i in range(1000))
98
assert len(result) > 0
99
100
@pytest.mark.benchmark(group="string_operations")
101
def test_string_format():
102
result = " ".join(f"item_{i}" for i in range(1000))
103
assert len(result) > 0
104
105
# Control timing for slow operations
106
@pytest.mark.benchmark(max_time=10.0, max_rounds=20)
107
def test_expensive_computation():
108
result = sum(i ** 2 for i in range(10000))
109
assert result > 0
110
111
# Ensure minimum measurement time for fast operations
112
@pytest.mark.benchmark(min_time=0.1)
113
def test_fast_operation():
114
result = 1 + 1
115
assert result == 2
116
```
117
118
### Measurement Modes
119
120
pytest-codspeed supports two measurement approaches with different accuracy/overhead tradeoffs.
121
122
```python { .api }
123
# Measurement modes
124
MeasurementMode.WallTime # Wall-clock time measurement (default locally)
125
MeasurementMode.Instrumentation # Instruction-based measurement (default in CI)
126
```
127
128
#### Walltime Mode
129
- Measures actual execution time using high-resolution timers
130
- Includes statistical analysis with outlier detection
131
- Configurable warmup, timing limits, and round counts
132
- Provides detailed performance statistics and terminal reporting
133
- Default mode for local development
134
135
#### Instrumentation Mode
136
- Uses Valgrind-based instruction counting for deterministic results
137
- Immune to system load and timing variations
138
- Single measurement per benchmark (ignores rounds/iterations settings)
139
- Requires CodSpeed environment for native extension
140
- Default mode in CI environments
141
142
## Configuration Precedence
143
144
Configuration values are resolved in this order (highest to lowest priority):
145
146
1. Marker options (per-test customization)
147
2. Command line options (pytest invocation)
148
3. Environment variables (CI/CD configuration)
149
4. Default values (built-in sensible defaults)
150
151
## Advanced Configuration
152
153
### Native Extension Control
154
155
```python
156
# Force native extension build (raises error if impossible)
157
export PYTEST_CODSPEED_FORCE_EXTENSION_BUILD=1
158
159
# Skip native extension (falls back to walltime mode)
160
export PYTEST_CODSPEED_SKIP_EXTENSION_BUILD=1
161
```
162
163
### Results Output
164
165
```python
166
# Default local results location
167
.codspeed/results_{timestamp}.json
168
169
# CI results location (when CODSPEED_PROFILE_FOLDER set)
170
{CODSPEED_PROFILE_FOLDER}/results/{pid}.json
171
```
172
173
## Compatibility Settings
174
175
pytest-codspeed automatically handles compatibility with other benchmark plugins:
176
177
- Disables `pytest-benchmark` when active
178
- Disables `pytest-speed` when active
179
- Preserves original fixtures as `__benchmark` if needed
180
- Maintains compatibility with existing test suites