0
# Command Line Interface
1

2
## Overview
3

4
pytest-benchmark provides standalone command-line tools for managing and analyzing saved benchmark results outside of pytest. The CLI interface allows you to list, compare, and export benchmark data with flexible filtering and formatting options.
5

6
## Core APIs
7

8
### pytest-benchmark Command
9

10
```bash { .api }
11
pytest-benchmark [-h] [--storage STORAGE] [--netrc] [--verbose] [--quiet] 
12
                 [--import-mode {prepend,append,importlib}] 
13
                 {list,compare,help} ...
14
```
15

16
Entry point for all CLI operations. Available as both `pytest-benchmark` and `py.test-benchmark`.
17

18
**Global Options:**
19
- `--storage`, `-s`: Storage URI (default: `file://./.benchmarks`)
20
- `--netrc`: Load Elasticsearch credentials from netrc file (default: `~/.netrc`)
21
- `--verbose`, `-v`: Enable verbose diagnostic and progress information
22
- `--quiet`, `-q`: Disable reporting (verbose mode takes precedence)
23
- `--import-mode`: How to load hooks from conftest (choices: `prepend`, `append`, `importlib`)
24

25
### Subcommands
26

27
#### list Command
28

29
```bash { .api }
30
pytest-benchmark list [-h]
31
```
32

33
Lists all saved benchmark runs in the configured storage location.
34

35
#### compare Command
36

37
```bash { .api }
38
pytest-benchmark compare [-h] [--sort {min,max,mean,stddev,name,fullname}]
39
                        [--group-by {group,name,fullname,func,fullfunc,param}]
40
                        [--columns COLUMNS] [--name {short,normal,long,trial}]
41
                        [--histogram [FILENAME_PREFIX]] [--csv [FILENAME]]
42
                        [glob_or_file ...]
43
```
44

45
Compare and display results from saved benchmark runs.
46

47
**Display Options:**
48
- `--sort`: Column to sort by (default: `min`)
49
- `--group-by`: How to group tests (default: `group`)
50
- `--columns`: Comma-separated list of columns to display
51
- `--name`: Format for test names (default: `normal`)
52

53
**Output Options:**
54
- `--histogram [FILENAME_PREFIX]`: Generate SVG histogram plots
55
- `--csv [FILENAME]`: Export results to CSV format
56

57
**File Selection:**
58
- `glob_or_file`: Positional arguments specifying which files/runs to load
59

60
#### help Command
61

62
```bash { .api }
63
pytest-benchmark help [command]
64
```
65

66
Display help information for commands.
67

68
## Usage Examples
69

70
### Listing Saved Benchmarks
71

72
```bash
73
# List all saved benchmark runs
74
pytest-benchmark list
75

76
# List with verbose output
77
pytest-benchmark --verbose list
78

79
# List from custom storage location
80
pytest-benchmark --storage file:///custom/path list
81
```
82

83
### Comparing Benchmark Results
84

85
#### Basic Comparison
86

87
```bash
88
# Compare all saved benchmarks
89
pytest-benchmark compare
90

91
# Compare specific run by number
92
pytest-benchmark compare 0001
93

94
# Compare multiple specific runs
95
pytest-benchmark compare 0001 0002 0003
96
```
97

98
#### Pattern-Based Selection
99

100
```bash
101
# Compare all benchmarks from specific interpreter
102
pytest-benchmark compare 'Linux-CPython-3.5-64bit/*'
103

104
# Compare benchmarks matching pattern
105
pytest-benchmark compare '*test_performance*'
106

107
# Compare specific files by path
108
pytest-benchmark compare /path/to/0001_abc.json /path/to/0002_def.json
109
```
110

111
#### Output Customization
112

113
```bash
114
# Sort by different column
115
pytest-benchmark compare --sort mean 0001
116

117
# Group by function name instead of group
118
pytest-benchmark compare --group-by func 0001
119

120
# Show only specific columns
121
pytest-benchmark compare --columns min,max,mean 0001
122

123
# Use short test names
124
pytest-benchmark compare --name short 0001
125
```
126

127
#### Export and Visualization
128

129
```bash
130
# Generate histogram plots
131
pytest-benchmark compare --histogram mytest_ 0001
132

133
# Export to CSV
134
pytest-benchmark compare --csv results.csv 0001
135

136
# Both histogram and CSV
137
pytest-benchmark compare --histogram plots_ --csv data.csv 0001
138
```
139

140
### Storage Configuration
141

142
#### File Storage
143

144
```bash
145
# Use default location (./.benchmarks)
146
pytest-benchmark compare 0001
147

148
# Use custom file location
149
pytest-benchmark --storage file:///custom/benchmarks compare 0001
150

151
# Use relative path
152
pytest-benchmark --storage file://./results compare 0001
153
```
154

155
#### Elasticsearch Storage
156

157
```bash
158
# Connect to Elasticsearch
159
pytest-benchmark --storage http://localhost:9200 compare 0001
160

161
# Use authentication with netrc
162
pytest-benchmark --storage https://user:pass@es.example.com --netrc compare 0001
163
```
164

165
## File and Pattern Matching
166

167
### Glob Patterns
168

169
The CLI supports glob patterns for flexible file selection:
170

171
```bash
172
# All benchmarks from Linux CPython 3.8
173
pytest-benchmark compare 'Linux-CPython-3.8*/*'
174

175
# All benchmarks containing 'performance' in name
176
pytest-benchmark compare '*performance*'
177

178
# All JSON files in subdirectories
179
pytest-benchmark compare '*/\*.json'
180
```
181

182
**Note**: Use single quotes to prevent shell glob expansion.
183

184
### Run Number Selection
185

186
```bash
187
# First run from all interpreters
188
pytest-benchmark compare 0001
189

190
# Specific runs
191
pytest-benchmark compare 0001 0005 0010
192

193
# Range using shell expansion
194
pytest-benchmark compare 000{1..5}
195
```
196

197
### Direct File Paths
198

199
```bash
200
# Absolute paths
201
pytest-benchmark compare /full/path/to/benchmark.json
202

203
# Relative paths
204
pytest-benchmark compare ./results/test_001.json ../other/test_002.json
205

206
# Mixed selection
207
pytest-benchmark compare 0001 ./custom.json 'Linux*/*'
208
```
209

210
## Output Formats
211

212
### Table Display
213

214
Default tabular output with customizable columns:
215

216
```bash
217
# Default columns: min, max, mean, stddev, median, iqr, outliers, ops, rounds, iterations
218
pytest-benchmark compare 0001
219

220
# Custom columns
221
pytest-benchmark compare --columns min,mean,ops,rounds 0001
222

223
# Minimal display
224
pytest-benchmark compare --columns name,min,ops 0001
225
```
226

227
### Histogram Generation
228

229
Visual analysis with SVG plots:
230

231
```bash
232
# Generate with automatic filename
233
pytest-benchmark compare --histogram 0001
234

235
# Custom filename prefix
236
pytest-benchmark compare --histogram mytest_comparison_ 0001
237

238
# Creates files like: mytest_comparison_test_function_name.svg
239
```
240

241
### CSV Export
242

243
Data export for further analysis:
244

245
```bash
246
# Export with automatic filename
247
pytest-benchmark compare --csv 0001
248

249
# Custom filename
250
pytest-benchmark compare --csv benchmark_results.csv 0001
251

252
# Export includes all benchmark metadata and statistics
253
```
254

255
## Integration Examples
256

257
### Continuous Integration
258

259
```bash
260
#!/bin/bash
261
# Compare current run with baseline
262
pytest-benchmark compare --csv ci_results.csv baseline current
263

264
# Check for performance regressions
265
if [ $? -ne 0 ]; then
266
    echo "Performance regression detected!"
267
    exit 1
268
fi
269
```
270

271
### Performance Monitoring
272

273
```bash
274
# Weekly performance report
275
pytest-benchmark compare --histogram weekly_report_ --csv weekly_data.csv 'week_*'
276

277
# Generate trending analysis
278
for run in $(pytest-benchmark list | grep -o '[0-9]\{4\}'); do
279
    pytest-benchmark compare --csv "trend_$run.csv" "$run"
280
done
281
```
282

283
### Development Workflow
284

285
```bash
286
# Compare feature branch against main
287
pytest-benchmark compare main_baseline feature_test
288

289
# Quick performance check
290
pytest-benchmark compare --columns name,min,ops --name short latest
291

292
# Detailed analysis with visuals
293
pytest-benchmark compare --histogram analysis_ --csv detailed.csv latest
294
```
295

296
## Error Handling and Troubleshooting
297

298
### Common Issues
299

300
```bash
301
# Storage not found
302
pytest-benchmark --storage file:///missing/path list
303
# Error: Storage location does not exist
304

305
# Invalid glob pattern
306
pytest-benchmark compare 'unclosed_quote
307
# Error: Invalid glob pattern
308

309
# No matching files
310
pytest-benchmark compare nonexistent_pattern
311
# Warning: No benchmark files found matching pattern
312
```
313

314
### Verbose Mode
315

316
```bash
317
# Debug storage and file loading issues
318
pytest-benchmark --verbose compare problematic_pattern
319

320
# Shows:
321
# - Storage backend initialization
322
# - File discovery and loading
323
# - Data processing steps
324
# - Error details with stack traces
325
```
326

327
### Help and Documentation
328

329
```bash
330
# General help
331
pytest-benchmark --help
332

333
# Command-specific help
334
pytest-benchmark help compare
335
pytest-benchmark compare --help
336

337
# Show examples
338
pytest-benchmark help
339
```