CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/pypi-pytest-benchmark

A pytest fixture for benchmarking code that automatically calibrates test runs for accurate performance measurements.

—

Pending
Overview
Eval results
Files

cli-tools.mddocs/

Command Line Interface

Overview

pytest-benchmark provides standalone command-line tools for managing and analyzing saved benchmark results outside of pytest. The CLI interface allows you to list, compare, and export benchmark data with flexible filtering and formatting options.

Core APIs

pytest-benchmark Command

pytest-benchmark [-h] [--storage STORAGE] [--netrc] [--verbose] [--quiet] 
                 [--import-mode {prepend,append,importlib}] 
                 {list,compare,help} ...

Entry point for all CLI operations. Available as both pytest-benchmark and py.test-benchmark.

Global Options:

  • --storage, -s: Storage URI (default: file://./.benchmarks)
  • --netrc: Load Elasticsearch credentials from netrc file (default: ~/.netrc)
  • --verbose, -v: Enable verbose diagnostic and progress information
  • --quiet, -q: Disable reporting (verbose mode takes precedence)
  • --import-mode: How to load hooks from conftest (choices: prepend, append, importlib)

Subcommands

list Command

pytest-benchmark list [-h]

Lists all saved benchmark runs in the configured storage location.

compare Command

pytest-benchmark compare [-h] [--sort {min,max,mean,stddev,name,fullname}]
                        [--group-by {group,name,fullname,func,fullfunc,param}]
                        [--columns COLUMNS] [--name {short,normal,long,trial}]
                        [--histogram [FILENAME_PREFIX]] [--csv [FILENAME]]
                        [glob_or_file ...]

Compare and display results from saved benchmark runs.

Display Options:

  • --sort: Column to sort by (default: min)
  • --group-by: How to group tests (default: group)
  • --columns: Comma-separated list of columns to display
  • --name: Format for test names (default: normal)

Output Options:

  • --histogram [FILENAME_PREFIX]: Generate SVG histogram plots
  • --csv [FILENAME]: Export results to CSV format

File Selection:

  • glob_or_file: Positional arguments specifying which files/runs to load

help Command

pytest-benchmark help [command]

Display help information for commands.

Usage Examples

Listing Saved Benchmarks

# List all saved benchmark runs
pytest-benchmark list

# List with verbose output
pytest-benchmark --verbose list

# List from custom storage location
pytest-benchmark --storage file:///custom/path list

Comparing Benchmark Results

Basic Comparison

# Compare all saved benchmarks
pytest-benchmark compare

# Compare specific run by number
pytest-benchmark compare 0001

# Compare multiple specific runs
pytest-benchmark compare 0001 0002 0003

Pattern-Based Selection

# Compare all benchmarks from specific interpreter
pytest-benchmark compare 'Linux-CPython-3.5-64bit/*'

# Compare benchmarks matching pattern
pytest-benchmark compare '*test_performance*'

# Compare specific files by path
pytest-benchmark compare /path/to/0001_abc.json /path/to/0002_def.json

Output Customization

# Sort by different column
pytest-benchmark compare --sort mean 0001

# Group by function name instead of group
pytest-benchmark compare --group-by func 0001

# Show only specific columns
pytest-benchmark compare --columns min,max,mean 0001

# Use short test names
pytest-benchmark compare --name short 0001

Export and Visualization

# Generate histogram plots
pytest-benchmark compare --histogram mytest_ 0001

# Export to CSV
pytest-benchmark compare --csv results.csv 0001

# Both histogram and CSV
pytest-benchmark compare --histogram plots_ --csv data.csv 0001

Storage Configuration

File Storage

# Use default location (./.benchmarks)
pytest-benchmark compare 0001

# Use custom file location
pytest-benchmark --storage file:///custom/benchmarks compare 0001

# Use relative path
pytest-benchmark --storage file://./results compare 0001

Elasticsearch Storage

# Connect to Elasticsearch
pytest-benchmark --storage http://localhost:9200 compare 0001

# Use authentication with netrc
pytest-benchmark --storage https://user:pass@es.example.com --netrc compare 0001

File and Pattern Matching

Glob Patterns

The CLI supports glob patterns for flexible file selection:

# All benchmarks from Linux CPython 3.8
pytest-benchmark compare 'Linux-CPython-3.8*/*'

# All benchmarks containing 'performance' in name
pytest-benchmark compare '*performance*'

# All JSON files in subdirectories
pytest-benchmark compare '*/\*.json'

Note: Use single quotes to prevent shell glob expansion.

Run Number Selection

# First run from all interpreters
pytest-benchmark compare 0001

# Specific runs
pytest-benchmark compare 0001 0005 0010

# Range using shell expansion
pytest-benchmark compare 000{1..5}

Direct File Paths

# Absolute paths
pytest-benchmark compare /full/path/to/benchmark.json

# Relative paths
pytest-benchmark compare ./results/test_001.json ../other/test_002.json

# Mixed selection
pytest-benchmark compare 0001 ./custom.json 'Linux*/*'

Output Formats

Table Display

Default tabular output with customizable columns:

# Default columns: min, max, mean, stddev, median, iqr, outliers, ops, rounds, iterations
pytest-benchmark compare 0001

# Custom columns
pytest-benchmark compare --columns min,mean,ops,rounds 0001

# Minimal display
pytest-benchmark compare --columns name,min,ops 0001

Histogram Generation

Visual analysis with SVG plots:

# Generate with automatic filename
pytest-benchmark compare --histogram 0001

# Custom filename prefix
pytest-benchmark compare --histogram mytest_comparison_ 0001

# Creates files like: mytest_comparison_test_function_name.svg

CSV Export

Data export for further analysis:

# Export with automatic filename
pytest-benchmark compare --csv 0001

# Custom filename
pytest-benchmark compare --csv benchmark_results.csv 0001

# Export includes all benchmark metadata and statistics

Integration Examples

Continuous Integration

#!/bin/bash
# Compare current run with baseline
pytest-benchmark compare --csv ci_results.csv baseline current

# Check for performance regressions
if [ $? -ne 0 ]; then
    echo "Performance regression detected!"
    exit 1
fi

Performance Monitoring

# Weekly performance report
pytest-benchmark compare --histogram weekly_report_ --csv weekly_data.csv 'week_*'

# Generate trending analysis
for run in $(pytest-benchmark list | grep -o '[0-9]\{4\}'); do
    pytest-benchmark compare --csv "trend_$run.csv" "$run"
done

Development Workflow

# Compare feature branch against main
pytest-benchmark compare main_baseline feature_test

# Quick performance check
pytest-benchmark compare --columns name,min,ops --name short latest

# Detailed analysis with visuals
pytest-benchmark compare --histogram analysis_ --csv detailed.csv latest

Error Handling and Troubleshooting

Common Issues

# Storage not found
pytest-benchmark --storage file:///missing/path list
# Error: Storage location does not exist

# Invalid glob pattern
pytest-benchmark compare 'unclosed_quote
# Error: Invalid glob pattern

# No matching files
pytest-benchmark compare nonexistent_pattern
# Warning: No benchmark files found matching pattern

Verbose Mode

# Debug storage and file loading issues
pytest-benchmark --verbose compare problematic_pattern

# Shows:
# - Storage backend initialization
# - File discovery and loading
# - Data processing steps
# - Error details with stack traces

Help and Documentation

# General help
pytest-benchmark --help

# Command-specific help
pytest-benchmark help compare
pytest-benchmark compare --help

# Show examples
pytest-benchmark help

Install with Tessl CLI

npx tessl i tessl/pypi-pytest-benchmark

docs

analysis-reporting.md

aspect-benchmarking.md

cli-tools.md

configuration.md

core-benchmarking.md

index.md

storage-comparison.md

tile.json