CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/pypi-pytest-codspeed

Pytest plugin to create CodSpeed benchmarks

—

Pending
Overview
Eval results
Files

configuration.mddocs/

Configuration

Command-line options and marker parameters for controlling benchmark behavior. pytest-codspeed provides extensive configuration through CLI flags, environment variables, and marker options.

Capabilities

Command Line Options

Options for controlling pytest-codspeed behavior from the command line.

# Enable CodSpeed benchmarking
--codspeed

# Set measurement mode  
--codspeed-mode {instrumentation,walltime}

# Walltime mode timing options
--codspeed-warmup-time FLOAT     # Warmup time in seconds
--codspeed-max-time FLOAT        # Maximum benchmark time in seconds  
--codspeed-max-rounds INT        # Maximum number of rounds

Usage Examples

# Basic benchmarking with walltime mode
pytest tests/ --codspeed

# Use instrumentation mode for more precise measurement
pytest tests/ --codspeed --codspeed-mode instrumentation

# Customize timing for walltime mode
pytest tests/ --codspeed --codspeed-warmup-time 2.0 --codspeed-max-time 10.0 --codspeed-max-rounds 50

Environment Variables

Environment variables that control pytest-codspeed behavior, primarily used in CI/CD environments.

# Core environment variables
CODSPEED_ENV                           # Enables CodSpeed when set (any value)
CODSPEED_RUNNER_MODE                   # Sets default mode: "walltime" or other
CODSPEED_PROFILE_FOLDER               # Directory for results output

# Extension build control  
PYTEST_CODSPEED_FORCE_EXTENSION_BUILD # Force native extension compilation
PYTEST_CODSPEED_SKIP_EXTENSION_BUILD  # Skip native extension compilation

Usage Examples

# Enable CodSpeed in CI environment
export CODSPEED_ENV=1
pytest tests/

# Force walltime mode in CI
export CODSPEED_ENV=1
export CODSPEED_RUNNER_MODE=walltime
pytest tests/

# Custom results directory
export CODSPEED_ENV=1  
export CODSPEED_PROFILE_FOLDER=/custom/results/path
pytest tests/

Marker Options

Parameters that can be passed to benchmark markers to customize individual test behavior.

@pytest.mark.benchmark(
    group: str = None,           # Group name for benchmark organization
    min_time: float = None,      # Minimum time per round (seconds, walltime only)
    max_time: float = None,      # Maximum total benchmark time (seconds, walltime only)  
    max_rounds: int = None       # Maximum number of rounds (walltime only)
)

@pytest.mark.codspeed_benchmark(
    group: str = None,           # Same options as benchmark marker
    min_time: float = None,
    max_time: float = None,
    max_rounds: int = None
)

Usage Examples

import pytest

# Group related benchmarks
@pytest.mark.benchmark(group="string_operations")
def test_string_join():
    result = "".join(str(i) for i in range(1000))
    assert len(result) > 0

@pytest.mark.benchmark(group="string_operations") 
def test_string_format():
    result = " ".join(f"item_{i}" for i in range(1000))
    assert len(result) > 0

# Control timing for slow operations
@pytest.mark.benchmark(max_time=10.0, max_rounds=20)
def test_expensive_computation():
    result = sum(i ** 2 for i in range(10000))
    assert result > 0

# Ensure minimum measurement time for fast operations  
@pytest.mark.benchmark(min_time=0.1)
def test_fast_operation():
    result = 1 + 1
    assert result == 2

Measurement Modes

pytest-codspeed supports two measurement approaches with different accuracy/overhead tradeoffs.

# Measurement modes
MeasurementMode.WallTime        # Wall-clock time measurement (default locally)
MeasurementMode.Instrumentation # Instruction-based measurement (default in CI)

Walltime Mode

  • Measures actual execution time using high-resolution timers
  • Includes statistical analysis with outlier detection
  • Configurable warmup, timing limits, and round counts
  • Provides detailed performance statistics and terminal reporting
  • Default mode for local development

Instrumentation Mode

  • Uses Valgrind-based instruction counting for deterministic results
  • Immune to system load and timing variations
  • Single measurement per benchmark (ignores rounds/iterations settings)
  • Requires CodSpeed environment for native extension
  • Default mode in CI environments

Configuration Precedence

Configuration values are resolved in this order (highest to lowest priority):

  1. Marker options (per-test customization)
  2. Command line options (pytest invocation)
  3. Environment variables (CI/CD configuration)
  4. Default values (built-in sensible defaults)

Advanced Configuration

Native Extension Control

# Force native extension build (raises error if impossible)
export PYTEST_CODSPEED_FORCE_EXTENSION_BUILD=1

# Skip native extension (falls back to walltime mode)
export PYTEST_CODSPEED_SKIP_EXTENSION_BUILD=1

Results Output

# Default local results location  
.codspeed/results_{timestamp}.json

# CI results location (when CODSPEED_PROFILE_FOLDER set)
{CODSPEED_PROFILE_FOLDER}/results/{pid}.json

Compatibility Settings

pytest-codspeed automatically handles compatibility with other benchmark plugins:

  • Disables pytest-benchmark when active
  • Disables pytest-speed when active
  • Preserves original fixtures as __benchmark if needed
  • Maintains compatibility with existing test suites

Install with Tessl CLI

npx tessl i tessl/pypi-pytest-codspeed

docs

advanced-benchmarking.md

basic-benchmarking.md

configuration.md

index.md

tile.json