or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.md
tile.json

tessl/pypi-pytest-randomly

Pytest plugin to randomly order tests and control random.seed.

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/pytest-randomly@3.16.x

To install, run

npx @tessl/cli install tessl/pypi-pytest-randomly@3.16.0

index.mddocs/

pytest-randomly

A pytest plugin that introduces controlled randomness into test execution to improve test quality and discover hidden dependencies. It randomly shuffles test execution order and resets random seeds to ensure reproducible test runs.

Package Information

  • Package Name: pytest-randomly
  • Package Type: pypi
  • Language: Python
  • Installation: pip install pytest-randomly

Core Imports

import pytest_randomly

The plugin is automatically discovered by pytest via entry points - no explicit imports needed.

Basic Usage

# Install the plugin
# pip install pytest-randomly

# Run tests with automatic randomization (default behavior)
pytest

# Run tests with specific seed for reproducibility
pytest --randomly-seed=1234

# Reuse seed from previous run
pytest --randomly-seed=last

# Disable test order shuffling but keep seed resetting
pytest --randomly-dont-reorganize

# Disable seed resetting but keep order shuffling
pytest --randomly-dont-reset-seed

# Completely disable the plugin
pytest -p no:randomly

Architecture

The plugin provides controlled randomness through:

  • Test Order Randomization: Shuffles tests at module, class, and function levels using deterministic seeding
  • Random State Management: Resets global random seeds before each test for reproducibility
  • Third-party Integration: Automatically resets random states for factory-boy, faker, model-bakery, and numpy
  • pytest-xdist Support: Distributes consistent seeds across parallel worker processes
  • Extensibility: Entry point system for third-party random seeder registration

Capabilities

Plugin Configuration

Adds command-line options and configures the plugin when pytest starts.

def pytest_addoption(parser: Parser) -> None:
    """
    Adds command-line options for controlling randomness behavior.
    
    Options added:
    - --randomly-seed: Set seed value (int, 'default', or 'last')
    - --randomly-dont-reset-seed: Disable random seed reset per test
    - --randomly-dont-reorganize: Disable test order shuffling
    """

def pytest_configure(config: Config) -> None:
    """
    Configures the plugin when pytest starts.
    
    Handles seed generation, xdist integration, and caching.
    """

Test Execution Hooks

Controls random state during different phases of test execution.

def pytest_runtest_setup(item: Item) -> None:
    """
    Resets random state before each test setup phase.
    
    Args:
        item: The test item being set up
    """

def pytest_runtest_call(item: Item) -> None:
    """
    Resets random state before each test execution phase.
    
    Args:
        item: The test item being executed
    """

def pytest_runtest_teardown(item: Item) -> None:
    """
    Resets random state after each test teardown phase.
    
    Args:
        item: The test item being torn down
    """

Test Collection and Ordering

Randomly reorders collected test items for better test isolation detection.

@hookimpl(tryfirst=True)
def pytest_collection_modifyitems(config: Config, items: list[Item]) -> None:
    """
    Randomly reorders collected test items by module, class, then function.
    
    Uses deterministic shuffling based on seed value to ensure
    reproducible test order when using the same seed.
    
    Args:
        config: pytest configuration object
        items: List of collected test items to reorder
    """

Reporting

Provides information about the randomization seed being used.

def pytest_report_header(config: Config) -> str:
    """
    Returns header line showing the seed being used.
    
    Args:
        config: pytest configuration object
        
    Returns:
        Header string with seed information (e.g., "Using --randomly-seed=1234")
    """

Utility Functions

Helper functions for seed validation and argument parsing.

def seed_type(string: str) -> str | int:
    """
    Argument parser type converter for --randomly-seed option.
    
    Args:
        string: String value to convert
        
    Returns:
        Integer seed value, or 'default'/'last' string values
        
    Raises:
        argparse.ArgumentTypeError: For invalid seed values
    """

def reduce_list_of_lists(lists: list[list[T]]) -> list[T]:
    """
    Flattens a list of lists into a single list.
    
    Args:
        lists: List of lists to flatten
        
    Returns:
        Single flattened list containing all elements
    """

pytest-xdist Integration

Handles parallel test execution with consistent seeding across workers.

class XdistHooks:
    """
    Hooks container for pytest-xdist integration.
    
    Automatically registered when xdist plugin is detected.
    """
    
    def pytest_configure_node(self, node: Item) -> None:
        """
        Configures worker nodes in xdist with consistent seed values.
        
        Args:
            node: Worker node to configure
        """

faker Integration

Provides seed configuration for the faker pytest plugin when faker is installed.

Note: This fixture is only defined when the faker package is available at import time.

# Only available when faker is installed
if have_faker:
    @fixture(autouse=True)
    def faker_seed(pytestconfig: Config) -> int:
        """
        Provides faker seed configuration to faker pytest plugin.
        
        Conditionally defined fixture that only exists when faker package 
        is available at plugin import time.
        
        Args:
            pytestconfig: pytest configuration object
            
        Returns:
            Current randomly seed value for faker integration
        """

Types

# Module-level variables
default_seed: int  # Default random seed generated at import time
random_states: dict[int, tuple[Any, ...]]  # Cache for random states by seed
np_random_states: dict[int, Any]  # Cache for numpy random states by seed
entrypoint_reseeds: list[Callable[[int], None]] | None  # Cached entry point reseeder functions

# Third-party library availability flags
have_factory_boy: bool  # True if factory-boy is available
have_faker: bool        # True if faker is available
have_model_bakery: bool # True if model-bakery is available
have_numpy: bool        # True if numpy is available

# Optional module imports
xdist: ModuleType | None  # pytest-xdist module if available, None otherwise

# Type variable for generic list flattening
T = TypeVar("T")

Third-party Integration

The plugin automatically detects and integrates with popular Python testing libraries when they are available:

  • factory-boy (if have_factory_boy is True): Resets factory.random state for consistent factory generation
  • faker (if have_faker is True): Resets faker.generator.random state for reproducible fake data
  • model-bakery (if have_model_bakery is True): Resets model_bakery.random_gen.baker_random state
  • numpy (if have_numpy is True): Resets numpy.random legacy global state for consistent array generation
  • pytest-xdist (if detected): Distributes seed values to parallel worker processes

Extension API

Third-party packages can register custom random seeders via the pytest_randomly.random_seeder entry point group:

# In your setup.py or pyproject.toml
entry_points = {
    "pytest_randomly.random_seeder": [
        "your_lib = your_package:reseed_function"
    ]
}

# Your reseed function signature
def reseed_function(seed: int) -> None:
    """Reset your random generator with the provided seed."""
    your_random_generator.seed(seed)

Command Line Options

  • --randomly-seed=<int|'default'|'last'>: Control the random seed value
  • --randomly-dont-reset-seed: Disable per-test seed reset while keeping order shuffling
  • --randomly-dont-reorganize: Disable test order shuffling while keeping seed reset
  • -p no:randomly: Completely disable the plugin