schemathesis-fuzzing

Overview

Schemathesis is a property-based API testing tool that "automatically generates property-based tests from your OpenAPI or GraphQL schema and exercises the edge cases that break your API" (schemathesis-readme). The schema is the single source of truth - every endpoint, parameter, and response shape becomes a generator that produces hundreds of targeted variations per run.

This is complementary to example-based API testing (postman-collections, tavern-testing, restassured-testing, karate-testing) - example-based tests verify happy paths; Schemathesis attacks the boundaries the team forgot.

When to use

• The repo has an OpenAPI 2.0 / 3.0 / 3.1 / 3.2 spec or a GraphQL schema (June 2018+).
• The team wants automatic test coverage of new endpoints without authoring per-endpoint tests.
• Negative-scenario coverage ("what happens with a malformed request?") is missing or thin.
• A CI gate on 5xx responses (any unexpected server error) is desired.

If the API has no schema, Schemathesis cannot help - generate a schema first (FastAPI, Flask-Smorest, NestJS Swagger, dropwizard, or hand-author a spec), then return.

How to use

1. Confirm the API ships an OpenAPI 2/3.x or GraphQL schema (see When to use).
2. Install the CLI (pip install schemathesis, or uvx for a one-off run).
3. Run schemathesis run <schema> against staging with the default checks; reproduce any failure from the printed curl command + Hypothesis seed.
4. Promote it to a first-class pytest test (@schema.parametrize()) and gate CI on schema-conformance + 5xx detection - deep CI wiring and advanced auth hooks live in references/ci-and-pytest-integration.md.

Install

pip install schemathesis

(Per schemathesis-readme.)

For the latest CLI without modifying the project's Python env:

uvx schemathesis run <schema-url>

(Adapted from schemathesis-docs - uvx is the uv runner for one-off tool execution.)

Running via CLI

Basic invocation per schemathesis-readme:

schemathesis run <schema-url>

<schema-url> can be:

• Remote URL: https://api.example.com/openapi.json.
• Local file: ./openapi.yaml.

Key flags

Per schemathesis-readme:

Worked example

schemathesis run https://api.example.com/openapi.json \
  --base-url https://staging.example.com \
  --checks status_code_conformance \
  --checks response_schema_conformance \
  --checks not_a_server_error \
  --hypothesis-max-examples 200 \
  --workers 4 \
  --header "Authorization: Bearer $API_TOKEN"

Built-in checks

Per schemathesis-readme, the canonical checks:

Run all checks (default), or restrict via repeated --checks:

# Strict: every check active
schemathesis run <schema>

# Only flag 5xx errors (cheap smoke test)
schemathesis run <schema> --checks not_a_server_error

A failing check produces a deterministic reproduction - Schemathesis prints the exact curl command and Hypothesis seed to reproduce the generated request.

Pytest integration

For projects that want Schemathesis cases as first-class pytest tests (schemathesis-readme):

# tests/api/test_schemathesis.py
import schemathesis

schema = schemathesis.openapi.from_url("https://your-api.com/openapi.json")

@schema.parametrize()
def test_api(case):
    case.call_and_validate()

@schema.parametrize() generates one pytest test per endpoint × method combination. case.call_and_validate() issues the generated request and runs every default check.

For project-specific auth injection, header signatures, or response post-processing, use before_call hooks - see references/ci-and-pytest-integration.md.

Operating in CI

Gate PRs on Schemathesis with a shallow per-PR run (--hypothesis-max-examples 50) and a deeper nightly cron (200+), always against staging via --base-url. Hypothesis shrinks any failure to a minimal reproducer, so a 50-example PR run catches most regressions while the nightly depth surfaces the rare ones. The full GitHub Actions workflow (JUnit reporting, artifact upload) and the per-PR / nightly / weekly cadence table live in references/ci-and-pytest-integration.md.

Anti-patterns

Limitations

• Authentication state. Schemathesis can attach a token but doesn't model multi-step auth flows (login → token → use). Combine with tavern-testing or restassured-testing for the auth chain.
• Stateful sequences. Out of the box, every Schemathesis case is independent. For stateful API fuzzing (POST → GET created resource → DELETE), use restler-fuzzing, which is built for stateful sequences.
• Schema drift. If the schema doesn't match the implementation, every case fails for the wrong reason. Run a Schemathesis baseline pre-PR to catch schema drift early.
• OpenAPI 2.0 limitations. Some OpenAPI 2.0 features (e.g. formData) generate weaker variations than 3.x; consider migrating the schema to 3.x.

References

• schemathesis-readme - main repo: install, CLI flags, built-in checks, pytest integration with @schema.parametrize().
• schemathesis-docs - full docs (CLI reference, hooks, authentication strategies).
• restler-fuzzing - stateful fuzzing complement.
• postman-collections, tavern-testing, restassured-testing, karate-testing - example-based authoring; use alongside Schemathesis for happy-path coverage.