CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl-labs/pytest-api-testing

Pytest patterns for Python APIs -- httpx AsyncClient, conftest fixtures, database isolation, parametrize edge cases, error response testing, auth flows, factory fixtures

99

1.23x
Quality

99%

Does it follow best practices?

Impact

100%

1.23x

Average score across 5 eval scenarios

SecuritybySnyk

Passed

No known issues

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-5/

{
  "context": "Tests whether the agent writes auth flow tests (with/without credentials, invalid token), creates an auth_headers fixture in conftest.py, and uses monkeypatch (not unittest.mock) for simulating external service failures.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "No-token test",
      "description": "There is a test that calls a protected endpoint without any Authorization header and expects a 401",
      "max_score": 10
    },
    {
      "name": "Invalid token test",
      "description": "There is a test that sends an invalid/fake Bearer token and expects a 401",
      "max_score": 10
    },
    {
      "name": "Valid token grants access",
      "description": "There is a test that uses a valid token and expects a non-401 success response",
      "max_score": 10
    },
    {
      "name": "auth_headers fixture",
      "description": "conftest.py defines an auth_headers (or equivalent) fixture that returns a dict with an Authorization Bearer header",
      "max_score": 12
    },
    {
      "name": "auth_headers in conftest",
      "description": "The auth helper fixture is defined in tests/conftest.py (not inline in the test function)",
      "max_score": 8
    },
    {
      "name": "monkeypatch for SMTP failure",
      "description": "The external service failure test uses monkeypatch.setattr (not unittest.mock.patch) to simulate the ConnectionError",
      "max_score": 15
    },
    {
      "name": "No unittest.mock for simple patch",
      "description": "The test file does NOT import unittest.mock or use mock.patch for the simple service replacement (monkeypatch is used instead)",
      "max_score": 10
    },
    {
      "name": "503 on service failure",
      "description": "The SMTP-down test asserts a 503 response status",
      "max_score": 10
    },
    {
      "name": "Login flow in fixture",
      "description": "The auth_headers fixture performs a full login flow (calls the /api/auth/login endpoint or equivalent) rather than constructing a token manually from the secret key",
      "max_score": 10
    },
    {
      "name": "asyncio_mode config",
      "description": "pyproject.toml includes asyncio_mode = \"auto\" under [tool.pytest.ini_options]",
      "max_score": 5
    }
  ]
}

evals

tile.json