Pytest patterns for Python APIs -- httpx AsyncClient, conftest fixtures, database isolation, parametrize edge cases, error response testing, auth flows, factory fixtures
99
99%
Does it follow best practices?
Impact
100%
1.23xAverage score across 5 eval scenarios
Passed
No known issues
{
"context": "Tests whether the agent writes auth flow tests (with/without credentials, invalid token), creates an auth_headers fixture in conftest.py, and uses monkeypatch (not unittest.mock) for simulating external service failures.",
"type": "weighted_checklist",
"checklist": [
{
"name": "No-token test",
"description": "There is a test that calls a protected endpoint without any Authorization header and expects a 401",
"max_score": 10
},
{
"name": "Invalid token test",
"description": "There is a test that sends an invalid/fake Bearer token and expects a 401",
"max_score": 10
},
{
"name": "Valid token grants access",
"description": "There is a test that uses a valid token and expects a non-401 success response",
"max_score": 10
},
{
"name": "auth_headers fixture",
"description": "conftest.py defines an auth_headers (or equivalent) fixture that returns a dict with an Authorization Bearer header",
"max_score": 12
},
{
"name": "auth_headers in conftest",
"description": "The auth helper fixture is defined in tests/conftest.py (not inline in the test function)",
"max_score": 8
},
{
"name": "monkeypatch for SMTP failure",
"description": "The external service failure test uses monkeypatch.setattr (not unittest.mock.patch) to simulate the ConnectionError",
"max_score": 15
},
{
"name": "No unittest.mock for simple patch",
"description": "The test file does NOT import unittest.mock or use mock.patch for the simple service replacement (monkeypatch is used instead)",
"max_score": 10
},
{
"name": "503 on service failure",
"description": "The SMTP-down test asserts a 503 response status",
"max_score": 10
},
{
"name": "Login flow in fixture",
"description": "The auth_headers fixture performs a full login flow (calls the /api/auth/login endpoint or equivalent) rather than constructing a token manually from the secret key",
"max_score": 10
},
{
"name": "asyncio_mode config",
"description": "pyproject.toml includes asyncio_mode = \"auto\" under [tool.pytest.ini_options]",
"max_score": 5
}
]
}