Every backend service MUST use structured logging — pino/structlog/slog, JSON format, request IDs, proper log levels. console.log is never acceptable.
88
85%
Does it follow best practices?
Impact
97%
3.03xAverage score across 4 eval scenarios
Passed
No known issues
{
"context": "Tests whether the agent proactively sets up structured logging when building a Python FastAPI service. The task describes only business requirements for a notification preferences API -- it never mentions logging, structlog, request IDs, or observability. A production-grade FastAPI service should use structured logging from the start.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Structured logger used (not print)",
"description": "Uses structlog, python-json-logger, or Python's logging module with a JSON formatter for all logging. No bare print() statements for operational output in the application code.",
"max_score": 15
},
{
"name": "Request ID middleware",
"description": "Middleware assigns a unique request ID to each request -- reads from x-request-id header if present, otherwise generates a UUID. The request ID is available in route handlers.",
"max_score": 12
},
{
"name": "Request ID in response header",
"description": "The assigned request ID is written back to the response via the x-request-id header.",
"max_score": 5
},
{
"name": "Request ID propagated to log entries",
"description": "Log entries include the request_id field -- achieved via structlog.contextvars, logging filters, or equivalent mechanism so all logs within a request carry the ID.",
"max_score": 10
},
{
"name": "Business events logged at info level",
"description": "Key business events (preferences created, preferences updated, notification check performed) are logged at info level with structured fields like user_id and notification_type -- not string interpolation.",
"max_score": 10
},
{
"name": "404 not logged as error",
"description": "Not-found (404) responses when preferences don't exist are logged at warn or info level -- NOT at error level.",
"max_score": 8
},
{
"name": "Per-request completion log",
"description": "A middleware logs one structured line per completed request containing HTTP method, path, status code, and response duration.",
"max_score": 8
},
{
"name": "Startup logged with structured logger",
"description": "Service startup is logged using the structured logger (not print).",
"max_score": 7
},
{
"name": "No sensitive data logged",
"description": "User IDs are acceptable in logs, but no PII like email addresses or phone numbers should appear in log output.",
"max_score": 7
},
{
"name": "Log level configurable via environment variable",
"description": "Logger level is set from an environment variable (e.g. LOG_LEVEL) with a sensible default.",
"max_score": 7
},
{
"name": "Context cleanup after request",
"description": "If using structlog contextvars, context is properly cleaned up (clear_contextvars or unbind_contextvars) to prevent context leaking between requests.",
"max_score": 6
},
{
"name": "Structured fields not string interpolation",
"description": "Log calls pass data as keyword arguments or dict fields rather than f-strings or string concatenation.",
"max_score": 5
}
]
}