CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl-labs/flask-testing

Write correct Flask tests -- app factory with test config, application context fixtures, database isolation, file uploads, auth testing, error handlers, mock.patch placement, and essential API test patterns

98

1.15x
Quality

99%

Does it follow best practices?

Impact

97%

1.15x

Average score across 5 eval scenarios

SecuritybySnyk

Passed

No known issues

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-1/

{
  "context": "Tests whether the agent proactively applies Flask testing best practices when writing a blog API test suite. The task does NOT mention application context, fixture structure, or database isolation patterns -- the agent should apply these on its own.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "App context in fixture",
      "description": "The agent creates an app fixture that wraps in 'with app.app_context():' and yields from inside it, ensuring the test client operates within an application context.",
      "max_score": 16
    },
    {
      "name": "Conftest fixture structure",
      "description": "The agent creates a conftest.py with separate app, client fixtures. The client fixture depends on the app fixture. The app fixture handles database initialization.",
      "max_score": 14
    },
    {
      "name": "Test config with TESTING=True",
      "description": "The agent passes test configuration to create_app with TESTING=True and a separate/in-memory database, not using the production database.",
      "max_score": 12
    },
    {
      "name": "Database isolation per test",
      "description": "The agent uses function-scoped fixtures (default scope) so each test gets a fresh database. Does NOT use scope='session' or scope='module' for the app/database fixture.",
      "max_score": 12
    },
    {
      "name": "Auth tests both paths",
      "description": "Tests verify both authenticated and unauthenticated access -- e.g., unauthenticated POST to /api/posts returns 401/403, authenticated POST succeeds. Auth is done through the login endpoint, not by faking cookies.",
      "max_score": 12
    },
    {
      "name": "Essential test patterns",
      "description": "The test suite includes: happy path (200 for GET list), validation rejection (400 for empty/invalid POST), 404 for missing posts, and persistence verification (create then retrieve).",
      "max_score": 10
    },
    {
      "name": "Error response body checked",
      "description": "Tests check error response JSON body structure (not just status codes), verifying that error responses contain an error key with a message.",
      "max_score": 8
    },
    {
      "name": "json= for POST requests",
      "description": "The agent uses json= parameter for JSON POST/PUT requests to the test client, not data= with manual content type headers.",
      "max_score": 6
    },
    {
      "name": "get_json() for responses",
      "description": "The agent uses res.get_json() to parse Flask test client responses, not res.json or json.loads(res.data).",
      "max_score": 6
    },
    {
      "name": "Run command documented",
      "description": "The agent documents that tests are run with 'pytest -v' or 'pytest'.",
      "max_score": 4
    }
  ]
}

evals

scenario-1

criteria.json

task.md

tile.json