Review PR comments, address code issues in source files (not generated files), regenerate derived artifacts, run lint/format, commit, push, and reply to the comment thread confirming resolution.
93
Quality
89%
Does it follow best practices?
Impact
99%
1.19xAverage score across 5 eval scenarios
{
"context": "Tests whether the agent edits only source files (not generated files directly), regenerates derived artifacts after fixing the source, runs linting/verification, stages only relevant files, and writes a properly formatted commit message.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Source schema edited",
"description": "src/schemas/product.ts is modified to change the id type from number to string",
"max_score": 10
},
{
"name": "Generated types not hand-edited",
"description": "generated/types.ts content matches what the generate.js script would produce from the updated schema (i.e., was regenerated, not manually edited)",
"max_score": 10
},
{
"name": "Generation script executed",
"description": "workflow_log.md records that the generation script (scripts/generate.js) was run after editing the source",
"max_score": 10
},
{
"name": "Lint verification",
"description": "workflow_log.md records that linting (scripts/lint.js) was run after regeneration",
"max_score": 10
},
{
"name": "Minimal source changes",
"description": "Changes to src/schemas/product.ts are limited to the fields mentioned in the review comments (id and/or price) — no unrelated refactoring",
"max_score": 10
},
{
"name": "Handler updated consistently",
"description": "src/handlers/product.ts is updated to be consistent with the schema changes (e.g., typeof check updated from 'number' to 'string' for id)",
"max_score": 10
},
{
"name": "Staged files listed",
"description": "staged_files.txt lists specific files (both source and regenerated) rather than using a wildcard or staging everything",
"max_score": 10
},
{
"name": "Commit message format",
"description": "commit_message.txt starts with 'fix:' followed by a concise description",
"max_score": 10
},
{
"name": "Co-authored-by line",
"description": "commit_message.txt includes a 'Co-Authored-By: Claude' line (any version/variant)",
"max_score": 10
},
{
"name": "Workflow ordering",
"description": "workflow_log.md shows the correct order: fix source -> regenerate -> verify/lint (not lint before regenerate, or regenerate before fix)",
"max_score": 10
}
]
}Install with Tessl CLI
npx tessl i sahildmk/pr-comment-resolver