CtrlK
BlogDocsLog inGet started
Tessl Logo

sahildmk/pr-comment-resolver

Review PR comments, address code issues in source files (not generated files), regenerate derived artifacts, run lint/format, commit, push, and reply to the comment thread confirming resolution.

93

1.19x

Quality

89%

Does it follow best practices?

Impact

99%

1.19x

Average score across 5 eval scenarios

Overview
Skills
Evals
Files

rubric.jsonevals/scenario-3/

{
  "context": "Tests whether the agent edits only source files (not generated files directly), regenerates derived artifacts after fixing the source, runs linting/verification, stages only relevant files, and writes a properly formatted commit message.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Source schema edited",
      "description": "src/schemas/product.ts is modified to change the id type from number to string",
      "max_score": 10
    },
    {
      "name": "Generated types not hand-edited",
      "description": "generated/types.ts content matches what the generate.js script would produce from the updated schema (i.e., was regenerated, not manually edited)",
      "max_score": 10
    },
    {
      "name": "Generation script executed",
      "description": "workflow_log.md records that the generation script (scripts/generate.js) was run after editing the source",
      "max_score": 10
    },
    {
      "name": "Lint verification",
      "description": "workflow_log.md records that linting (scripts/lint.js) was run after regeneration",
      "max_score": 10
    },
    {
      "name": "Minimal source changes",
      "description": "Changes to src/schemas/product.ts are limited to the fields mentioned in the review comments (id and/or price) — no unrelated refactoring",
      "max_score": 10
    },
    {
      "name": "Handler updated consistently",
      "description": "src/handlers/product.ts is updated to be consistent with the schema changes (e.g., typeof check updated from 'number' to 'string' for id)",
      "max_score": 10
    },
    {
      "name": "Staged files listed",
      "description": "staged_files.txt lists specific files (both source and regenerated) rather than using a wildcard or staging everything",
      "max_score": 10
    },
    {
      "name": "Commit message format",
      "description": "commit_message.txt starts with 'fix:' followed by a concise description",
      "max_score": 10
    },
    {
      "name": "Co-authored-by line",
      "description": "commit_message.txt includes a 'Co-Authored-By: Claude' line (any version/variant)",
      "max_score": 10
    },
    {
      "name": "Workflow ordering",
      "description": "workflow_log.md shows the correct order: fix source -> regenerate -> verify/lint (not lint before regenerate, or regenerate before fix)",
      "max_score": 10
    }
  ]
}

Install with Tessl CLI

npx tessl i sahildmk/pr-comment-resolver@0.3.1

evals

tile.json