sahildmk/pr-comment-resolver
Review PR comments, address code issues in source files (not generated files), regenerate derived artifacts, run lint/format, commit, push, and reply to the comment thread confirming resolution.
93
Quality
89%
Does it follow best practices?
Impact
99%
1.19xAverage score across 5 eval scenarios
Evaluation results
95%
5%PR Comment Filtering Tool
PR comment filtering logic
Review comments endpoint
100%
100%
Authenticated user lookup
100%
100%
Exclude self-replied
100%
100%
Top-level only filtering
100%
100%
Agent commit timestamp
60%
100%
Since parameter usage
100%
100%
First-run handling
100%
100%
Stop when empty
40%
50%
Reply-user matching
100%
100%
Test coverage
100%
100%
Without context: $0.4503 · 1m 55s · 15 turns · 64 in / 7,635 out tokens
With context: $0.9782 · 3m 58s · 30 turns · 295 in / 15,254 out tokens
100%
2%PR Review Comment Assessment Report
Comment assessment and user confirmation
Three decision categories
100%
100%
Generated file recognition
100%
100%
OpenAPI generated file
80%
100%
Defer large refactor
100%
100%
Rationale provided
100%
100%
Defer/disagree reasoning
100%
100%
User confirmation plan
100%
100%
Plan shows all comments
100%
100%
Critical assessment
100%
100%
Source file targets only
100%
100%
Without context: $0.3326 · 1m 40s · 12 turns · 61 in / 5,351 out tokens
With context: $0.4415 · 2m · 16 turns · 63 in / 6,967 out tokens
100%
10%Fix Review Feedback in a TypeScript API Project
Source-only editing and artifact regeneration
Source schema edited
100%
100%
Generated types not hand-edited
100%
100%
Generation script executed
100%
100%
Lint verification
100%
100%
Minimal source changes
100%
100%
Handler updated consistently
100%
100%
Staged files listed
100%
100%
Commit message format
100%
100%
Co-authored-by line
0%
100%
Workflow ordering
100%
100%
Without context: $0.8583 · 3m 4s · 32 turns · 33 in / 11,202 out tokens
With context: $1.0396 · 3m 50s · 38 turns · 303 in / 13,074 out tokens
100%
15%GitHub PR Reply Automation Script
Commit format and reply workflow
Replies endpoint
100%
100%
Top-level comment ID
100%
100%
Diagnose before fallback
100%
100%
Verify IDs on failure
100%
100%
Issue comment fallback last
100%
100%
Short addressed reply
80%
100%
Short defer reply
70%
100%
Disagree with reason
100%
100%
Commit message format
100%
100%
Co-authored-by attribution
0%
100%
Without context: $0.5755 · 2m 57s · 17 turns · 66 in / 10,658 out tokens
With context: $0.8218 · 3m 15s · 31 turns · 79 in / 11,239 out tokens
100%
48%PR Comment Resolver CLI Tool
End-to-end PR comment resolver workflow
Review comments API
100%
100%
Comment filtering stage
62%
100%
Assessment stage
90%
100%
User confirmation stage
100%
100%
Source-only editing
0%
100%
Regeneration stage
0%
100%
Verification stage
0%
100%
Correct workflow order
40%
100%
Reply via thread endpoint
100%
100%
One commit per comment
0%
100%
Commit format
50%
100%
Empty result handling
100%
100%
Without context: $0.5238 · 2m 44s · 12 turns · 13 in / 11,076 out tokens
With context: $0.6014 · 2m 47s · 18 turns · 17 in / 10,621 out tokens
Install with Tessl CLI
npx tessl i sahildmk/pr-comment-resolver- Evaluated
- Agent
- Claude Code
- Model
- Claude Sonnet 4.6