Auto-syncs stale docstrings and README when function signatures change. Detects documentation drift after refactors, parameter additions, or return type changes. Dry-run by default — proposes before writing.
87
100%
Does it follow best practices?
Impact
86%
1.59xAverage score across 17 eval scenarios
Passed
No known issues
{
"context": "Tests whether the agent makes a minimal, surgical docstring edit — adding only the new timeout parameter documentation while leaving the Examples, Notes, See Also, Raises, and Returns sections completely untouched.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Timeout param documented",
"description": "src/connection.py docstring Args section contains a line documenting the timeout parameter (e.g. 'timeout: Connection timeout in seconds.')",
"max_score": 30
},
{
"name": "Examples section preserved",
"description": "src/connection.py docstring still contains the exact Examples section with `>>> conn = connect(\"localhost\", 8080)` and `>>> conn.send(b\"ping\")` unchanged",
"max_score": 45
},
{
"name": "Notes section preserved",
"description": "src/connection.py docstring still contains 'Connections are not thread-safe. Use one connection per thread.' unchanged",
"max_score": 45
},
{
"name": "See Also section preserved",
"description": "src/connection.py docstring still contains the See Also section with disconnect and reconnect entries unchanged",
"max_score": 45
},
{
"name": "Existing args untouched",
"description": "The 'host' and 'port' Args entries in the docstring are unchanged — no reordering or reformatting occurred",
"max_score": 45
},
{
"name": "Unified report format",
"description": "doc-sync-report.md contains '## Doc Sync Report' as a top-level heading",
"max_score": 45
}
]
}evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10
scenario-11
scenario-12
scenario-13
scenario-14
scenario-15
scenario-16
scenario-17