akshay-babbar/doc-sync
Auto-syncs stale docstrings and README when function signatures change. Detects documentation drift after refactors, parameter additions, or return type changes. Dry-run by default — proposes before writing.
87
100%
Does it follow best practices?
Impact
86%
1.59xAverage score across 17 eval scenarios
Passed
No known issues
task.mdevals/scenario-3/
Scenario: Python deprecated decorator added; must surface deprecation and flag README
A Python library added a @deprecated decorator to a previously documented function without changing its signature. This is a caller-visible contract change (the function is now deprecated) even though the signature is identical. The README references the function in a code span.
Your job is to update only previously documented symbols whose caller-visible contract changed, never auto-write markdown, and flag uncertainty.
Baseline (committed) state
src/api.py
from __future__ import annotations
def fetch_user(user_id: int) -> dict:
"""Fetch a user record.
Args:
user_id: The user's integer ID.
Returns:
A user dict.
"""
return {"id": user_id}README.md
# user-service
Call `fetch_user` to load user data.
Do not auto-apply markdown edits.Working tree (current) state
A @deprecated decorator was added. Signature and behavior otherwise unchanged.
src/api.py
from __future__ import annotations
def deprecated(reason: str):
def _wrap(fn):
fn.__deprecated_reason__ = reason
return fn
return _wrap
@deprecated("Use get_user(user_id) instead")
def fetch_user(user_id: int) -> dict:
"""Fetch a user record.
Args:
user_id: The user's integer ID.
Returns:
A user dict.
"""
return {"id": user_id}README.md (unchanged)
# user-service
Call `fetch_user` to load user data.
Do not auto-apply markdown edits.Git setup
Init repo, commit baseline, overwrite with working tree so git diff HEAD catches the decorator addition and nothing else:
git init
mkdir -p src
cat > src/api.py <<'EOF'
from __future__ import annotations
def fetch_user(user_id: int) -> dict:
"""Fetch a user record.
Args:
user_id: The user's integer ID.
Returns:
A user dict.
"""
return {"id": user_id}
EOF
cat > README.md <<'EOF'
# user-service
Call `fetch_user` to load user data.
Do not auto-apply markdown edits.
EOF
git add -A && git commit -m "baseline"
cat > src/api.py <<'EOF'
from __future__ import annotations
def deprecated(reason: str):
def _wrap(fn):
fn.__deprecated_reason__ = reason
return fn
return _wrap
@deprecated("Use get_user(user_id) instead")
def fetch_user(user_id: int) -> dict:
"""Fetch a user record.
Args:
user_id: The user's integer ID.
Returns:
A user dict.
"""
return {"id": user_id}
EOFOutput spec
Sync the documentation for this change and write the results to doc-sync-report.md. It must:
- Update the
fetch_userdocstring to explicitly note the deprecation (minimal change). - Flag the README mention of
fetch_userfor human review (do not auto-edit markdown).
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10
scenario-11
scenario-12
scenario-13
scenario-14
scenario-15
scenario-16
scenario-17