akshay-babbar/doc-sync
Auto-syncs stale docstrings and README when function signatures change. Detects documentation drift after refactors, parameter additions, or return type changes. Dry-run by default — proposes before writing.
87
100%
Does it follow best practices?
Impact
86%
1.59xAverage score across 17 eval scenarios
Passed
No known issues
task.mdevals/scenario-15/
Scenario: Protected file mention exists alongside a valid README proposal
A Python package changed a documented function signature. The symbol is mentioned in two markdown files: a regular README.md section that should receive a propose-only update, and a CHANGELOG.md entry that must be ignored completely. The skill must update the docstring, propose the README change, and never mention or propose the changelog entry.
Baseline (committed) state
src/reports.py
def publish_report(report_id: str) -> str:
"""Publish a stored report.
Args:
report_id: Report identifier to publish.
Returns:
Published report identifier.
"""
return report_idREADME.md
# reports-kit
Use `publish_report` to publish a generated report for downstream systems.
README updates are propose-only.CHANGELOG.md
# Changelog
## v1.4.0
- Improved `publish_report` reliability during weekend batch runs.Working tree (current) state
The function adds a new optional parameter include_archived. README and changelog are otherwise unchanged.
src/reports.py
def publish_report(report_id: str, include_archived: bool = False) -> str:
"""Publish a stored report.
Args:
report_id: Report identifier to publish.
Returns:
Published report identifier.
"""
_ = include_archived
return report_idREADME.md (unchanged)
# reports-kit
Use `publish_report` to publish a generated report for downstream systems.
README updates are propose-only.CHANGELOG.md (unchanged)
# Changelog
## v1.4.0
- Improved `publish_report` reliability during weekend batch runs.Git setup
git init
mkdir -p src
cat > src/reports.py <<'EOF'
def publish_report(report_id: str) -> str:
"""Publish a stored report.
Args:
report_id: Report identifier to publish.
Returns:
Published report identifier.
"""
return report_id
EOF
cat > README.md <<'EOF'
# reports-kit
Use `publish_report` to publish a generated report for downstream systems.
README updates are propose-only.
EOF
cat > CHANGELOG.md <<'EOF'
# Changelog
## v1.4.0
- Improved `publish_report` reliability during weekend batch runs.
EOF
git add -A && git commit -m "baseline"
cat > src/reports.py <<'EOF'
def publish_report(report_id: str, include_archived: bool = False) -> str:
"""Publish a stored report.
Args:
report_id: Report identifier to publish.
Returns:
Published report identifier.
"""
_ = include_archived
return report_id
EOFOutput Specification
Sync the documentation and write the results to doc-sync-report.md. It must update the publish_report docstring, include a propose-only entry for README.md, and completely ignore CHANGELOG.md (no proposal, no flag, no mention).
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10
scenario-11
scenario-12
scenario-13
scenario-14
scenario-15
scenario-16
scenario-17