General-purpose coding policy for Baruch's AI agents
90
91%
Does it follow best practices?
Impact
90%
1.30xAverage score across 18 eval scenarios
Advisory
Suggest reviewing before use
{
"context": "Tests whether the agent applies this plugin's frontmatter convention for a conditional rule whose prescriptions only fire in specific files. The plugin prescribes: alwaysApply: false in the rule file frontmatter, plus an applyTo: field combining glob patterns (matching common dependency manifests like package.json, pyproject.toml, Cargo.toml, go.mod) with a natural-language clause introduced by an em dash. In the plugin manifest form, scope lives only in the rule file frontmatter — .tessl-plugin/plugin.json lists rule paths and carries no per-rule config. Baseline agents without the plugin typically default to alwaysApply: true on every rule (matching the only convention they've seen) and do not know about the applyTo glob+prose pattern. Even a baseline agent that does scope the rule typically omits the natural-language clause or splits the scope across multiple fields.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Rule file frontmatter declares alwaysApply: false",
"description": "The rules/pin-dependencies.md file has a YAML frontmatter block at the top with alwaysApply: false. Scores zero if frontmatter is missing, if alwaysApply is true, or if alwaysApply is omitted from the block. Baseline agents commonly default to alwaysApply: true",
"max_score": 16
},
{
"name": "Rule file frontmatter declares applyTo with glob patterns",
"description": "The rules/pin-dependencies.md frontmatter contains an applyTo field (or accepted alias: globs, paths) whose value includes glob patterns matching at least two common dependency manifest filenames (e.g., package.json, pyproject.toml, Cargo.toml, go.mod, Gemfile, composer.json). Scores partial credit (12) if applyTo is present but only matches one manifest type; scores zero if the field is missing or contains no glob patterns",
"max_score": 22
},
{
"name": "applyTo value combines globs with a natural-language clause",
"description": "The applyTo value includes both a glob list and a natural-language clause separated by a literal em dash (—, U+2014), e.g., 'package.json, pyproject.toml — when editing dependency manifests'. The clause expresses the action-level scope in prose alongside the file-level glob scope. Scores zero if the value is glob-only with no prose, prose-only with no globs, or uses a different separator (hyphen, double hyphen, en dash) where the rule prescribes an em dash",
"max_score": 20
},
{
"name": "plugin.json rules array includes the new rule path",
"description": "The updated .tessl-plugin/plugin.json rules array includes the path rules/pin-dependencies.md. Scores zero if the path is missing or wrong, or if the agent adds a steering map or a per-rule alwaysApply field to the manifest — that is the legacy tile.json model; plugin.json lists rule paths only and the scope lives in the rule file frontmatter",
"max_score": 16
},
{
"name": "Rule body has H1 title matching the filename concept",
"description": "The rule body starts with an H1 heading like # Pin Dependencies (or close equivalent) matching the filename. Scores zero if there is no H1 or it is unrelated to the file's concept",
"max_score": 10
},
{
"name": "Existing rules and manifest entries are preserved unchanged",
"description": "The two pre-existing rule paths (rules/commit-conventions.md, rules/spaces-not-tabs.md) remain in the plugin.json rules array, and the inputs/rules/spaces-not-tabs.md file is not modified. Scores zero if any pre-existing path is dropped or a pre-existing rule file is altered",
"max_score": 16
}
]
}.tessl-plugin
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10
scenario-11
scenario-12
scenario-13
scenario-14
scenario-15
scenario-16
scenario-17
scenario-18
rules
skills
adopt-fork-pr
eval-curation
install-reviewer