CtrlK
BlogDocsLog inGet started
Tessl Logo

jbaruch/coding-policy

General-purpose coding policy for Baruch's AI agents

90

1.30x
Quality

91%

Does it follow best practices?

Impact

90%

1.30x

Average score across 18 eval scenarios

SecuritybySnyk

Advisory

Suggest reviewing before use

Overview
Quality
Evals
Security
Files

criteria.jsonevals/scenario-16/

{
  "context": "Tests whether the agent correctly resists path-scoping a rule whose prescriptions span multiple contexts. The rule has two halves: (a) 'prefer stdlib over third-party' fires when writing code, (b) 'pin if you add it' fires when editing dependency manifests. Path-scoping the rule to manifests only (a tempting move because the sibling rule pin-dependencies.md is already scoped that way) would silently drop half (a) — the stdlib-first guidance would no longer fire when the agent is writing code. In the plugin manifest form, scope lives only in the rule file frontmatter; .tessl-plugin/plugin.json lists rule paths and carries no per-rule config. The plugin prescribes keeping mixed-scope rules universal (alwaysApply: true, no applyTo) so every prescription continues to fire in its native context. Baseline agents may either (1) default everything to alwaysApply: true and miss the deliberate decision, or (2) copy the pin-dependencies pattern and over-scope to manifests, which is the failure mode this rule guards against.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Rule file frontmatter declares alwaysApply: true",
      "description": "The rules/stdlib-first.md file has a YAML frontmatter block with alwaysApply: true. The rule must stay universal because part (a) fires when writing code, not when editing manifests. Setting alwaysApply: false here (matching the pin-dependencies pattern) is the failure this scenario tests for — it would scope the rule to manifests only and silently drop the stdlib-first guidance when the agent is writing code",
      "max_score": 24
    },
    {
      "name": "Rule file frontmatter declares no scoping fields",
      "description": "The rules/stdlib-first.md frontmatter does NOT contain applyTo, globs, or paths fields. Any scoping field narrows the rule's activation; with mixed-scope content, scoping to one context drops the other. The plugin prescribes omitting scoping fields when the rule's prescriptions span multiple contexts",
      "max_score": 24
    },
    {
      "name": "plugin.json rules array includes the new rule path",
      "description": "The updated .tessl-plugin/plugin.json rules array includes the path rules/stdlib-first.md. Scores zero if the path is missing or wrong, or if the agent adds a steering map or a per-rule alwaysApply field to the manifest — that is the legacy tile.json model; plugin.json lists rule paths only",
      "max_score": 14
    },
    {
      "name": "Rule body covers the stdlib-first practice",
      "description": "The rule body has at least one bullet covering the 'prefer stdlib over third-party' practice — this is the half that fires when writing code. Scores zero if the body only covers pinning and omits the stdlib-first guidance",
      "max_score": 10
    },
    {
      "name": "Rule body covers the dependency-pinning practice",
      "description": "The rule body has at least one bullet covering the 'pin if you add it' practice — the manifest-level half. Scores zero if the body only covers stdlib-first and omits the pinning guidance",
      "max_score": 10
    },
    {
      "name": "Existing rules and manifest entries are preserved unchanged",
      "description": "The three pre-existing rule paths (rules/commit-conventions.md, rules/spaces-not-tabs.md, rules/pin-dependencies.md) remain in the plugin.json rules array, and the inputs/rules/pin-dependencies.md file is not modified (its alwaysApply: false and applyTo must not be flipped or removed). Scores zero if any pre-existing path is dropped or the sibling rule file is altered",
      "max_score": 18
    }
  ]
}

README.md

tile.json