CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

eval-creator-ci

[Beta] CI-only eval regression runner using gh-aw (GitHub Agentic Workflows). Runs all eval cases in .evals/ on a schedule or per-PR, reports pass/fail results, and can block merges on regressions. Also creates new eval cases from promoted patterns flagged by learning-aggregator-ci. Use when: you want automated regression testing of promoted rules in CI/headless pipelines. For interactive eval creation and runs, use eval-creator.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

The canonical home for this skill is eval-creator-ci in pskoett/pskoett-ai-skills

SKILL.md
Quality
Evals
Security

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured CI skill body with concrete commands, a complete output schema, explicit gate policies, and proper use of a single one-level-deep reference file for the workflow template. All four dimensions land at 4 due to minor gaps: unspecified eval-case frontmatter format, no in-run error recovery, and some redundancy across the Purpose/Differences/Integration sections.

Suggestions

Specify the eval case file format (frontmatter fields and an example) inline or in a dedicated reference so create mode is fully executable.

Add a brief error-recovery step for malformed or unreadable eval case files in the run-mode workflow (e.g., record as failed with a parse-error reason).

Trim redundancy between the Purpose section, the "Differences from Interactive Version" table, and the Integration/Data Flow sections.

DimensionReasoningScore

Conciseness

The body is efficient — tables, commands, and a precise YAML output schema with no explanations of concepts Claude already knows. Minor over-explanation could be trimmed ("Per-PR runs catch regressions before merge. Weekly runs catch drift in the harness itself") and the Purpose section plus the "Differences from Interactive Version" table restate overlapping content. Anchor 4 fits: efficient with minor instances of trimming possible, not anchor 3's "some unnecessary explanation" and not anchor 5's every-token-earns-its-place.

4 / 5

Actionability

Mostly executable guidance: exact install commands ("gh skill install pskoett/pskoett-skills eval-creator-ci"), validation ("gh aw compile --actionlint --zizmor"), a copy-paste trigger YAML block, and concrete per-check semantics ("grep-check: Search target files for pattern, compare to expected (found/not_found)"). The gap that keeps it from anchor 5: create mode says "Create the eval case file in .evals/cases/ with proper frontmatter" without ever specifying or referencing the eval case frontmatter format.

4 / 5

Workflow Clarity

Both modes are clearly numbered sequences with explicit checkpoints ("Check preconditions — if not met, mark as skip", "Compare result to expected outcome", "If gate policy is enabled and any eval fails: fail the check run"), so the batch-operation validation cap does not apply. It falls short of anchor 5 because there is no in-run error-recovery loop (e.g., what to do with a malformed eval case file — failures only flow downstream to self-improvement-ci), but it is well above anchor 3's implicit checkpoints.

4 / 5

Progressive Disclosure

The bundle reference references/workflow-example.md exists and holds the bulk gh-aw workflow template, clearly signaled one level deep ("Copy references/workflow-example.md into .github/workflows/eval-creator-ci.md"), with only key snippets (trigger YAML, output schema) kept inline. Minor organization gaps — the Install section lists four redundant fallback command variants and the Data Flow diagram partially duplicates the Integration tables — keep it at anchor 4 rather than 5.

4 / 5

Total

16

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, comprehensive description that clearly states capabilities, an explicit trigger clause, and a boundary against the interactive sibling skill. The only deductions are the second-person voice in the 'Use when' clause and some missing natural keyword synonyms.

Suggestions

Rewrite the trigger clause in third person (e.g., "Use when the user requests automated regression testing of promoted rules in CI/headless pipelines") to avoid the second-person voice penalty.

Add natural trigger synonyms such as "GitHub Actions", "merge gate", or "automated eval runs" to broaden keyword coverage toward the top anchor.

DimensionReasoningScore

Specificity

The description lists multiple concrete, comprehensive actions ("Runs all eval cases in .evals/ on a schedule or per-PR, reports pass/fail results, and can block merges on regressions. Also creates new eval cases from promoted patterns"), matching the anchor-5 example — but the second-person phrasing "Use when: you want automated regression testing..." triggers the rubric's voice penalty of -1. It is well above anchor 3, which covers only 1-2 actions.

4 / 5

Completeness

It explicitly answers both what ("Runs all eval cases in .evals/... reports pass/fail results, and can block merges... Also creates new eval cases") and when ("Use when: you want automated regression testing of promoted rules in CI/headless pipelines"), plus a concrete boundary clause ("For interactive eval creation and runs, use eval-creator"). This matches anchor 5 exactly; anchor 4 would require the 'when' to be less explicit.

5 / 5

Trigger Term Quality

Natural phrases users would say are present: "regression testing", "CI/headless pipelines", "block merges", "per-PR", ".evals", "eval regression runner". It falls short of anchor 5's comprehensive synonym coverage (missing "GitHub Actions", "merge gate", "automated testing") but is clearly better than anchor 3, which expects missing common variations.

4 / 5

Distinctiveness Conflict Risk

Clear niche with distinct triggers: "CI-only", "gh-aw (GitHub Agentic Workflows)", ".evals/", "learning-aggregator-ci", and an explicit disambiguation pointing interactive use to "eval-creator". Minimal conflict risk — matches anchor 5, not anchor 4, since the sibling-skill overlap is explicitly resolved.

5 / 5

Total

18

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

pskoett/pskoett-ai-skills

Path

plugin/skills/eval-creator-ci/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.