CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/model-performance-regression-gate

Computes held-out metrics (accuracy, F1, AUC, RMSE) for a retrained model and compares them against the current production model, failing promotion when any metric regresses beyond a configured tolerance. Adds per-segment checks via Deepchecks WeakSegmentsPerformance so a model that improves globally but regresses on a key slice is still blocked. Use when a retrained model is a candidate for promotion and the CI pipeline must enforce a per-metric pass/fail gate before the artifact is pushed to the model registry.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-built CI-gate skill: fully sequenced steps with runnable code, an appropriate one-level-deep reference split, and useful failure-mode documentation. The main gaps are minor: a couple of trimmable justification sentences, multiclass/regression handled as asides rather than code, and the empty-segment guard left to Limitations instead of the Step 5 code.

Suggestions

Trim the tool-justification prose in Steps 1, 3, and 5 (e.g., 'Deepchecks is the primary framework for segment-level checks' and the 'explicit, inspectable numeric values' rationale) — the code and links already justify the choices (conciseness).

Make the multiclass AUC path concrete: either show the `roc_auc_score(y_true, proba_matrix, average='weighted', multi_class='ovr')` call or state explicitly that the sample code is binary-only and must be adapted (actionability).

Move the empty-segment guard from Limitations into Step 5's code, e.g., fail or warn when `seg_result.value["weak_segments_list"]` is empty, so a silently vacuous segment check cannot pass the gate (workflow_clarity).

DimensionReasoningScore

Conciseness

The ~300-line body is mostly executable code with no explanations of concepts Claude already knows (it never explains what accuracy or F1 is, and the WeakSegmentsPerformance mechanism description is library-specific gating knowledge). A few justification sentences could be trimmed ("Use scikit-learn metric functions directly so the gate has explicit, inspectable numeric values rather than relying on internal scorer defaults", "Deepchecks is the primary framework for segment-level checks"), placing it at the 4 anchor rather than 5's 'every token earns its place'.

4 / 5

Actionability

Every step has concrete, runnable code: pip install, joblib loads, sklearn metric computation, tolerance gate with sys.exit, Deepchecks condition API calls, and a complete GitHub Actions YAML. It is not the 5 anchor because the multiclass case is only name-dropped ("multiclass: average='weighted', multi_class='ovr'") while the code hardcodes binary `predict_proba(X_test)[:, 1]`, and the regression variant is a commented-out fragment rather than ready code — minor gaps requiring user adaptation.

4 / 5

Workflow Clarity

Steps 1-8 are explicitly sequenced, each gate prints per-metric PASS/FAIL and exits non-zero, failed-gate reports are uploaded via `if: always()` for triage, and the anti-patterns table plus limitations section act as checklists. It falls short of the 5 anchor because one validation checkpoint is implicit rather than in the code: the guard "Verify `seg_result.value['weak_segments_list']` is non-empty" lives only in Limitations, not in Step 5's gate code, so a silently-empty segment check can pass without an explicit check.

4 / 5

Progressive Disclosure

The split is appropriate: the runnable gate stays in SKILL.md while tunables and full metric signatures are externalized to references/gate-config-and-metrics.md, which exists and is linked twice with proper markdown links at exactly the point of need (Steps 3 and 8). References are one level deep (verified against the actual bundle), section headers are clear, and external doc links are defined at the bottom. This matches the 5 anchor's clear overview with well-signaled one-level-deep references.

5 / 5

Total

17

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: third-person voice, concrete named metrics and library, explicit what and when clauses, and a clearly delineated niche (promotion gating with pairwise comparison and tolerances). The only weakness is modest synonym coverage in trigger terms.

DimensionReasoningScore

Specificity

The description lists multiple concrete actions with named artifacts: "Computes held-out metrics (accuracy, F1, AUC, RMSE)", "compares them against the current production model", "failing promotion when any metric regresses beyond a configured tolerance", and "Adds per-segment checks via Deepchecks WeakSegmentsPerformance". Coverage is comprehensive and every capability is a specific concrete action, matching the 5 anchor; it is not the 4 anchor because there are no meaningful gaps in the action list.

5 / 5

Completeness

It explicitly answers both questions: the "what" ("Computes held-out metrics... compares them against the current production model, failing promotion when any metric regresses beyond a configured tolerance") and the "when" ("Use when a retrained model is a candidate for promotion and the CI pipeline must enforce a per-metric pass/fail gate before the artifact is pushed to the model registry"). This matches the 5 anchor's requirement of a clear what AND an explicit when with concrete trigger conditions; the when clause is fully explicit, so the 4 anchor's 'could be more specific' caveat does not apply.

5 / 5

Trigger Term Quality

Natural phrases an ML engineer would say are present: "retrained model", "candidate for promotion", "CI pipeline", "pass/fail gate", "model registry", "metric regresses", "key slice". It falls short of the 5 anchor because it lacks synonym variations users might naturally say (e.g., "deploy", "model rollout", "candidate model evaluation", "model comparison"), but it is well above the 3 anchor since the terms present are the natural vocabulary for this task rather than jargon-only.

4 / 5

Distinctiveness Conflict Risk

The niche is sharp: a pairwise candidate-vs-production comparison with per-metric tolerances acting as a promotion gate before registry push. Triggers like "compares them against the current production model", "per-metric pass/fail gate", and "pushed to the model registry" distinguish it from general model-evaluation suites and drift-monitoring skills. Conflict risk is minimal, matching the 5 anchor; it is not the 4 anchor because no closely related skill would capture these triggers.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

On this page