CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

agent-platform-autorater-customization

Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel).

Quality

92%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-engineered process skill: sequenced workflow with genuine validation and approval checkpoints, real one-level-deep bundle references, and concrete commands and SDK type mappings. The residual weaknesses are mild — a few sentences that could be trimmed and script invocations that could be shown inline instead of deferred to --help.

Suggestions

Trim the intro paragraph and the longest multi-clause sentences in Steps 2, 5, and 6 (conciseness).

Show one example invocation of scripts/run_metrics.py and scripts/build_review_page.py inline, including the --threshold flag, instead of relying only on --help (actionability).

In the Scripts section, add a one-line usage example per script so the common cases are copy-paste ready without opening --help (actionability).

DimensionReasoningScore

Conciseness

The body is dense and directive — e.g., "Keep the quotes; unquoted, the shell reads `>=` as a redirect" is a useful gotcha, not padding — and it avoids explaining concepts Claude already knows. It is not a 5 because several passages could still be tightened (the intro paragraph and some multi-clause sentences in Steps 2, 5, and 6 carry trimming candidates).

4 / 5

Actionability

Guidance is mostly executable: a copy-paste install command with version pin ("pip install --upgrade 'google-cloud-aiplatform[evaluation]>=2.8.0,<3'"), concrete gcloud commands, a decision table mapping criteria to specific SDK types ("types.CodeExecutionMetric", "types.StepBasedLLMMetric"), named client methods, and script entry points with a documented --threshold flag. Minor gaps keep it below 5: exact invocations of run_metrics.py / build_review_page.py are deferred to "--help" rather than shown.

4 / 5

Workflow Clarity

Seven steps are clearly sequenced with explicit validation checkpoints and feedback loops: Approval gates ("Wait for a clear yes; silence is not approval"), spend accounting with re-approval before extra runs, "Measure judge noise before trusting a change", held-back cases with overfitting reporting ("if a held-back case breaks, report overfitting"), and "Every number you report comes from a result file". This matches the anchor: clear sequence, explicit validation, error-recovery loops, and checklists.

5 / 5

Progressive Disclosure

The body is a well-sectioned overview-plus-workflow with one-level-deep references that all exist (references/attribution.md, converting_trajectories.md, metric_templates.md, writing_checks.md, multi_turn.md, noise_and_agreement.md, sdk_gotchas.md) plus four real scripts, each clearly signaled at its point of use and indexed in a References section. Large material (the 24KB metric_templates.md) is correctly externalized rather than inlined.

5 / 5

Total

18

/

20

Passed

Description

95%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete actions, natural trigger phrasings with synonyms, explicit use/don't-use boundaries, and clear separation from the sibling eval-flywheel skill. The only weakness is slight incompleteness in the action inventory (metric registration and agreement measurement are not surfaced).

DimensionReasoningScore

Specificity

The description lists several concrete actions — "Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories", "turn failing trajectories into rubrics", "agree with human labels" — which goes beyond naming the domain. It stops short of anchor 5 because metric building/registration and agreement measurement are not mentioned, leaving minor gaps in coverage.

4 / 5

Completeness

Both questions are answered explicitly: the what ("Customizes an autorater ... so its grades match the user's own judgment") and the when ("Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels"), with concrete trigger phrases and an explicit negative scope ("Don't use for quick evals with built-in metrics or for improving the agent itself"). This clearly matches the anchor-5 example structure.

5 / 5

Trigger Term Quality

It includes four natural user phrasings ("wants a metric tailored to their agent", "says an existing autorater grades their agent wrong", "wants to turn failing trajectories into rubrics", "wants an autorater to agree with human labels") plus the technical synonyms users would plausibly use: "autorater hillclimbing", "judge calibration", "evaluation metric". Coverage of natural terms including synonyms is comprehensive for this niche.

5 / 5

Distinctiveness Conflict Risk

The skill occupies a clear niche (autorater/judge calibration) and explicitly disambiguates from the closest sibling: "Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platform-eval-flywheel)". Triggers are distinct and conflict risk is minimal.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

google/skills

Path

skills/cloud/agent-platform-autorater-customization/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.