CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

agent-evaluation

为 Agent 建立或改进评估体系时使用——设计评估指标(Pass@k/Pass^k/成本)、搭建可重复运行的评估环境、选择确定性验证器或 LLM-as-a-Judge 及各自适用边界、编写 Rubric、做失败归因与回归任务、评估驱动的模型选型、判断统计显著性、建设可观测性与内部评估基础设施(消融、AB 测试、特性开关、提示词回归)、从 Benchmark 报告走到系统改进。

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A dense, high-signal methodology skill: nearly every line encodes an actionable heuristic, quantified rule, or named procedure, with gated validation loops in the improvement-cycle sections. Its main weaknesses are that all detail is inlined in one long file and that every referenced bundle path (chapter7/*, book/chapter7.md) is missing from the skill directory, breaking progressive disclosure navigation.

Suggestions

Ship the referenced bundle files (chapter7/* code directories and book/chapter7.md sections) or remove the 配套代码/深度阅读 sections — every reference currently points at a nonexistent path.

Split the ten 实践模式 subsections into a small set of one-level-deep reference files (e.g., metrics-and-statistics.md, failure-attribution.md, model-selection.md), keeping SKILL.md as a lean overview with well-signaled links.

Trim the passages that re-explain knowledge Claude already has (prefill/decode latency mechanics, Goodhart's law, McNemar testing) down to just the skill-specific guidance they frame.

DimensionReasoningScore

Conciseness

The body is densely packed — nearly every line carries a non-obvious heuristic, formula, or quantified example ("Pass@k = 1-(1-p)^k", "100 个用例、成功率 70% 时 95% 置信区间约 ±9 个百分点", "28.3% 与 17.5% 合用只有 30%"), with no padding. It is not a 5 because a few passages explain concepts Claude already knows (the prefill/decode latency model, the Goodhart's-law gloss, McNemar testing), which could be trimmed.

4 / 5

Actionability

For an instruction-only skill the guidance is largely concrete: a structured attribution record with named fields (任务名、首错步号、错误类别、责任方、原文证据、置信度), a named three-rule pre-filter for suspect trajectories, the trajectory-prefix bisection procedure, and specific operational parameters ("每个配置跑 3–5 个随机种子"). It falls short of fully executable because there is no copy-paste-ready code or command, and several items remain at the level of principles ("看 p95 尾部延迟而非均值").

4 / 5

Workflow Clarity

Several sections present clear sequences with explicit validation gates and feedback loops — section 10's "小范围通过只获得扩大测试的资格:标准环境、全任务 × 多种子…完整复测通过前不部署" and the hypothesis-chain loop; section 5 defines acceptable-action sets rather than single answers. Not a 5 because the skill aggregates ten parallel practice areas rather than one coherent end-to-end workflow, and entry-point sequencing across them is implicit.

4 / 5

Progressive Disclosure

The single SKILL.md inlines all ~110 lines of dense practice detail (content that, given the topic breadth, plausibly belongs in separate reference files), and every path it points to — "chapter7/tau2-bench-eval/", "chapter7/agent-cost-analysis/", "book/chapter7.md" — does not exist in the bundle, so the 配套代码/深度阅读 sections lead nowhere. Sections and reference labels are clear, placing this above anchor 2, but the dangling references and inline bulk keep it below the well-organized, working-references level of anchor 4.

3 / 5

Total

15

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it opens with an explicit use-when clause, then enumerates concrete capabilities with named techniques and boundaries, all in third person. Keyword coverage is good though a few natural synonyms (评测/测试集) are absent.

DimensionReasoningScore

Specificity

The description enumerates ten-plus concrete actions — "设计评估指标(Pass@k/Pass^k/成本)", "搭建可重复运行的评估环境", "选择确定性验证器或 LLM-as-a-Judge 及各自适用边界", "编写 Rubric", "做失败归因与回归任务", "判断统计显著性", "建设可观测性与内部评估基础设施(消融、AB 测试、特性开关、提示词回归)" — matching the anchor for multiple specific concrete actions with comprehensive coverage. Score 4 would require minor gaps in coverage, but the action list spans the skill's full surface.

5 / 5

Completeness

Explicitly answers both questions: the leading clause "为 Agent 建立或改进评估体系时使用" is a concrete 'when', and the long action list is a clear 'what'. This matches the anchor for clearly and explicitly answering both with concrete trigger phrases.

5 / 5

Trigger Term Quality

Natural trigger terms are present — "评估体系", "评估指标", "Rubric", "失败归因", "统计显著性", "AB 测试", "Benchmark 报告" — giving good keyword coverage. Missing a few natural synonyms and variations users would say ("评测", "测试集/评估集", "回归测试"), so it sits between anchor 4 and 5, closer to 4.

4 / 5

Distinctiveness Conflict Risk

The niche is unambiguous — building/improving evaluation systems for Agents — and distinctive terms (Pass^k, LLM-as-a-Judge 及适用边界, 失败归因) give it minimal overlap with generic testing or infrastructure skills. Only "AB 测试"/"特性开关" lightly touch adjacent infra topics, which is not enough to drop below the clear-niche anchor.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

bojieli/ai-agent-book

Path

skills/agent-evaluation/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.