CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

eval-dataset-design

设计 Agent 评估数据集或单条评估任务时使用——解剖评估任务的四个组成部分、设计验证器与验收标准(含防敷衍性修复)、划分任务难度与陷阱任务、防范训练/评估数据泄漏(参数化实例、canary)、建设评估集的质量控制与长期维护、选择公开基准/自建业务集/生产轨迹回流三个来源。覆盖 τ²-bench、SWE-bench、AndroidWorld、OSWorld、Terminal-Bench、GAIA 的可迁移设计手法。

Quality

87%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A dense, well-organized design knowledge base with concrete field-level patterns, explicit manual validation loops, and clearly signaled external references. The main gaps are some pattern/anti-pattern repetition between 核心原则/实践模式 and 常见陷阱, no inline copy-paste task template, and reference paths that do not exist in the skill bundle.

Suggestions

Add an inline YAML/JSON skeleton of the four-part task anatomy (ticket / user_scenario / initial_state / evaluation_criteria) so task authors have a copy-paste starting template, since the concrete examples currently live only in external chapter7/ directories.

Frame the 实践模式 sections as an explicit numbered authoring workflow with validation checkpoints (draft four-part task → write validator → hand-execute each task and ask the two diagnostic questions → triage), turning the implicit lifecycle into a checkable sequence.

Fix or relocate the 配套代码/深度阅读 references — chapter7/* and book/chapter7.md do not exist in the skill bundle; move them under references/ or inline the key excerpts so the pointers resolve.

DimensionReasoningScore

Conciseness

Dense, opinionated design guidance carrying curated facts Claude does not already hold ('SWE-bench Verified 从 2294 个原始任务…仅 500 个通过', 'OSWorld…134 个独立评估函数'), assuming competence throughout with no padding. However, 常见陷阱 re-derives points already made in 实践模式/核心原则 ('只验 FAIL_TO_PASS 不验 PASS_TO_PASS', 敷衍性修复, 事实锚定 each appear twice), so minor trimming is possible — anchor 4, not 5.

4 / 5

Actionability

Concrete field-level guidance ('ticket / user_scenario / initial_state / evaluation_criteria', 'known_info 与 task_instructions 的拆分', 'env_assertions 验终态…reward_basis 是聚合规则') plus an explicit manual validation routine ('每道题亲手做一遍。做完问两个问题:…验证器拦得住吗?'). But no copy-paste task or validator template is inline — concrete examples are delegated to external 'chapter7/' directories — so anchor 4 ('mostly executable guidance, minor gaps') rather than 5.

4 / 5

Workflow Clarity

The 实践模式 sections follow a recognizable lifecycle order (task anatomy → validators → difficulty → leak prevention → QA/maintenance → sources) and include explicit checkpoints ('每道题亲手做一遍', the SWE-bench five-criteria human triage, the OSWorld four-class repair taxonomy). But the sequence is implicit — never framed as a numbered authoring workflow with validate-fix-retry loops — so anchor 4 rather than 5; not 3, since checkpoints are present and named rather than missing.

4 / 5

Progressive Disclosure

The body appropriately keeps principles/patterns in SKILL.md and signals one-level-deep references in dedicated sections ('配套代码' with three chapter7 dirs each described; '深度阅读' with book/chapter7.md). However, no references/scripts/assets bundle exists and none of the cited paths (chapter7/*, book/chapter7.md) resolve inside the skill directory — references are clear but point at unverifiable targets, so anchor 4 ('minor organization gaps') rather than 5.

4 / 5

Total

16

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: an explicit use-when clause plus a comprehensive list of concrete design actions and distinctive triggers including six benchmark names. The only minor gap is a few missing natural synonyms (评测/出题/防作弊) for how users might phrase the need.

DimensionReasoningScore

Specificity

Lists six concrete actions — '解剖评估任务的四个组成部分、设计验证器与验收标准(含防敷衍性修复)', '划分任务难度与陷阱任务', '防范训练/评估数据泄漏(参数化实例、canary)', '建设评估集的质量控制与长期维护', '选择公开基准/自建业务集/生产轨迹回流三个来源' — comprehensively covering the design-to-maintenance lifecycle. Matches the anchor for multiple specific concrete actions with comprehensive coverage, not 4 (which allows only minor gaps).

5 / 5

Completeness

'设计 Agent 评估数据集或单条评估任务时使用' is an explicit when-clause with concrete trigger phrases, and the dash-separated action list clearly states what the skill does. Both what and when are explicitly present, matching anchor 5 rather than 4 (where the 'when' would be less specific).

5 / 5

Trigger Term Quality

Good natural keywords — '评估数据集', '评估任务', '验证器', '验收标准', '数据泄漏', 'canary' plus six benchmark names (τ²-bench, SWE-bench, AndroidWorld, OSWorld, Terminal-Bench, GAIA) — but common variations users might say ('评测集', '出题/写评估题', '防作弊', 'eval harness') are absent. Fits anchor 4 ('good keyword coverage; a few natural terms missing'), not 5, which requires synonym-level coverage.

4 / 5

Distinctiveness Conflict Risk

'设计…评估数据集' carves a clear design-time niche distinct from running or analyzing evals, and distinctive triggers (validator design, canary, 防泄漏, named benchmarks) make wrong-skill triggering unlikely. Clear niche with minimal conflict risk → anchor 5, not 4, since overlap with generic model-evaluation skills is minimal given the explicit design scope.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

bojieli/ai-agent-book

Path

skills/eval-dataset-design/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.