CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

bad-case-to-dpo

把生产 bad case / 失败轨迹转成训练数据时使用——从失败归因、轨迹前缀截取、DPO 偏好对构造、LoRA 训练到边界集与保留集双集验证的完整链路;也用于判断某类问题该走 SFT、DPO、RL 还是先修 Harness/工具/tokenizer,以及如何从运行轨迹中提取带证据的结构化学习信号。触发词:bad case、失败归因、DPO、偏好对、轨迹前缀、拒绝采样、过度矫正、边界集、保留集、过早结束、reward hacking、LoRA 微调、训练评估隔离。

Quality

87%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

70%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body delivers an unusually rigorous, well-sequenced workflow with explicit dual-set validation gates and concrete commands, and its decision table (material → training usage) is a genuine strength. Its weaknesses are redundancy between the principles and pitfalls sections, reference-grade experiment detail inlined in SKILL.md, and external pointers with no bundled reference files to back them.

Suggestions

Merge the 常见陷阱 section into the corresponding 核心原则 entries (or keep pitfalls as one-line pointers) — roughly eight pitfalls currently restate principles verbatim, costing tokens without adding new guidance.

Move the 案例参照 experiment details (the 8-17 numbers, free-generation diagnostics, hyperparameters) into a references/ file such as case-study.md, keeping only the headline lesson in SKILL.md, since the rubric expects reference-grade detail to live one level deep.

Add one concrete inline example of the key artifact — a minimal preference-pair or trajectory-prefix sample — so the data-construction step is reproducible without opening the external chapter8/ code, and bundle the referenced code paths (or note explicitly that they live in the parent book repo) so the 深度阅读 pointers resolve from the skill itself.

DimensionReasoningScore

Conciseness

The body is dense and mostly operational, but the 常见陷阱 (pitfalls) section restates roughly eight points already made verbatim in 核心原则 (overcorrection, train/eval isolation, model self-report as completion signal, tokenizer audit, DPO mixing with general data), and the 案例参照 section carries full experimental result tables (3/12→11/12, 6/8→0/8, hyperparameters) that read as reference material. This fits 'Mostly efficient but includes some unnecessary explanation or could be tightened' rather than level 4, where over-explanation would be only minor.

3 / 5

Actionability

Concrete, runnable commands are given for the companion code (python demo.py, python -m pytest -q test_pipeline.py, python build_preference_data.py --teacher, python train_dpo.py --smoke, python evaluate.py --decision-score), along with reference hyperparameters (Qwen2.5-7B-Instruct + bf16 LoRA, 4 epochs, lr 3e-5, ~24GB) and exact metric definitions (过度矫正率 = 1 − 正常收尾率). It stops short of level 5 because no concrete artifact examples are shown inline — e.g., a sample preference-pair JSON or a truncated trajectory-prefix record — so a fresh user cannot fully reproduce the data construction without opening the external chapter8/ code.

4 / 5

Workflow Clarity

The pipeline is a clearly numbered ①–⑦ sequence (collect/attribute → triage whether to train → choose training form via table → extract prefixes and build preference pairs → LoRA DPO → dual-set validation → record/receipt) with explicit validation checkpoints: step ⑥ defines which metrics must fall, hold, or stay low after training, with the explicit expected direction for each; isolation rules are enforced by unit tests; tokenizer round-trip audit gates training. This matches 'Clear sequence with explicit validation steps; feedback loops for error recovery'.

5 / 5

Progressive Disclosure

The body has clear sections and dedicated 配套代码 / 深度阅读 navigation sections, but no bundle files exist (no references/, scripts/, assets/) — the chapter8/ and book/ paths point outside the skill, so navigation cannot be satisfied from the bundle itself. Meanwhile, the case-study experiment results, per-case handling details, and the duplicated pitfalls content are inlined in SKILL.md where they belong in separate reference files, fitting 'Some structure but could be better organized ... content that should be separate is inline' rather than level 4's 'most content is appropriately placed'.

3 / 5

Total

15

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is exemplary: it states a complete, concrete capability chain, gives an explicit usage condition, and supplies a rich bilingual trigger-word list that closely mirrors natural user phrasing. Voice is appropriately impersonal and there is no vague padding.

DimensionReasoningScore

Specificity

The description enumerates multiple concrete actions spanning the full pipeline: 失败归因 (failure attribution), 轨迹前缀截取 (trajectory prefix truncation), DPO 偏好对构造 (preference pair construction), LoRA 训练, 边界集与保留集双集验证 (boundary/retention dual-set validation), plus routing decisions between SFT/DPO/RL vs. fixing Harness/工具/tokenizer, and extracting evidence-backed learning signals from trajectories. This matches the anchor 'Lists multiple specific concrete actions; comprehensive coverage' — no generic filler like 'helps with training data'.

5 / 5

Completeness

Both 'what' and 'when' are explicit. What: the complete chain from attribution through prefix extraction, preference pair construction, LoRA training, to dual-set validation. When: '把生产 bad case / 失败轨迹转成训练数据时使用' plus the explicit trigger-word clause. This matches the top anchor 'Clearly and explicitly answers both what AND when with concrete trigger phrases'; the 'Use when...' guidance is present as required.

5 / 5

Trigger Term Quality

An explicit 触发词 (trigger word) list provides comprehensive natural-term coverage with both Chinese and English synonyms: bad case, 失败归因, DPO, 偏好对, 轨迹前缀, 拒绝采样, 过度矫正, 边界集, 保留集, 过早结束, reward hacking, LoRA 微调, 训练评估隔离. These are exactly the phrases a user working on post-training failure cases would say, matching the anchor for comprehensive coverage including synonyms.

5 / 5

Distinctiveness Conflict Risk

This is a clear niche (converting production failure cases into post-training data) with highly distinct triggers — DPO preference pairs, boundary/retention sets, reward hacking, premature completion — that no adjacent skill (general fine-tuning, SFT data synthesis, evaluation harness) would naturally claim. It matches 'Clear niche with distinct triggers; minimal conflict risk'.

5 / 5

Total

20

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

bojieli/ai-agent-book

Path

skills/bad-case-to-dpo/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.