CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

reward-design

为 RL 训练设计奖励函数或排查奖励黑客时使用——决定奖励来自规则、人类偏好还是模型评判,结果奖励还是过程奖励,标量、向量还是生成式诊断,以及如何用路径约束与 RLVP 惩罚违规动作;也用于多轮 Agent 的信用分配、隐藏测试与早停判定设计。触发词:奖励设计、reward hacking、reward seeking、RLVR、RLHF、GRM、ORM、PRM、RLVP、信用分配、隐藏测试、过程奖励、奖励塑形、组内差异。

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, dense instruction skill: concrete design rules, real numbers, explicit validation checkpoints, and a clear decision framework, with only minor duplication and a placeholder-level code sketch. Its main structural weakness is that all companion-code and deep-reading pointers reference files that do not exist in the provided bundle, leaving the progressive-disclosure layer unresolvable.

Suggestions

Ship the referenced bundle files (or place them under references/) so paths like chapter8/RLVP/ and book/chapter8.md actually resolve; if they cannot be shipped, replace the pointers with brief inline summaries of the key results.

Deduplicate the reward-seeking/early-stop discussion, which currently appears in 核心原则, 实践模式2, and 常见陷阱 — consolidate it into one section and cross-reference the pitfalls list.

Make the reward-merging snippet concrete by showing one real deterministic_path_signal example (e.g., a specific forbidden-action check such as skipped authentication or destructive shell commands) so the pattern is closer to executable.

DimensionReasoningScore

Conciseness

The body is dense with judgment-carrying content ('惩罚是通常可达的那一半,进展奖励是受可达性门控的那一半', concrete numbers like '3.71→0.66' and '从 7.0 降至 4.4') rather than padded explanation. However, the reward-seeking/early-stop behavior is explained repeatedly across 核心原则, 实践模式2, and 常见陷阱, and PPO/GRPO credit-assignment mechanics appear in both 核心原则 and 实践模式5 — minor duplication that could be trimmed, matching the 'efficient with minor over-explanation' anchor rather than the flawless one.

4 / 5

Actionability

Guidance is mostly concrete and executable in intent: '通过 +1、不通过 −1', '计算策略梯度时必须屏蔽这些反馈 token', '两路各自归一化后再合并', plus four numbered RLVP design rules. The reward-merging code sketch relies on placeholder functions (verify_final_state, deterministic_path_signal, normalize) with no worked example of a single real path rule, so it falls short of fully copy-paste-ready — the 'mostly executable with minor gaps' anchor.

4 / 5

Workflow Clarity

There is a clear ordered engineering path ('先用结果奖励建立可靠基线,再只为真正可验证的中间事件加入过程信号') with explicit validation checkpoints ('先用少量 rollout 检查"任务是否可完成、验证器能否区分对错",再扩大采样规模'; '先测信号可达性,再决定是否增加奖励维度'). It falls short of the top anchor because there is no explicit error-recovery loop describing what to change when the rollout checks fail.

4 / 5

Progressive Disclosure

The body itself is well-sectioned with clearly annotated pointers (配套代码 and 深度阅读 entries each describe their target's content), but every referenced path (chapter8/RLVP/, chapter8/premature-completion-dpo/, book/chapter8.md, …) is absent from the bundle — no references/, scripts/, assets/, or any other files exist — so the promised deeper layer dangles and all substantive guidance is inlined in SKILL.md. This sits above the 'references not clearly signaled' midpoint but below good structure, since references that cannot resolve are more than a minor organization gap.

3 / 5

Total

15

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: it states concrete capabilities across all key design dimensions, gives an explicit use-when clause, and appends a comprehensive bilingual trigger-word list with synonyms. Both what and when are answered explicitly, and the niche is clearly distinct from other skills.

DimensionReasoningScore

Specificity

The description lists multiple concrete actions — '为 RL 训练设计奖励函数或排查奖励黑客', '决定奖励来自规则、人类偏好还是模型评判', '结果奖励还是过程奖励', '标量、向量还是生成式诊断', '信用分配、隐藏测试与早停判定设计' — comprehensively covering source, timing, representation, path constraints, and credit assignment. This matches the anchor for comprehensive coverage of specific concrete actions rather than the anchor with minor gaps.

5 / 5

Completeness

It explicitly answers 'what' (design reward functions, decide source/timing/representation, apply RLVP path constraints, credit assignment, hidden tests, early-stop determination) and 'when' ('为 RL 训练设计奖励函数或排查奖励黑客时使用' plus a dedicated trigger-word list). Both what and when are explicit with concrete trigger phrases, matching the top anchor.

5 / 5

Trigger Term Quality

The explicit trigger list '奖励设计、reward hacking、reward seeking、RLVR、RLHF、GRM、ORM、PRM、RLVP、信用分配、隐藏测试、过程奖励、奖励塑形、组内差异' covers the natural phrases an RL practitioner would actually say, including synonyms in both Chinese and English (过程奖励/PRM, reward hacking/reward seeking). Comprehensive coverage with synonyms matches the top anchor; no meaningful synonym group is missing to justify a 4.

5 / 5

Distinctiveness Conflict Risk

The description occupies a clear niche (RL reward design) with distinctive triggers such as RLVP, reward seeking, 奖励塑形, and 组内差异 that would not naturally fire for unrelated skills. Minimal conflict risk matches the top anchor.

5 / 5

Total

20

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

bojieli/ai-agent-book

Path

skills/reward-design/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.