fine-tuning-with-trl

Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

80%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable, executable content with good progressive disclosure to real reference files. The main weakness is workflow_clarity: the multi-step training workflows lack explicit validation/verification checkpoints, which the rubric caps at 3 for batch operations.

Suggestions

Add validation checkpoints to each workflow (e.g. after SFT: 'Verify eval loss decreased before proceeding to reward modeling'; after reward model: 'Check reward model accuracy on held-out preferences').

Trim narrating sentences like 'Optimize policy using reward model:' that merely restate the heading, letting the code speak for itself.

Include a short 'verify training succeeded' step (check logs/loss curves, run a sanity generation) before marking each checklist item complete.

Dimension	Reasoning	Score
Conciseness	Largely efficient code-first content with no padding about basic concepts, but includes some narrating prose ('Train base model on instruction-following data', 'Optimize policy using reward model') and tutorial-style comments that could be trimmed.	4 / 5
Actionability	Fully executable, copy-paste-ready code and CLI commands for SFT, DPO, PPO, GRPO, and reward modeling, with concrete configs and real dataset names covering the common cases.	5 / 5
Workflow Clarity	Three workflows have clear sequences and checklists, but none include validation checkpoints (no 'verify loss decreased / eval before proceeding / check reward model accuracy'), and these batch training runs warrant feedback loops — capping at 3 per the rubric.	3 / 5
Progressive Disclosure	Clear overview body with four well-signaled, one-level-deep references (sft-training, dpo-variants, reward-modeling, online-rl) that are real files, with advanced detail appropriately split out.	5 / 5
	Total	17 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that concretely lists the supported methods and provides an explicit, natural-language trigger clause. Minor grammar ('Use when need RLHF') and missing synonym/extension terms keep trigger_term_quality just below the top anchor.

Suggestions

Soften the slightly ungrammatical 'Use when need RLHF' to 'Use when you need RLHF' or keep third-person 'Use when RLHF is needed'.

Add a couple of common synonyms/file cues (e.g. 'preference tuning', 'RL post-training') to broaden natural trigger coverage.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions across methods — 'SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training' — giving comprehensive coverage of the skill's capabilities.	5 / 5
Completeness	Clearly answers 'what' (fine-tuning methods enumerated) and explicitly answers 'when' via 'Use when need RLHF, align model with preferences, or train from human feedback'.	5 / 5
Trigger Term Quality	Includes natural user-facing phrases ('RLHF', 'align model with preferences', 'train from human feedback') but is missing some common synonyms and any file/extension terms; good but not comprehensive.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (TRL reinforcement-learning fine-tuning) with distinct triggers and minimal overlap risk against generic fine-tuning skills.	5 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: OpenLAIR/dr-claw
Path: skills/post-training/trl-fine-tuning/SKILL.md
Commit: 4b35756

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.