fine-tuning-with-trl

Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with a clean progressive-disclosure structure pointing to real reference files. It loses points for mild Quick-start/workflow redundancy and for lacking explicit validation checkpoints in the long-running training workflows.

Suggestions

Deduplicate the Quick-start SFT/DPO snippets against Workflow 1 Step 1 and Workflow 2 Step 3, or have the Quick start point forward to the detailed workflows instead of repeating the code.

Add validation/feedback checkpoints to each training workflow (e.g., confirm loss decreases and run an eval split before saving/proceeding to the next stage), so Workflow clarity can reach 3.

Tighten the "Use when need RLHF" phrasing in the description and the redundant config blocks in "Common issues" to recover conciseness.

Dimension	Reasoning	Score
Conciseness	The body is code-forward and largely avoids explaining concepts Claude knows, but the Quick-start SFT/DPO snippets are restated nearly verbatim inside Workflow 1/2, so not every token earns its place and it could be tightened.	2 / 3
Actionability	Throughout the skill provides fully executable Python, CLI commands (`python -m trl.scripts.ppo`, `trl dpo`), concrete configs, and copy-paste-ready examples rather than vague direction.	3 / 3
Workflow Clarity	Workflows are sequenced with task checklists, but training is a batch operation and there are no validation/feedback checkpoints (e.g., verify loss/eval before proceeding), which caps this at 2 per the rubric.	2 / 3
Progressive Disclosure	An "Advanced topics" section cleanly signals one-level-deep references to sft-training.md, dpo-variants.md, reward-modeling.md, and online-rl.md, all of which exist as real bundle files, with the body acting as overview.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific, trigger-rich, and explicitly answers both what the skill does and when to use it, with a clear distinct niche. Minor grammatical roughness ("Use when need RLHF") does not materially reduce quality.

Dimension	Reasoning	Score
Specificity	"SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training" lists multiple specific concrete actions rather than vague language, matching the top anchor.	3 / 3
Completeness	It states what the skill does (fine-tune via SFT/DPO/PPO/GRPO/reward modeling) and gives an explicit "Use when..." trigger clause, clearly answering both what and when.	3 / 3
Trigger Term Quality	"Use when need RLHF, align model with preferences, or train from human feedback" plus "HuggingFace Transformers" covers natural terms a user would say, with good variation; not merely technical jargon.	3 / 3
Distinctiveness Conflict Risk	The TRL reinforcement-learning fine-tuning niche with RLHF/preference-alignment triggers is a clear, distinct scope unlikely to fire for unrelated skills.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/trl-fine-tuning/SKILL.md
Commit: 52845c3

Reviewed: about 9 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.