fine-tuning-with-trl

Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.

Quality

87%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

86%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable reference: executable code for every TRL method, checklist-driven workflows, and a clean progressive-disclosure split into four real, one-level-deep reference files. The main improvement opportunities are de-duplicating the Quick start against the workflows and adding explicit mid-pipeline validation checkpoints (e.g., verify reward model before PPO).

Suggestions

Trim the Quick start's SFT and DPO code blocks, which are duplicated in full in Workflows 1-2; keep the quick start to a minimal one-liner per method.

Add an explicit validation checkpoint in Workflow 1 before PPO (e.g., verify reward model accuracy on held-out preference pairs), since PPO quality silently depends on it.

Merge the 'Use TRL when' bullets with the 'Method selection' list to remove redundancy.

Dimension	Reasoning	Score
Conciseness	The body is code-dense with almost no concept padding (it never explains what RLHF or transformers are), but the Quick start duplicates SFT and DPO code that reappears in full in Workflows 1-2, and the 'Use TRL when' bullets partially duplicate 'Method selection'. This matches 'efficient with minor instances that could be trimmed' rather than the every-token-earns-its-place anchor.	4 / 5
Actionability	Code throughout is fully executable and copy-paste ready with concrete hyperparameters, real dataset names, and CLI alternatives ('trl dpo', 'trl grpo', 'python -m trl.scripts.ppo') covering the common cases for every method. Specific examples cover SFT, DPO, PPO, GRPO, and reward modeling.	5 / 5
Workflow Clarity	Each workflow has a copyable checklist and clearly sequenced numbered steps, and Workflows 1-2 end with runnable Evaluate steps. It falls short of the top anchor because there are no mid-training validation checkpoints or explicit feedback loops (e.g., verifying reward model accuracy before starting PPO), though the Common issues section partially compensates.	4 / 5
Progressive Disclosure	The body is an overview with well-signaled, bold-labeled, one-level-deep references, and all four linked files (references/sft-training.md, dpo-variants.md, reward-modeling.md, online-rl.md) exist with no nested references. Advanced content is appropriately split out, matching the clear-overview top anchor.	5 / 5
	Total	18 / 20 Passed

Description

88%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that concretely enumerates all four TRL methods with their purposes and pairs them with an explicit 'Use when' trigger clause. The main weakness is a few missing natural trigger synonyms and slight overlap risk on the SFT/instruction-tuning trigger.

Suggestions

Add natural trigger synonyms users commonly say, e.g. 'preference tuning', 'instruction tuning', or 'train a reward model'.

Fix the grammatical stumble 'Use when need RLHF' to 'Use when you need RLHF' so triggers read naturally.

Clarify the SFT trigger so it doesn't collide with generic fine-tuning skills, e.g. 'SFT for RLHF-stage instruction tuning'.

Dimension	Reasoning	Score
Specificity	The description lists multiple specific concrete actions mapped to purposes ('SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training') with comprehensive coverage of the library's methods. It fits the comprehensive anchor with no coverage gaps.	5 / 5
Completeness	Explicitly answers both what ('Fine-tune LLMs using reinforcement learning with TRL - SFT... DPO... PPO/GRPO... reward model training') and when ('Use when need RLHF, align model with preferences, or train from human feedback') with concrete trigger phrases. The when-clause is explicit with multiple triggers, matching the top anchor.	5 / 5
Trigger Term Quality	Good natural keyword coverage ('RLHF', 'align model with preferences', 'train from human feedback', 'preference alignment'), matching the 'good coverage, a few natural terms missing' anchor. Common user phrasings like 'preference tuning', 'instruction tune', or 'LoRA/PEFT' are absent.	4 / 5
Distinctiveness Conflict Risk	A clear RLHF/TRL niche with distinct triggers, but minor overlap risk with closely related skills since 'instruction tuning' via SFT is also a trigger for generic fine-tuning skills. It is mostly distinct, matching the 4 anchor rather than the minimal-conflict 5 anchor.	4 / 5
	Total	18 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
metadata_version	'metadata.version' is missing	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	14 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/trl-fine-tuning/SKILL.md
Commit: 4082a2e

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.