fine-tuning-with-trl

Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.

Quality

79%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/ml-training/trl-fine-tuning/SKILL.md

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with executable code and CLI commands for every method, and it uses progressive disclosure well by offloading advanced topics to four real reference files. Its main weakness is conciseness (some redundant snippets and over-commented code) and the absence of explicit validation checkpoints in the multi-step RLHF workflow.

Suggestions

Trim redundant inline comments and remove quick-start snippets that duplicate the full workflow code to tighten token efficiency.

Add explicit validation/evaluation checkpoints in Workflow 1 (e.g. verify SFT loss converges and run a quick eval before training the reward model; sanity-check reward model accuracy before PPO).

Collapse the four near-identical 'Copy this checklist' blocks or move them to a reference, since they add length without proportional clarity.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient with copy-paste code, but it includes some over-explanation (e.g. '# Favor longer responses', '# Single reward score', restating what each method does) and redundant quick-start snippets that overlap the workflow sections, keeping it from being fully lean.	3 / 5
Actionability	Provides fully executable, copy-paste-ready code and CLI commands across SFT, DPO, GRPO, reward modeling and PPO, with concrete model names, dataset paths, and hyperparameters covering the common cases.	5 / 5
Workflow Clarity	Workflows are clearly sequenced with per-workflow checklists and labeled steps; however the RLHF pipeline lacks explicit validation checkpoints (e.g. eval/loss checks before moving SFT -> reward model -> PPO), and the rubric flags missing validation for batch/destructive training operations, capping it below 5.	4 / 5
Progressive Disclosure	Clear overview structure with well-signaled, one-level-deep references to four real files in references/ (sft-training, dpo-variants, reward-modeling, online-rl) that are verified to exist; content is appropriately split between the overview and the reference bundles.	5 / 5
	Total	17 / 20 Passed

Description

80%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific and comprehensive, naming concrete TRL methods (SFT, DPO, PPO/GRPO, reward modeling) and providing an explicit 'Use when' trigger clause covering RLHF and preference alignment. It is slightly held back from a top score by trigger-term exhaustiveness and the terseness/roughness of the 'when' phrasing.

Suggestions

Add natural synonyms and file/extension triggers to the 'Use when' clause (e.g. 'preference pairs', 'chosen/rejected data', 'reward model', 'PPO/GRPO training') to broaden keyword coverage.

Tighten the 'when' phrasing for clarity and grammar, e.g. 'Use when the user needs RLHF, wants to align a model with preferences, or is training from human feedback.'

Make the third-person voice consistent and explicit about the package scope so it is unambiguously distinguishable from generic HuggingFace fine-tuning skills.

Dimension	Reasoning	Score
Specificity	Lists multiple specific concrete actions - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training - giving comprehensive coverage of TRL's methods.	5 / 5
Completeness	Clearly states both what (fine-tuning methods) and when ('Use when need RLHF, align model with preferences, or train from human feedback'), though the 'when' clause has minor grammatical roughness that slightly weakens its explicitness.	4 / 5
Trigger Term Quality	Includes natural user phrases like 'RLHF', 'align model with preferences', 'train from human feedback' but is missing a few common synonyms or extension-based triggers; good coverage but not exhaustive.	4 / 5
Distinctiveness Conflict Risk	Targets a clear niche (TRL-based RLHF/post-training) with distinct triggers; minor overlap risk with general HuggingFace fine-tuning skills, but the RL/preference-alignment framing is mostly distinct.	4 / 5
	Total	17 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/trl-fine-tuning/SKILL.md
Commit: fcd2e11

Reviewed: 2 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.