fine-tuning-with-trl

TRL: SFT, DPO, PPO, GRPO, reward modeling for LLM RLHF.

Quality

75%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./optional-skills/mlops/training/trl-fine-tuning/SKILL.md

Quality

Content

77%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with executable code, clear sequenced workflows, and checklists, but it is somewhat verbose with redundant examples and includes a broken template reference that weakens navigation.

Suggestions

Fix or remove the broken reference to templates/basic_grpo_training.py (no templates/ directory exists), since the GRPO production script is promised but missing.

De-duplicate the SFT/DPO setup shown in Quick start versus Workflows 1 and 2 to trim the body.

Move the full per-method training scripts into reference files so SKILL.md stays a lean overview pointing one level deep.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient and code-focused without explaining concepts Claude already knows, but the Quick-start SFT/DPO examples are repeated nearly verbatim inside Workflows 1 and 2, and the ~450-line body could be tightened.	2 / 3
Actionability	Provides fully executable, copy-paste-ready Python (SFTTrainer, DPOTrainer, GRPOTrainer, RewardTrainer), concrete CLI commands (trl dpo / trl grpo / python -m trl.scripts.ppo), and specific configs with real hyperparameters.	3 / 3
Workflow Clarity	Multi-step pipelines (RLHF, DPO, GRPO) are clearly sequenced with copyable checklists, an end-step evaluation checkpoint, and a dedicated 'Common issues' section providing error-recovery guidance.	3 / 3
Progressive Disclosure	Structure is good — an overview with a well-signaled 'Advanced topics' section pointing one level deep to real reference files (sft-training.md, dpo-variants.md, reward-modeling.md, online-rl.md, grpo-training.md) — but templates/basic_grpo_training.py is referenced twice and does not exist, and substantial inline workflow code could be split out.	2 / 3
	Total	10 / 12 Passed

Description

72%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is terse and domain-distinctive with strong trigger keywords, but it lacks an explicit use-when trigger clause and uses a bare-noun telegraphic style instead of action verbs.

Suggestions

Add an explicit 'Use when...' clause, e.g. 'Use when fine-tuning or aligning LLMs with SFT, DPO, PPO, GRPO, or reward modeling (RLHF).'

Reword the method list with action verbs (e.g. 'Run SFT, DPO, PPO, GRPO, and reward modeling for LLM RLHF') to lift specificity to the top anchor.

Include a couple of full-form natural variations (e.g. 'supervised fine-tuning', 'preference alignment') alongside the abbreviations.

Dimension	Reasoning	Score
Specificity	Names the domain (TRL, LLM RLHF) and several concrete methods ("SFT, DPO, PPO, GRPO, reward modeling"), but as a telegraphic noun-list with no action verbs, stopping short of the fully concrete verb-phrase action style of the top anchor.	2 / 3
Completeness	It states what the skill does (post-training methods for LLM RLHF) but has no "Use when..." clause or equivalent explicit trigger guidance, so the "when" is only implied — capping completeness at 2 per the rubric guideline.	2 / 3
Trigger Term Quality	Covers the natural terms practitioners actually say for this domain — "SFT", "DPO", "PPO", "GRPO", "reward modeling", "RLHF", "LLM" — giving good keyword coverage rather than generic or overly abstract jargon.	3 / 3
Distinctiveness Conflict Risk	Names a specific library (TRL) and a distinct set of post-training/RLHF method triggers that are unlikely to fire for unrelated skills, giving it a clear niche.	3 / 3
	Total	10 / 12 Passed

Validation

75%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 12 / 16 Passed

Validation for skill structure

Criteria	Description	Result
metadata_version	'metadata.version' is missing	Warning
metadata_field	'metadata' should map string keys to string values	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
relative_links	Relative link issues: 2 missing	Warning

	Total	12 / 16 Passed

Repository: NousResearch/hermes-agent
Path: optional-skills/mlops/training/trl-fine-tuning/SKILL.md
Commit: 4c9628e

Reviewed: about 14 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.