simpo-training

Simple Preference Optimization for LLM alignment. Reference-free alternative to DPO with better performance (+6.4 points on AlpacaEval 2.0). No reference model needed, more efficient than DPO. Use for preference alignment when want simpler, faster training than DPO/PPO.

Quality

72%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/ml-training/simpo/SKILL.md

Quality

Content

70%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with copy-paste-ready configs and commands, and progressive disclosure is clean (overview + three real one-level references). It is held back by minor verbosity (repeating the frontmatter summary and overlapping algorithm-comparison lists) and by the absence of validation/evaluation checkpoints for a long-running training process.

Suggestions

Add explicit verification checkpoints to the training workflow, e.g. 'After step N, check `wandb`/logs that loss is decreasing and run `lm-eval`/AlpacaEval on the output checkpoint before declaring success'.

Remove the redundant one-line intro that repeats the frontmatter description, and merge the overlapping 'Use SimPO when' and 'Algorithm selection' bullets into a single decision table.

Move the model-size-specific VRAM/batch recommendations or the three full YAML configs into a reference file to keep the overview leaner, since hyperparameters.md already covers tuning.

Dimension	Reasoning	Score
Conciseness	The body is mostly tight and reference-rich, but it restates concepts Claude already knows (e.g. 'SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model' repeats the frontmatter) and the 'When to use vs alternatives' / 'Algorithm selection' lists overlap. It is efficient but could be trimmed further.	2 / 3
Actionability	It provides fully executable copy-paste-ready bash commands and complete YAML configs with inline comments (e.g. 'beta: 2.0 # Reward scaling (2.0-10.0)', 'learning_rate: 5e-7 # Critical: 3e-7 to 1e-6'). Three distinct workflows and a concrete issue-fix section give specific, runnable guidance.	3 / 3
Workflow Clarity	Workflows are clearly labeled and sequenced (install → config → launch), and the issue section maps symptoms to fixes. However training is a long-running, resource-intensive operation with no validation/verification checkpoints (e.g. how to check loss is converging, how to evaluate the trained model, how to verify output checkpoints), so the cap-at-2 guideline for missing validation in batch/expensive operations applies.	2 / 3
Progressive Disclosure	The SKILL.md is a concise overview that offloads depth to three real, one-level-deep references — [references/loss-functions.md], [references/hyperparameters.md], [references/datasets.md] — all of which exist in ./references/. Navigation is clearly signaled in the 'Advanced topics' section and the split is appropriate.	3 / 3
	Total	10 / 12 Passed

Description

75%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is well-structured: it states what SimPO is, gives an explicit use-when clause, and distinguishes it from DPO/PPO. The main weakness is specificity — it leans on benefit claims rather than enumerating concrete actions, which keeps it at the 2-level for both specificity and trigger coverage.

Suggestions

Replace benefit phrasing ('better performance (+6.4 points)', 'more efficient') with concrete actions the skill performs, e.g. 'Trains preference-optimized models from chosen/rejected pairs, configures beta/gamma loss, launches DeepSpeed ZeRO-3 training runs'.

Expand trigger terms to include common variations users actually say: 'RLHF', 'reward model', 'fine-tuning', 'instruction tuning', 'preference data'.

Tighten the 'Use for' clause grammar ('when want' → 'when you want') for a cleaner trigger phrase.

Dimension	Reasoning	Score
Specificity	It names the domain (SimPO/LLM alignment) and concrete actions (preference alignment, training), but the action list is thin — 'better performance (+6.4 points)' and 'more efficient' describe benefits rather than concrete capabilities. It does not enumerate multiple specific actions the way a score-3 anchor ('Extract text and tables, fill forms, merge documents') does.	2 / 3
Completeness	It answers 'what' (reference-free alternative to DPO for preference alignment) and 'when' explicitly via 'Use for preference alignment when want simpler, faster training than DPO/PPO', satisfying the explicit 'Use when' trigger clause.	3 / 3
Trigger Term Quality	It includes natural user-facing terms like 'preference alignment', 'DPO', 'PPO', 'training' and 'LLM alignment', but omits common variations a user might actually say ('RLHF', 'reward model', 'fine-tuning', 'instruction tuning'). Coverage is relevant but not comprehensive.	2 / 3
Distinctiveness Conflict Risk	SimPO occupies a clear niche (reference-free preference optimization) with distinct triggers (DPO/PPO alternative, no reference model), making it unlikely to fire for unrelated skills.	3 / 3
	Total	10 / 12 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
referenced_paths_exist	Referenced path issues: 3 missing	Warning

	Total	14 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/simpo/SKILL.md
Commit: 52845c3

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.