simpo

Reference-free preference alignment, simpler than DPO.

Quality

64%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./optional-skills/mlops/simpo/SKILL.md

Quality

Content

80%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is strong and actionable with copy-paste commands, complete configs, and well-structured progressive disclosure to real reference files. The main gap is the absence of explicit validation/verification checkpoints in the training workflows, which a long batch operation should include.

Suggestions

Add a post-training validation step to each workflow, e.g. 'Verify the output checkpoint loads and evaluate on test_prefs before declaring success.'

Confirm or inline the externally referenced scripts/paths (scripts/run_simpo.py, training_configs/*.yaml, accelerate_configs/deepspeed_zero3.yaml) so the commands are guaranteed runnable.

Trim minor redundancy such as restating 'no reference model' / 'Simplest, best performance' that duplicates the description.

Dimension	Reasoning	Score
Conciseness	The body is efficient with terse configs, commands, and inline comments, but has minor trimmable redundancy in the alternatives blurb and a few restated comments.	4 / 5
Actionability	It provides copy-paste-ready accelerate launch commands and complete YAML configs across three workflows covering the common cases, with concrete troubleshooting snippets.	5 / 5
Workflow Clarity	The configure-to-launch sequence is clear, but the training workflows lack validation checkpoints (e.g. verifying loss convergence or output checkpoints), and the rubric caps batch/long operations without validation at 3.	3 / 5
Progressive Disclosure	The body is a clear overview that points to three real one-level-deep references (loss-functions.md, hyperparameters.md, datasets.md), all of which exist, with well-signaled navigation and appropriately split content.	5 / 5
	Total	17 / 20 Passed

Description

48%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is concise and distinct but undersized: it states the technique's category and key differentiator without listing concrete actions or any 'use when' trigger guidance. Adding explicit triggers and a verb-based capability list would substantially raise specificity and completeness.

Suggestions

Add a 'Use when...' clause with concrete triggers, e.g. 'Use when aligning LLMs on preference data without a reference model, or when DPO is too costly.'

Replace the noun-phrase framing with concrete actions, e.g. 'Trains LLMs with reference-free preference optimization (SimPO); simpler and higher-performing than DPO.'

Include natural user synonyms such as 'preference optimization', 'RLHF alternative', or 'fine-tune on chosen/rejected pairs' to improve trigger-term coverage.

Dimension	Reasoning	Score
Specificity	The description names the domain ('preference alignment') but lists no concrete actions a user would invoke, matching the score-2 anchor rather than the action-listing anchors above.	2 / 5
Completeness	It gives a clear 'what' (reference-free preference alignment) but entirely omits any 'when to use' trigger clause, which per the rubric caps completeness at 3.	3 / 5
Trigger Term Quality	'DPO' and 'preference alignment' are relevant domain keywords, but common natural variations like 'preference optimization', 'RLHF alternative', or 'fine-tune on preferences' are missing.	3 / 5
Distinctiveness Conflict Risk	'Reference-free' and 'simpler than DPO' carve a fairly distinct niche with specific triggers, with only minor overlap risk against DPO-related skills.	4 / 5
	Total	12 / 20 Passed

Validation

75%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 12 / 16 Passed

Validation for skill structure

Criteria	Description	Result
metadata_version	'metadata.version' is missing	Warning
metadata_field	'metadata' should map string keys to string values	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
referenced_paths_exist	Referenced path issues: 3 missing	Warning

	Total	12 / 16 Passed

Repository: NousResearch/hermes-agent
Path: optional-skills/mlops/simpo/SKILL.md
Commit: 6680afb

Reviewed: about 2 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.