openrlhf-training

High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Medium

Suggest reviewing before use

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable skill body with executable Ray/vLLM commands and good progressive disclosure via real reference files. Its main gaps are a duplicated full command block and a lack of validation checkpoints for expensive batch training workflows.

Suggestions

De-duplicate the PPO command: show the full command once and have Workflow 1 Step 2 reference it or show only the differing flags (e.g., --reward_pretrain ./output/llama3-8b-rm).

Add validation checkpoints to the multi-step pipeline, e.g. verify the Ray cluster is healthy before job submit, and confirm the reward model's accuracy/loss before starting PPO.

Add a brief feedback loop for training instability (check KL divergence / reward curve, then adjust --init_kl_coef) instead of only listing static fix flags.

Dimension	Reasoning	Score
Conciseness	Mostly efficient commands and tables without explaining concepts Claude already knows, but the full PPO command block is duplicated near-verbatim in Quick start and Workflow 1 Step 2, which could be tightened by referencing the earlier block.	2 / 3
Actionability	Fully executable `ray job submit` and `deepspeed --module` commands with real flags, model IDs, and concrete env-var fixes — copy-paste ready with no pseudocode.	3 / 3
Workflow Clarity	Workflows are sequenced (SFT→RM→PPO) but these are large-scale batch training jobs with no validation checkpoints or feedback loops (e.g., verify the reward model before PPO, monitor KL/loss before proceeding), which caps workflow clarity at 2.	2 / 3
Progressive Disclosure	A concise overview with four well-signaled one-level-deep references (hybrid-engine, algorithm-comparison, multi-node-training, custom-rewards), all real files, each labeled by topic and cleanly split out of the main body.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that names concrete capabilities, includes an explicit Use-for trigger covering the natural algorithm terms, and carves out a distinct niche. It uses third-person voice and avoids vague fluff.

Dimension	Reasoning	Score
Specificity	Names the framework, the Ray+vLLM+ZeRO-3 stack, concrete algorithms (PPO, GRPO, RLOO, DPO), model scale (7B-70B+), and a specific benchmark ("2× faster than DeepSpeedChat") — multiple specific concrete actions.	3 / 3
Completeness	Explicitly answers both what ("High-performance RLHF framework with Ray+vLLM acceleration") and when ("Use for PPO, GRPO, RLOO, DPO training of large models"), with an explicit Use-for trigger clause.	3 / 3
Trigger Term Quality	"Use for PPO, GRPO, RLOO, DPO training" plus "RLHF" and "large models (7B-70B+)" are the natural terms a practitioner would say, giving good coverage of real trigger vocabulary.	3 / 3
Distinctiveness Conflict Risk	A clear distributed-RLHF niche with distinct, specific triggers (named algorithms, Ray/vLLM, 7B-70B+ scale) makes it unlikely to fire for unrelated skills.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/openrlhf/SKILL.md
Commit: 52845c3

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.