openrlhf-training

High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Critical

Do not install without reviewing

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with executable commands and good progressive disclosure into real reference files, but it loses points on a duplicated PPO command and on missing validation checkpoints for batch training workflows.

Suggestions

Remove the verbatim duplication of the PPO command between Quick start and Workflow 1 Step 2 — reference the first invocation or factor the shared flags to reclaim tokens.

Add validation checkpoints to the multi-step pipelines (e.g., 'confirm ./output/llama3-8b-rm checkpoint exists before starting PPO', 'check reward-model accuracy before training') to lift workflow clarity.

Trim the comparative marketing line ('2x faster than DeepSpeedChat') in the body or move it to a benchmark reference, since it adds little actionable guidance.

Dimension	Reasoning	Score
Conciseness	Prose is lean and avoids explaining concepts Claude knows, but the near-identical long PPO command is duplicated verbatim between Quick start and Workflow 1 Step 2, so not every token earns its place; it could be tightened by referencing the earlier command or factoring shared flags.	2 / 3
Actionability	Provides fully executable, copy-paste-ready commands for installation, PPO, reward-model, GRPO, and DPO training with concrete flags and hyperparameters, plus specific troubleshooting commands in Common issues.	3 / 3
Workflow Clarity	Workflows are sequenced (Workflow 1: Train RM -> PPO) and the GRPO variant is clearly parameterized, but these batch training jobs lack validation checkpoints or feedback loops (e.g., confirm a checkpoint exists before the PPO step, verify reward-model quality), which caps workflow clarity at 2 per the batch-operations guideline.	2 / 3
Progressive Disclosure	The body is a clear overview that pushes advanced detail to four well-signaled, one-level-deep references (hybrid-engine, algorithm-comparison, multi-node-training, custom-rewards), all of which exist as real files; content is appropriately split and easy to navigate.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is strong: it names concrete training algorithms, gives an explicit 'Use for' trigger, and stakes out a distinct RLHF-training niche. Minor weakness is comparative marketing language ('2x faster than DeepSpeedChat') that borders on over-claim.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions ('PPO, GRPO, RLOO, DPO training of large models (7B-70B+)') with specific model-size ranges rather than vague language; the only soft spot is marketing fluff like 'High-performance' and '2x faster', but the named actions are concrete enough for the top anchor.	3 / 3
Completeness	Explicitly answers both 'what' ('High-performance RLHF framework with Ray+vLLM acceleration') and 'when' ('Use for PPO, GRPO, RLOO, DPO training of large models'), satisfying the explicit-trigger requirement that would otherwise cap at 2.	3 / 3
Trigger Term Quality	Includes the natural terms an RLHF practitioner would actually say ('PPO', 'GRPO', 'RLOO', 'DPO', 'RLHF', 'large models') via the 'Use for' clause; the jargon (Ray, vLLM, ZeRO-3) is appropriate for this audience rather than generic filler.	3 / 3
Distinctiveness Conflict Risk	Occupies a clear niche (OpenRLHF/Ray/vLLM-based RLHF training) with distinct algorithm triggers unlikely to fire for unrelated skills; no overlap with generic training or document skills.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: Orchestra-Research/AI-Research-SKILLs
Path: 06-post-training/openrlhf/SKILL.md
Commit: 773a529

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.