openrlhf-training

High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Medium

Suggest reviewing before use

Quality

Content

87%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is lean, highly actionable, and well-structured with appropriate progressive disclosure to four reference files. Its main weakness is the absence of explicit validation/feedback checkpoints in the training workflows, which is material for batch GPU operations.

Suggestions

Add explicit validation checkpoints to training workflows (e.g., verify checkpoint save and inspect reward/loss curves before proceeding to the next stage).

Include a brief feedback loop for the common-issues section (e.g., 'if OOM persists after disabling colocation, reduce micro_train_batch_size and retry').

Mark the Docker container version (25.02) and DeepSpeed/OpenRLHF versions as time-sensitive so future staleness is easy to spot.

Dimension	Reasoning	Score
Conciseness	Lean, mostly executable command blocks with minimal prose; assumes Claude's competence and adds no padding or basic concept explanations.	5 / 5
Actionability	Copy-paste-ready bash commands cover PPO, GRPO, DPO, and reward-model training plus troubleshooting, with concrete flags and realistic hyperparameters throughout.	5 / 5
Workflow Clarity	Workflows are clearly sequenced and labeled (e.g., RM then PPO), but batch training operations lack explicit validation checkpoints or feedback loops for error recovery, capping the score.	3 / 5
Progressive Disclosure	A clear overview body with four well-signaled, one-level-deep references to real files in references/, each labeled by topic and easy to navigate.	5 / 5
	Total	18 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is concrete, trigger-rich, and clearly scoped to OpenRLHF-based RLHF training, answering both what and when explicitly. Minor room for improvement only in synonym breadth for trigger terms.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete capabilities (PPO, GRPO, RLOO, DPO training) plus the enabling stack (Ray, vLLM, ZeRO-3), giving comprehensive coverage of the framework's actions.	5 / 5
Completeness	Explicitly answers 'what' (high-performance RLHF framework with Ray+vLLM acceleration) and 'when' ('Use for PPO, GRPO, RLOO, DPO training of large models 7B-70B+') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Strong natural terms a user would say (PPO, GRPO, RLOO, DPO, RLHF, Ray, vLLM), but lacks synonym breadth and casual phrasings that would push it to comprehensive.	4 / 5
Distinctiveness Conflict Risk	A sharply defined RLHF post-training niche with specific algorithms, a named stack, and a named competitor (DeepSpeedChat), yielding minimal overlap with other skills.	5 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/openrlhf/SKILL.md
Commit: fcd2e11

Reviewed: 2 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.