openrlhf-training

High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.

Quality

73%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Critical

Do not install without reviewing

Fix and improve this skill with Tessl

tessl review fix ./skills/post-training/openrlhf/SKILL.md

Quality

Content

72%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with complete executable commands and excellent progressive disclosure into real reference files. Its main weaknesses are duplicated long command blocks (conciseness) and the absence of validation checkpoints in the training workflows (workflow clarity).

Suggestions

De-duplicate the PPO command by defining the shared base once and showing only the differing flags in Quick start vs Workflow 1, to recover token budget.

Add validation/verification checkpoints to the training workflows (e.g., inspect reward-model accuracy/loss before launching PPO, monitor KL and reward curves, verify checkpoint saves) to lift workflow clarity.

Move the duplicated '2× faster than DeepSpeedChat' performance claim to a single location to avoid repetition.

Dimension	Reasoning	Score
Conciseness	The body is mostly lean code with no concept over-explanation, but the ~25-line PPO command is repeated nearly verbatim in both Quick start and Workflow 1 Step 2, and the '2× faster than DeepSpeedChat' claim is duplicated — token waste that could be tightened via a shared base command.	3 / 5
Actionability	Fully executable, copy-paste-ready commands with real model identifiers and concrete flags cover PPO, GRPO, DPO, and reward-model training, plus specific troubleshooting fixes like 'export RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1'.	5 / 5
Workflow Clarity	Workflow 1 (SFT → RM → PPO) is sequenced with labeled steps, but there are no validation/verification checkpoints for these expensive batch training jobs; per the batch-operation cap, a destructive/batch workflow without validation cannot score above 3.	3 / 5
Progressive Disclosure	Clear overview with a well-signaled 'Advanced topics' section pointing one level deep to four real reference files (hybrid-engine.md, algorithm-comparison.md, multi-node-training.md, custom-rewards.md), each described by topic, giving easy navigation.	5 / 5
	Total	16 / 20 Passed

Description

75%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific, third-person, and includes an explicit 'Use for' trigger clause covering the main RLHF algorithms and the Ray+vLLM stack. It is slightly terse on scenario framing and could add synonyms to broaden trigger coverage.

Suggestions

Add a 'Use when the user wants to...' clause framing concrete scenarios (e.g., distributed RLHF training on multi-node GPU clusters) rather than only listing algorithm names.

Include synonyms/expansions such as 'reinforcement learning from human feedback' and 'preference optimization' to improve trigger term coverage.

Mention reward-model and SFT support in the description if applicable, to make capability coverage more comprehensive.

Dimension	Reasoning	Score
Specificity	Lists several concrete training actions — 'PPO, GRPO, RLOO, DPO training of large models (7B-70B+)' plus 'Ray+vLLM acceleration' and 'GPU resource sharing' — but omits related actions like reward-model/SFT training, leaving minor coverage gaps rather than comprehensive coverage.	4 / 5
Completeness	Has a clear 'what' ('High-performance RLHF framework with Ray+vLLM acceleration') and an explicit 'Use for...' when clause naming the algorithms, but the when is terse and lists algorithms rather than framing concrete user-facing scenarios.	4 / 5
Trigger Term Quality	Strong natural domain keywords a user would say ('RLHF', 'PPO', 'GRPO', 'RLOO', 'DPO', 'Ray', 'vLLM', 'distributed training'), but missing common synonyms/expansions like 'reinforcement learning from human feedback' or 'preference optimization'.	4 / 5
Distinctiveness Conflict Risk	A clear niche (OpenRLHF / Ray+vLLM / ZeRO-3 stack for distributed RLHF) with distinct triggers, but minor overlap risk with closely related RLHF frameworks like TRL and veRL.	4 / 5
	Total	16 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: OpenLAIR/dr-claw
Path: skills/post-training/openrlhf/SKILL.md
Commit: 4b35756

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.