moe-training

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.

Quality

78%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./skills/emerging-techniques/moe-training/SKILL.md

Quality

Content

68%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with extensive executable code, but it is verbose for the context window — it explains MoE basics Claude already knows and repeats config/command blocks — and lacks an explicit training workflow with validation checkpoints for a heavy batch operation. Progressive disclosure is good but references are buried in a trailing 'See Also' rather than integrated inline.

Suggestions

Collapse the duplicated DeepSpeed MoE config JSON and the two deepspeed training command blocks into a single canonical version; reference it from both sections instead of restating it.

Replace the 'Core Concepts' explanations of experts/router/top-k (which Claude already knows) with a brief pointer and keep only the routing-mechanism code comparisons; move the ASCII routing diagram to references/architectures.md.

Add an explicit numbered training workflow with validation checkpoints — e.g., launch a short smoke run, verify loss is decreasing and max/min expert usage ratio is under ~2.0, tune moe_loss_coeff/z-loss accordingly, then proceed to the full run.

Integrate the references inline at the relevant sections (e.g., 'For full Mixtral/Switch/DeepSeek-V3 architecture details, see [architectures.md]') instead of a trailing 'See Also' list.

Dimension	Reasoning	Score
Conciseness	Mostly concrete and useful, but the ~515-line body includes unnecessary explanation of concepts Claude already knows (the 'Core Concepts' section defines experts/routers/top-k and an ASCII routing diagram) and repeats configuration material — a DeepSpeed MoE config block appears both under 'Expert Parallelism' and again more fully under 'Training Configuration', and a deepspeed training command appears in both 'Quick Start' and 'Training Script'.	3 / 5
Actionability	Provides copy-paste-ready, executable guidance throughout — a full MoELayer implementation, a Mixtral-style MoE block, DeepSpeed launch commands, complete JSON configs, load-balancing/z-loss functions, and a sparse inference routine — covering the common MoE training cases.	5 / 5
Workflow Clarity	The overall flow (install → define architecture → configure → launch training) is implied but never presented as an explicit sequenced workflow, and there are no validation checkpoints (e.g., verify loss decrease, check expert-balance ratio, validate checkpoint) for what is a heavy batch training operation; per the rubric, missing validation in a batch operation caps this at 3.	3 / 5
Progressive Disclosure	The body is well-organized into clearly headed sections and points to real one-level-deep bundle files (references/architectures.md, training.md, inference.md) via a 'See Also' block; the gap is that references are signalled only at the end rather than inline at the relevant sections, and substantial architecture/training/inference content is inlined despite matching reference files existing.	4 / 5
	Total	15 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is strong: third-person voice, explicit 'what' and 'when' with concrete model-name triggers, and a clear MoE niche that minimizes conflict risk. The only weakness is that the 'Covers...' clause lists topic areas rather than additional concrete actions, slightly capping specificity.

Dimension	Reasoning	Score
Specificity	Names the domain ('Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace') plus a list of specific subtopics ('routing mechanisms, load balancing, expert parallelism, and inference optimization'), but the 'Covers...' clause enumerates topic areas rather than additional concrete actions, leaving minor gaps versus the comprehensive multi-action anchor 5.	4 / 5
Completeness	Explicitly states what ('Train Mixture of Experts (MoE) models...') and when ('Use when training large-scale models with limited compute..., implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity...') with concrete trigger phrases, matching the anchor 5 example.	5 / 5
Trigger Term Quality	Good coverage of natural terms a user would say — 'Mixture of Experts (MoE)', 'DeepSpeed', 'HuggingFace', 'Mixtral 8x7B', 'DeepSeek-V3', 'expert parallelism', 'sparse architectures' — but a few common synonyms/variations are absent, stopping short of the fully comprehensive anchor 5.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (MoE training) with distinct triggers (MoE, Mixtral, DeepSeek-V3, expert parallelism, sparse architectures) that are unlikely to fire for unrelated skills, matching the anchor 5 'clear niche with distinct triggers' example.	5 / 5
	Total	18 / 20 Passed

Validation

81%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 13 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (527 lines); consider splitting into references/ and linking	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
relative_links	Relative link issues: 1 missing	Warning

	Total	13 / 16 Passed

Repository: OpenLAIR/dr-claw
Path: skills/emerging-techniques/moe-training/SKILL.md
Commit: 4b35756

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.