moe-training

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with complete, executable code and configs, and uses progressive disclosure well with real reference files. Its main weaknesses are duplicated config/command blocks hurting conciseness and the absence of validation checkpoints or feedback loops for a costly large-scale training workflow.

Suggestions

Remove the duplicated DeepSpeed MoE JSON config and deepspeed pretrain command — keep one canonical copy and cross-reference it, improving token efficiency.

Add explicit validation/feedback steps for the training workflow, e.g. a checkpoint for detecting NaN/diverging loss with a remediation action (reduce LR, lower z-loss coeff), and resume-from-checkpoint guidance.

Move the advanced best-practices tuning detail (capacity factor, LR, loss coefficient) into references/training.md and link to it, keeping the body as a tighter overview.

Dimension	Reasoning	Score
Conciseness	The body is mostly code and actionable config rather than padded conceptual explanation, but it duplicates material: the DeepSpeed MoE JSON config appears in both "Expert Parallelism" and "Training Configuration", and the deepspeed pretrain command appears twice. Deduplicating these blocks would tighten it toward the lean anchor.	2 / 3
Actionability	Provides fully executable, copy-paste-ready material — complete MoELayer and MixtralMoEBlock Python, load_balancing_loss/router_z_loss functions, full deepspeed CLI invocations with all flags, and complete JSON configs.	3 / 3
Workflow Clarity	Sections are topically sequenced (install → quick start → concepts → config → advanced → best practices → inference), but for an expensive batch operation (500k iters, 8 GPUs) there are no explicit validation checkpoints or error-recovery feedback loops (e.g., NaN-loss handling, loss-not-decreasing checks), capping clarity at 2 per the destructive/batch-operations guideline.	2 / 3
Progressive Disclosure	SKILL.md is a clear overview with a well-signaled "See Also" section linking to three real, one-level-deep reference files (architectures.md, training.md, inference.md). The inline advanced material is reasonable core knowledge rather than misplaced reference content.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is strong: concrete actions, natural trigger terms including named models/frameworks, explicit what-and-when guidance, and a distinctive niche that minimizes conflict risk. It uses third-person/imperative voice with no first/second person, so no voice penalty applies.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions across a clear domain — "Train Mixture of Experts (MoE) models", "implementing sparse architectures", "scaling model capacity", plus a coverage clause naming routing, load balancing, and expert parallelism. The "Covers..." portion lists components rather than user-facing actions, but the action verbs and named sub-tasks meet the multiple-concrete-actions anchor.	3 / 3
Completeness	Explicitly answers both what ("Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace") and when ("Use when training large-scale models with limited compute..., implementing sparse architectures..., or scaling model capacity..."), with explicit triggers.	3 / 3
Trigger Term Quality	Includes natural terms a practitioner would actually say — "Mixture of Experts", "MoE", "DeepSpeed", "HuggingFace", "Mixtral 8x7B", "DeepSeek-V3" — giving strong coverage of framework and model-name keywords users would use when requesting this skill.	3 / 3
Distinctiveness Conflict Risk	The niche is sharply defined by named models (Mixtral 8x7B, DeepSeek-V3) and frameworks plus distinctive triggers (sparse architectures, expert parallelism, MoE), making it unlikely to fire for an unrelated skill.	3 / 3
	Total	12 / 12 Passed

Validation

81%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 13 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (528 lines); consider splitting into references/ and linking	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
relative_links	Relative link issues: 1 missing	Warning

	Total	13 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/moe-training/SKILL.md
Commit: 52845c3

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.