distributed-llm-pretraining-torchtitan

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

High

Do not use without reviewing

Quality

Content

80%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable, well-structured, and uses progressive disclosure effectively with real reference files. Its main weakness is the absence of explicit validation checkpoints in expensive batch training workflows, which the rubric caps at 3 for workflow clarity.

Suggestions

Add an explicit validation checkpoint after launching training in each workflow (e.g., confirm rank 0 logs 'Training started' and check first checkpoint exists before proceeding), since these are expensive batch operations the rubric requires validation for.

Embed a verify-resume step in Workflow 2/4 Step 4 (e.g., confirm checkpoint step and optimizer state loaded on all ranks) rather than only stating 'auto-resumes'.

Tighten the Float8 section by stating the 30-50% H100 speedup once and referencing it, reducing minor restatement across the Float8 workflow and benchmarks.

Dimension	Reasoning	Score
Conciseness	Information-dense with no padding of concepts Claude already knows, but a few sections restated config concepts (e.g., Float8 speedup noted in multiple places), leaving minor over-explanation to trim.	4 / 5
Actionability	Fully executable copy-paste guidance throughout — pip/torchrun/srun commands, complete TOML configs, and worked examples covering single-node, multi-node, Float8, and 4D-parallel cases.	5 / 5
Workflow Clarity	Workflows are clearly sequenced with checklists, but these are expensive batch operations (multi-node pretraining up to 512 GPUs) lacking explicit post-launch validation/checkpoint-verification steps, triggering the rubric's batch-operation cap of 3.	3 / 5
Progressive Disclosure	Clear overview with well-signaled one-level-deep references to fsdp.md, float8.md, checkpoint.md, and custom-models.md (all verified present), keeping quick-start inline and details externalized.	5 / 5
	Total	17 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is precise, action-oriented, and provides both a clear capability statement and an explicit trigger clause with concrete model/scale context. Minor trigger-synonym coverage gaps keep it just short of a perfect trigger-term score.

Dimension	Reasoning	Score
Specificity	Names multiple concrete capabilities (4D parallelism with FSDP2/TP/PP/CP, Float8, torch.compile, distributed checkpointing) with comprehensive coverage matching the score-5 anchor.	5 / 5
Completeness	Clearly states what it does ('Provides PyTorch-native distributed LLM pretraining...') and an explicit 'Use when' clause with concrete triggers (model families, 8-512+ GPUs, Float8), matching the score-5 anchor.	5 / 5
Trigger Term Quality	Strong natural terms users would say (pretraining, Llama 3.1, DeepSeek V3, GPUs, Float8) but lacks some synonyms and file extensions needed for full score-5 comprehensiveness.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (PyTorch-native at-scale pretraining via torchtitan) with distinct triggers and minimal overlap risk against fine-tuning or inference skills.	5 / 5
	Total	19 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
referenced_paths_exist	Referenced path issues: 2 missing	Warning

	Total	14 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/torchtitan/SKILL.md
Commit: fcd2e11

Reviewed: 2 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.