distributed-llm-pretraining-torchtitan

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

High

Do not use without reviewing

Quality

Content

80%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable skill body with executable code and clean progressive disclosure to real reference files. The main gap is the absence of explicit validation/verification checkpoints inside the training workflows, which are expensive batch operations that warrant feedback loops.

Suggestions

Add an explicit validation step to each workflow, e.g. after launch: confirm `nvidia-smi` shows GPU utilization and `./outputs/tb` is being written before considering training started.

Add a checkpoint-validation checkpoint before resume (e.g. verify `checkpoint/` contains a `step-N` dir matching the configured interval) so resume failures are caught early.

Trim a few explanatory asides like 'Float8 provides 30-50% speedup on H100 GPUs' into the benchmark table to tighten conciseness further.

Dimension	Reasoning	Score
Conciseness	Mostly lean and efficient with executable code blocks and compact tables; assumes Claude's competence and avoids explaining basic concepts. A few explanatory asides ('Float8 provides 30-50% speedup', 'Required for consistent initialization across PP stages') could be trimmed but earn their place.	4 / 5
Actionability	Fully executable, copy-paste-ready guidance throughout — install commands, torchrun invocations, complete TOML configs, SLURM scripts, and checkpoint-conversion commands covering the common single-node, multi-node, Float8, and 4D-parallelism cases.	5 / 5
Workflow Clarity	Four workflows are clearly sequenced with copy-paste checklists, but validation/verification checkpoints are absent for expensive batch GPU training jobs — there is no 'verify training started correctly' or 'confirm checkpoint validity before resume' step, which caps this dimension at 3 per the batch-operation guideline.	3 / 5
Progressive Disclosure	Clear overview structure with a dedicated 'Advanced topics' section signaling one-level-deep references (fsdp.md, float8.md, checkpoint.md, custom-models.md), all of which exist as real files, keeping the body an effective entry point.	5 / 5
	Total	17 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that concisely states what the skill does and when to use it with concrete model names, parallelism techniques, and scale ranges. No vague fluff or over-claims.

Dimension	Reasoning	Score
Specificity	Names the domain (distributed LLM pretraining via torchtitan) and lists multiple concrete capabilities — 4D parallelism (FSDP2, TP, PP, CP), Float8, torch.compile, distributed checkpointing — giving comprehensive coverage.	5 / 5
Completeness	Explicitly answers both 'what' ('Provides PyTorch-native distributed LLM pretraining...with 4D parallelism') and 'when' ('Use when pretraining Llama 3.1, DeepSeek V3...') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Natural trigger terms a user would say are present — 'pretraining Llama 3.1', 'DeepSeek V3', 'custom models', '8 to 512+ GPUs', 'Float8', 'torch.compile', 'distributed checkpointing' — including model-name synonyms and scale phrasing.	5 / 5
Distinctiveness Conflict Risk	A clear niche (torchtitan distributed pretraining) anchored by specific model names and GPU-scale ranges makes it highly distinct with minimal conflict risk against other skills.	5 / 5
	Total	20 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
referenced_paths_exist	Referenced path issues: 2 missing	Warning

	Total	14 / 16 Passed

Repository: OpenLAIR/dr-claw
Path: skills/model-architecture/torchtitan/SKILL.md
Commit: 4b35756

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.