distributed-llm-pretraining-torchtitan

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

High

Do not use without reviewing

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill with executable configs and clean progressive disclosure into verified reference files. Its main weaknesses are a marketing-fluff opener with minor redundancy and the absence of validation checkpoints in multi-step training workflows.

Suggestions

Remove or soften the unverifiable '65%+ speedups' marketing claim in the intro and de-duplicate the tokenizer-download step between Quick start and Workflow 1.

Add explicit validation checkpoints to each workflow (e.g., 'verify GPU utilization/loss after launch', 'confirm seed checkpoint exists before PP launch', 'validate checkpoint loads before resuming').

For destructive/large-scale operations like 512-GPU 4D-parallel launches, add a feedback loop (launch -> check first-step loss -> abort/fix -> relaunch).

Dimension	Reasoning	Score
Conciseness	Mostly efficient with dense code/configs, but the opening 'achieving 65%+ speedups over baselines on H100 GPUs' is marketing fluff and the tokenizer-download step is duplicated in Quick start and Workflow 1, so it could be tightened. Not a 3 because of these padding/redundancy issues; not a 2-below because it avoids explaining concepts Claude already knows.	2 / 3
Actionability	Provides copy-paste-ready executable guidance throughout — complete TOML configs, torchrun commands, SLURM scripts, and Float8/4D-parallelism launch commands with exact flags.	3 / 3
Workflow Clarity	Each workflow has numbered steps and a copyable checklist, but none include explicit validation checkpoints (e.g., verify loss is decreasing, confirm checkpoint integrity, validate seed checkpoint) for risky batch/distributed operations, which caps clarity at 2 per the rubric. Not a 1 because sequences are clearly ordered; not a 3 because feedback loops are absent.	2 / 3
Progressive Disclosure	SKILL.md is a concise overview whose 'Advanced topics' section links to four real one-level-deep references (fsdp.md, float8.md, checkpoint.md, custom-models.md), each verified to exist, with content appropriately split and easy to navigate.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that pairs concrete capabilities with an explicit 'Use when' trigger and natural user terminology. It is distinctive and unlikely to conflict with adjacent skills.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete capabilities — 'distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP)' plus 'Float8, torch.compile, and distributed checkpointing' — rather than vague language.	3 / 3
Completeness	Explicitly answers what ('Provides PyTorch-native distributed LLM pretraining...') and when via the 'Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs' trigger clause.	3 / 3
Trigger Term Quality	Natural terms users pretraining models would say are well covered: 'pretraining Llama 3.1, DeepSeek V3', '8 to 512+ GPUs', 'Float8', 'torch.compile'.	3 / 3
Distinctiveness Conflict Risk	Clear niche — PyTorch-native torchtitan pretraining at 8–512+ GPU scale — with distinct triggers unlikely to fire for fine-tuning or inference skills.	3 / 3
	Total	12 / 12 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
referenced_paths_exist	Referenced path issues: 2 missing	Warning

	Total	14 / 16 Passed

Repository: OpenLAIR/dr-claw
Path: skills/model-architecture/torchtitan/SKILL.md
Commit: 6da7f7c

Reviewed: about 7 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.