CtrlK
BlogDocsLog inGet started
Tessl Logo

distributed-llm-pretraining-torchtitan

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

69

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

High

Do not use without reviewing

SKILL.md
Quality
Evals
Security

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill with executable configs and clean progressive disclosure into verified reference files. Its main weaknesses are a marketing-fluff opener with minor redundancy and the absence of validation checkpoints in multi-step training workflows.

Suggestions

Remove or soften the unverifiable '65%+ speedups' marketing claim in the intro and de-duplicate the tokenizer-download step between Quick start and Workflow 1.

Add explicit validation checkpoints to each workflow (e.g., 'verify GPU utilization/loss after launch', 'confirm seed checkpoint exists before PP launch', 'validate checkpoint loads before resuming').

For destructive/large-scale operations like 512-GPU 4D-parallel launches, add a feedback loop (launch -> check first-step loss -> abort/fix -> relaunch).

DimensionReasoningScore

Conciseness

Mostly efficient with dense code/configs, but the opening 'achieving 65%+ speedups over baselines on H100 GPUs' is marketing fluff and the tokenizer-download step is duplicated in Quick start and Workflow 1, so it could be tightened. Not a 3 because of these padding/redundancy issues; not a 2-below because it avoids explaining concepts Claude already knows.

2 / 3

Actionability

Provides copy-paste-ready executable guidance throughout — complete TOML configs, torchrun commands, SLURM scripts, and Float8/4D-parallelism launch commands with exact flags.

3 / 3

Workflow Clarity

Each workflow has numbered steps and a copyable checklist, but none include explicit validation checkpoints (e.g., verify loss is decreasing, confirm checkpoint integrity, validate seed checkpoint) for risky batch/distributed operations, which caps clarity at 2 per the rubric. Not a 1 because sequences are clearly ordered; not a 3 because feedback loops are absent.

2 / 3

Progressive Disclosure

SKILL.md is a concise overview whose 'Advanced topics' section links to four real one-level-deep references (fsdp.md, float8.md, checkpoint.md, custom-models.md), each verified to exist, with content appropriately split and easy to navigate.

3 / 3

Total

10

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that pairs concrete capabilities with an explicit 'Use when' trigger and natural user terminology. It is distinctive and unlikely to conflict with adjacent skills.

DimensionReasoningScore

Specificity

Lists multiple concrete capabilities — 'distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP)' plus 'Float8, torch.compile, and distributed checkpointing' — rather than vague language.

3 / 3

Completeness

Explicitly answers what ('Provides PyTorch-native distributed LLM pretraining...') and when via the 'Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs' trigger clause.

3 / 3

Trigger Term Quality

Natural terms users pretraining models would say are well covered: 'pretraining Llama 3.1, DeepSeek V3', '8 to 512+ GPUs', 'Float8', 'torch.compile'.

3 / 3

Distinctiveness Conflict Risk

Clear niche — PyTorch-native torchtitan pretraining at 8–512+ GPU scale — with distinct triggers unlikely to fire for fine-tuning or inference skills.

3 / 3

Total

12

/

12

Passed

Validation

87%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation14 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

referenced_paths_exist

Referenced path issues: 2 missing

Warning

Total

14

/

16

Passed

Repository
Orchestra-Research/AI-Research-SKILLs
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.