SkillPublic
distributed-training
Multi-GPU and distributed training patterns with PyTorch DDP. Use when scaling training across GPUs.
Quality
81%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Securityby
Passed
No findings from the security scan
Distributed Training Best Practice
- Use DistributedDataParallel (DDP) over DataParallel for multi-GPU
- Initialize process group: dist.init_process_group(backend='nccl')
- Use DistributedSampler for data sharding
- Synchronize batch norm: nn.SyncBatchNorm.convert_sync_batchnorm()
- Only save checkpoint on rank 0
- Scale learning rate linearly with world size
- Use gradient accumulation for effectively larger batch sizes
- Repository
aiming-lab/AutoResearchClaw
- Revision
be4ba47
- Last updated
- First committed
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.