data-loading
Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.
63
Quality
73%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
View guide
Securityby
Passed
No findings from the security scan
Fix and improve this skill with Tessl
tessl review fix ./researchclaw/skills/builtin/tooling/data-loading/SKILL.mdEfficient Data Loading Best Practice
- Use num_workers = min(8, os.cpu_count()) for DataLoader
- Enable pin_memory=True when using GPU
- Use persistent_workers=True to avoid re-spawning
- Pre-compute and cache transformations when possible
- For image data: use torchvision.transforms.v2 (faster)
- For large datasets: consider memory-mapped files or WebDataset
- Profile with torch.utils.bottleneck to find I/O bottlenecks
- Repository
- aiming-lab/AutoResearchClaw
- Commit
e2e23c9
- Last updated
- First committed
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.