CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

emerging-techniques-knowledge-distillation

Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

Quality

72%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./cli-tool/components/skills/ai-research/emerging-techniques-knowledge-distillation/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

50%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is strong on concrete code (loss functions, a production Trainer, multi-teacher distillation) but weak on economy and architecture: KLD explanations and loss code are duplicated, known concepts are re-taught, validation feedback loops are missing, and the provided references/minillm.md file is never linked while its content is inlined. Note also that the forward/reverse KLD directionality ("Reverse KL: KL(Teacher || Student) ... mode-covering") is inverted relative to the MiniLLM paper it cites, which risks misleading execution despite running correctly.

Suggestions

Deduplicate: keep one forward-vs-reverse KLD explanation and one distillation-loss implementation (the production DistillationTrainer), and delete the repeated 'Why reverse KL?' / Core Concepts §3 / Quick Start copies to cut ~30% of tokens.

Link the orphaned bundle file (e.g., '## MiniLLM details: See [minillm.md](references/minillm.md)') and move the strategy deep-dives (two-stage, multi-teacher, response distillation) into it, leaving SKILL.md as an overview with the quick start.

Add a validation checkpoint after training: benchmark student vs teacher retention, then give a feedback loop (e.g., 'if retention < 90%, raise alpha toward 0.9 / adjust T in 2-5') instead of a standalone Evaluation snippet; also fix the inverted forward/reverse KLD mode-seeking/covering labels to match arXiv 2306.08543.

DimensionReasoningScore

Conciseness

Noticeably verbose with repeated material: the forward-vs-reverse KLD explanation appears twice ("Why reverse KL?" bullets and "Core Concepts → 3. Forward vs Reverse KLD"), the distillation loss is implemented twice (Quick Start and the production DistillationTrainer), paper links are listed in both "Papers:" and "Resources", and the Temperature Scaling section works through softmax math Claude already knows. Not 1 because the body is dense with genuinely useful code rather than prose padding; not 3 because duplication and known-concept explanation occur in several sections, not isolated instances.

2 / 5

Actionability

Mostly executable guidance: complete loss functions ("def distillation_loss(...)" with correct T^2 scaling), a full Trainer subclass with concrete TrainingArguments, multi-teacher ensemble code, and real model IDs ("meta-llama/Llama-2-70b-hf"). Not 5 because the Quick Start loop uses an undefined `optimizer`/`dataloader`, Strategy 2 is pseudocode ("student = distill(teacher, student, epochs=5)"), Response Distillation feeds untokenized dicts to Trainer, and Evaluation calls an undefined `calculate_similarity`. Not 3 because the hard core of the skill (losses, trainer, generation) is fully executable in multiple places.

4 / 5

Workflow Clarity

A rough sequence exists via section order and numbered comments ("# 1. Load teacher... # 3. Training loop"), but there are no validation checkpoints or feedback loops: the Evaluation section is a standalone afterthought with no 'verify student vs teacher retention, then adjust temperature/alpha' loop, and long training runs are batch operations, which caps this dimension at 3. Not 4 precisely because checkpoints are absent rather than minor-gapped.

3 / 5

Progressive Disclosure

The body is well-sectioned and internally navigable, but the one bundle reference (references/minillm.md) is never linked or signaled anywhere in the body, while its exact topic (reverse KLD / MiniLLM, mode-seeking vs mode-covering) is inlined twice — content that belongs in the separate file is duplicated in SKILL.md and the reference is orphaned. Not 2 because the body itself has clear headers and is not a structureless monolith; not 4 because references are entirely unsignaled rather than 'mostly clear'.

3 / 5

Total

12

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete, third-person, with an explicit 'Use when' clause covering three realistic deployment scenarios and a specific technique list. Its only weaknesses are missing common synonyms (e.g., 'model compression') and slight trigger overlap with adjacent compression/quantization skills.

DimensionReasoningScore

Specificity

Names the core action ("Compress large language models using knowledge distillation from teacher to student models") plus a concrete technique coverage list ("temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies"), which is several specific capabilities with only minor gaps (e.g., response distillation, covered in the body, is omitted). Not 5 because it is one primary action plus a technique enumeration rather than multiple distinct concrete actions; not 3 because coverage clearly exceeds 1-2 actions.

4 / 5

Completeness

Explicitly answers both questions: what ("Compress large language models using knowledge distillation from teacher to student models... Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies") and when ("Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs"). This matches the anchor-5 pattern of a clear what plus concrete 'Use when' trigger phrases; third-person voice, so no voice penalty applies.

5 / 5

Trigger Term Quality

Natural phrases users would actually say are present: "compress large language models", "knowledge distillation", "deploying smaller models with retained performance", "transferring GPT-4 capabilities to open-source models", "reducing inference costs". Not 5 because common synonyms like "model compression", "teacher/student models", or "distill a model" are missing; clearly above the 3 anchor's 'some relevant keywords but missing variations'.

4 / 5

Distinctiveness Conflict Risk

KD-specific terms ("reverse KLD", "MiniLLM", "soft targets", "logit distillation") establish a clear niche with minimal misfire risk. Not 5 because the broad triggers "Compress large language models" and "reducing inference costs" have minor overlap risk with closely related skills such as quantization, pruning, or general inference-optimization skills.

4 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

metadata_field

'metadata' should map string keys to string values

Warning

Total

15

/

16

Passed

Repository

davila7/claude-code-templates

Path

cli-tool/components/skills/ai-research/emerging-techniques-knowledge-distillation/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.