CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

knowledge-distillation

Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

56

Quality

71%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./skills/emerging-techniques/knowledge-distillation/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

46%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body delivers genuinely useful, mostly executable distillation code, but it is bloated: it re-explains basics Claude already knows, duplicates the loss implementation, uses comment-only/pseudocode filler, and inlines a MiniLLM section that duplicates an existing, never-referenced bundle file. There is also no explicit end-to-end workflow with validation checkpoints.

Suggestions

Replace the inline 'MiniLLM (Reverse KLD)' section with a brief summary plus a link to references/minillm.md, and move other advanced material (multi-teacher distillation, full hyperparameter guidance) into reference files so the body stays a lean overview.

Cut the 'Core Concepts' explanations of softmax, temperature scaling, and forward-vs-reverse KL that Claude already knows; keep only distillation-specific guidance such as the T=2-5 range and typical alpha values.

Replace comment-only or pseudocode blocks ('Two-Stage Distillation', 'Model Size Ratio', 'Data Quality') with either real code or short prose, define the undefined helpers (dataloader, train_dataset, calculate_similarity), and add an explicit numbered workflow ending in an evaluation checkpoint with guidance on what to adjust when student quality is poor.

DimensionReasoningScore

Conciseness

The 450-line body contains several padded sections: "Core Concepts" re-explains softmax, temperature, and forward-vs-reverse KL basics Claude already knows; the distillation loss implementation appears twice (Quick Start and Production Deployment); and "Model Size Ratio", "Data Quality" (invalid pseudo-Python like '"teacher_generated": 70%'), and "Two-Stage Distillation" are comment-only or pseudocode filler. This is noticeably verbose with multiple unnecessary explanations rather than 'some' (anchor 3).

2 / 5

Actionability

Most guidance is concrete, executable PyTorch/transformers code: the combined hard/soft distillation loss, the reverse-KL loss, and a complete DistillationTrainer subclass. It falls short of anchor 5 because key pieces are undefined when used — 'dataloader', 'optimizer', 'train_dataset' (passed to Trainer as raw dicts without tokenization), and 'calculate_similarity' — so examples are not fully copy-paste ready.

4 / 5

Workflow Clarity

Section order (Installation → Quick Start → Strategies → Production → Evaluation) implies a rough sequence, and an Evaluation section exists, but there is no explicit end-to-end workflow and no validation checkpoints or feedback loops (e.g., what to adjust when student-vs-teacher match quality is poor). This matches anchor 3: sequence present but checkpoints missing or implicit.

3 / 5

Progressive Disclosure

The bundle contains references/minillm.md (334 lines) but the body never links to it — no reference to any bundle file appears anywhere in SKILL.md. Instead, MiniLLM/reverse-KLD content is duplicated inline, and advanced material (multi-teacher distillation, full hyperparameter guidance) is inlined in a ~450-line monolithic body. This matches anchor 2: content that clearly belongs in separate files is inlined; not anchor 3, where references would at least be present but weakly signaled.

2 / 5

Total

11

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it clearly states what the skill does and when to use it, with an explicit 'Use when...' clause and concrete trigger scenarios. The main weaknesses are a technique list that names topics rather than actions and slight overlap risk with model-compression skills like quantization or pruning.

DimensionReasoningScore

Specificity

The description lists several concrete actions ("Compress large language models using knowledge distillation", "transferring GPT-4 capabilities to open-source models", "reducing inference costs") plus a specific technique list ("temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM"). It falls short of anchor 5 because the technique names are topics rather than actions and coverage has minor gaps (e.g., no mention of evaluation or synthetic-data generation), but it is well above anchor 3's '1-2 concrete actions'.

4 / 5

Completeness

It explicitly answers both questions: what ("Compress large language models using knowledge distillation from teacher to student models... Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies") and when ("Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs"). The 'Use when...' clause with concrete triggers matches the anchor 5 example structure.

5 / 5

Trigger Term Quality

It contains the natural phrases a user would say when needing this skill: "knowledge distillation", "compress large language models", "smaller models", "reducing inference costs", "GPT-4", "MiniLLM". Not anchor 5 because common synonyms such as "model compression", "shrink a model", or "make the model faster/smaller" are missing.

4 / 5

Distinctiveness Conflict Risk

Terms like "knowledge distillation", "reverse KLD", and "MiniLLM" form a clear niche distinct from most skills. However "Compress large language models" and "reducing inference costs" could also trigger quantization, pruning, or inference-optimization skills, giving minor overlap risk with closely related skills (anchor 4) rather than minimal conflict risk (anchor 5).

4 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

OpenLAIR/dr-claw

Path

skills/emerging-techniques/knowledge-distillation/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.