transformer-lens-interpretability

Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill body that leans on executable code and appropriately offloads detail to a real references bundle. The main gap is the absence of validation/verification steps in the experimental workflows, which keeps workflow clarity just below 5.

Suggestions

Add explicit validation/verification checkpoints to the activation-patching and circuit-analysis workflows (e.g., sanity-check baseline logit diff against known value, assert patched metric monotonically tracks expectation) so each workflow confirms results before interpretation.

Trim provenance/promotional framing ('de facto standard', '2,900+ stars') which does not aid execution and competes for context budget.

Add brief failure-mode notes or expected-output assertions to the code blocks so Claude can confirm a step worked before moving to the next.

Dimension	Reasoning	Score
Conciseness	Largely efficient with executable code blocks and minimal conceptual padding, though a few sentences (e.g. 'TransformerLens is the de facto standard library...' provenance framing) add tokens Claude does not strictly need.	4 / 5
Actionability	Provides copy-paste-ready, executable code for the common cases (activation caching, patching, circuit analysis, induction-head detection, SAE integration) with specific API calls and tensor shapes.	5 / 5
Workflow Clarity	Workflows are numbered with explicit step-by-step sequencing and checklists, but experimental workflows lack validation/verification checkpoints confirming results are sane before proceeding.	4 / 5
Progressive Disclosure	Body is a clear overview with well-signaled one-level-deep references to references/README.md, api.md, and tutorials.md (all real files), keeping bulk detail out of SKILL.md and easy to navigate.	5 / 5
	Total	18 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, well-structured description that answers both what and when with concrete, domain-specific triggers and minimal conflict risk. Minor expansion of the action list could push specificity higher.

Dimension	Reasoning	Score
Specificity	Names the domain and several concrete actions ('inspect and manipulate transformer internals via HookPoints and activation caching'), but stops short of listing the fuller set of operations covered in the body (circuit analysis, logit attribution).	4 / 5
Completeness	Explicitly states what the skill does and follows with a concrete 'Use when...' clause listing trigger scenarios, answering both what and when clearly.	5 / 5
Trigger Term Quality	Includes natural phrases users would say ('reverse-engineering model algorithms', 'studying attention patterns', 'activation patching experiments'), though a few synonyms or extension-style terms are absent.	4 / 5
Distinctiveness Conflict Risk	A clear TransformerLens mechanistic-interpretability niche with distinct, specific triggers makes conflict with unrelated skills minimal.	5 / 5
	Total	18 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/transformer-lens/SKILL.md
Commit: 1d182e9

Reviewed: 3 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.