transformer-lens-interpretability

Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable body with well-structured progressive disclosure, but it loses points on conciseness (promotional/biographical padding and duplication) and workflow clarity (batch operations lack validation checkpoints). Fixing these would raise it to the top band.

Suggestions

Remove non-actionable promotional context from the intro ('de facto standard', author names, star count) and de-duplicate the installation/basic-usage block that already appears in references/README.md.

Add an explicit validation checkpoint inside the activation patching workflow (e.g., assert the metric returns sane values, or verify hooks are reset with reset_hooks() before and after the batch loop) so the batch operation has a validate-fix-retry feedback loop.

Fix the dangling papers.md reference in references/README.md (the file does not exist in the bundle) to keep navigation trustworthy.

Dimension	Reasoning	Score
Conciseness	Mostly efficient executable code, but padded with non-actionable promotional/biographical context ('de facto standard library', 'Created by Neel Nanda and maintained by Bryce Meyer', '(2,900+ stars)') and duplicated install/basic-usage content that could be tightened; not lean enough for the top anchor.	2 / 3
Actionability	Provides multiple complete, executable Python examples with real TransformerLens API calls (run_with_cache, cache['resid_post', 5], the patching loop, SAELens integration) that are copy-paste ready, matching the top anchor.	3 / 3
Workflow Clarity	Workflows have numbered steps and checklists, but the activation patching workflow is a batch operation over every (layer, position) with no validation checkpoint or validate-fix-retry feedback loop inside it, which caps it at 2 per the batch-operation guideline.	2 / 3
Progressive Disclosure	A clear Reference Documentation table signals one-level-deep links to real files (README.md, api.md, tutorials.md) and the full API reference is appropriately pushed to references/; navigation is easy. (Note: references/README.md lists a non-existent papers.md, a minor bundle defect.)	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, well-targeted description: it states concrete capabilities and gives an explicit 'Use when' trigger using natural domain vocabulary, with a clearly distinct niche that is unlikely to conflict with other skills.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions ('inspect and manipulate transformer internals via HookPoints and activation caching', 'reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments'), matching the anchor for listing several specific concrete actions.	3 / 3
Completeness	Explicitly answers both what ('Provides guidance... to inspect and manipulate transformer internals') and when ('Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments') with an explicit trigger clause.	3 / 3
Trigger Term Quality	Uses natural domain vocabulary a mechanistic-interpretability user would actually say ('mechanistic interpretability', 'TransformerLens', 'activation patching', 'attention patterns'); these are precise domain terms rather than generic jargon, so it is not the level below.	3 / 3
Distinctiveness Conflict Risk	Occupies a clear niche (TransformerLens, HookPoints, activation patching) with distinct triggers unlikely to fire for unrelated skills; not generic enough to conflict as the level-2 anchor would.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: Orchestra-Research/AI-Research-SKILLs
Path: 04-mechanistic-interpretability/transformer-lens/SKILL.md
Commit: 773a529

Reviewed: about 12 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.