huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Quality

75%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./skills/tokenization/huggingface-tokenizers/SKILL.md

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with comprehensive executable examples and a clear pipeline structure, supported by real one-level-deep references. Its main weakness is conciseness: it re-explains algorithm concepts Claude already knows and duplicates detail also present in the reference files.

Suggestions

Trim or move the 'How it works' conceptual explanations of BPE/WordPiece/Unigram into algorithms.md, keeping only the executable training snippets inline.

Remove basic restatings such as 'Clean and standardize text' and 'Split text into word-like units' that assume Claude lacks the knowledge.

Add an explicit validation step after training (e.g. encode sample texts and inspect tokens before saving) to give the training workflow a feedback checkpoint.

Dimension	Reasoning	Score
Conciseness	The ~500-line body inlines conceptual 'How it works' explanations of BPE/WordPiece/Unigram and basic restatings (e.g. 'Clean and standardize text') that Claude already knows, going beyond minor over-explanation.	3 / 5
Actionability	Provides extensive copy-paste-ready, executable Python across loading, training all three algorithms, padding/truncation, alignment, transformers integration, and multiprocessing, with expected outputs shown.	5 / 5
Workflow Clarity	The normalization→pre-tokenization→model→post-processing pipeline is clearly sequenced with per-stage code, but lacks explicit validation/feedback checkpoints for the training workflows.	4 / 5
Progressive Disclosure	A clear References section links to four real, well-described bundle files (training.md, algorithms.md, pipeline.md, integration.md), but substantial deep-dive content is inlined in the body that those references also cover.	4 / 5
	Total	16 / 20 Passed

Description

75%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is strong: concrete, quantified, third-person, with explicit 'what' and 'when' clauses and natural trigger terms. It sits solidly at the 4-anchor across all dimensions, held below 5 only by limited trigger-phrase breadth and slightly narrow 'when' guidance.

Dimension	Reasoning	Score
Specificity	Lists several concrete actions (train custom vocabularies, track alignments, handle padding/truncation, integrate with transformers) with a quantified speed claim, matching the 'several specific actions; minor gaps' anchor.	4 / 5
Completeness	Explicitly answers both 'what' (supported algorithms, actions) and 'when' ('Use when you need high-performance tokenization or custom tokenizer training'), though the 'when' lists only two triggers and could be more specific.	4 / 5
Trigger Term Quality	Includes natural terms a user would say ('tokenization', 'tokenizer', BPE/WordPiece/Unigram) plus a 'Use when...' clause, but lacks broader synonyms or extension-style variations for a 5.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (HuggingFace/Rust tokenizers with named algorithms) and is mostly distinct, with only minor overlap risk against generic transformers/text-processing skills.	4 / 5
	Total	16 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (517 lines); consider splitting into references/ and linking	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	14 / 16 Passed

Repository: OpenLAIR/dr-claw
Path: skills/tokenization/huggingface-tokenizers/SKILL.md
Commit: 4b35756

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.