huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Quality

72%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/llm-tools/huggingface-tokenizers/SKILL.md

Quality

Content

61%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable skill with executable code for all major use cases, backed by real one-level-deep reference files. Its main weaknesses are verbosity from duplicated and already-known content, and the absence of validation steps in the tokenizer-training workflow.

Suggestions

Add an explicit validation step after training, e.g. decode a sample sentence back to text and assert the vocabulary size, before saving and wrapping the tokenizer.

Trim the per-algorithm 'How it works' explanations, benchmark tables, and 'Supported models' lists from SKILL.md and move them into references/algorithms.md, keeping only the quick-start training examples inline.

Replace the '# ... train tokenizer ...' placeholder with a complete runnable snippet and define the 'text' variable in the alignment example so code blocks are fully copy-paste ready.

Dimension	Reasoning	Score
Conciseness	The body re-explains BPE/WordPiece/Unigram mechanics ('Start with character-level vocabulary... find most frequent character pair') and includes benchmark tables and a 'Supported models' list that Claude already knows, plus full algorithm/pipeline sections that duplicate content in references/, fitting the 'mostly efficient but includes unnecessary explanation' anchor.	3 / 5
Actionability	Code examples are largely executable and copy-paste ready (load, train, padding, alignment, transformers wrapping), but there are minor gaps: a '# ... train tokenizer ...' placeholder in the convert example and an alignment snippet that references an undefined 'text' variable, so it fits anchor 4 rather than 5.	4 / 5
Workflow Clarity	The training workflow is sequenced (install, initialize, configure trainer, train, save, wrap), but there are no validation checkpoints (e.g. decode round-trip, vocab-size check) for a batch training operation, which the rubric guidelines cap at 3.	3 / 5
Progressive Disclosure	Four real, one-level-deep references are clearly signaled in the References section with accurate descriptions, but the body inlines substantial content (per-algorithm training code and pipeline component listings) that duplicates those reference files, fitting anchor 4 rather than 5.	4 / 5
	Total	14 / 20 Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that clearly states what the skill does with concrete, quantified capabilities and an explicit 'Use when' trigger clause. The main weaknesses are second-person voice in the trigger clause and slightly limited natural-term synonym coverage.

Suggestions

Rewrite the trigger clause in third person to avoid the voice penalty, e.g. 'Use when the user needs high-performance tokenization or custom tokenizer training.'

Add a few natural synonym terms such as 'subword tokenization', 'build a vocabulary', or 'tokenize text' to broaden trigger-term coverage.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions ('tokenizes 1GB in <20 seconds', 'Train custom vocabularies, track alignments, handle padding/truncation') with comprehensive coverage, matching the anchor-5 example, but the second-person phrasing 'Use when you need...' incurs the rubric's -1 voice penalty, bringing it to 4.	4 / 5
Completeness	Explicitly answers both 'what' (detailed capability list including performance, algorithms, training, alignment, padding/truncation, transformers integration) and 'when' ('Use when you need high-performance tokenization or custom tokenizer training') with concrete trigger phrases, matching the anchor-5 example.	5 / 5
Trigger Term Quality	Good keyword coverage ('tokenization', 'BPE, WordPiece, and Unigram', 'custom tokenizer training') that users would naturally say, though common variations like 'subword', 'vocabulary', or 'tokenize text' are missing, so it sits between the anchor-4 and anchor-5 examples.	4 / 5
Distinctiveness Conflict Risk	The tokenizer niche is mostly distinct, but 'Integrates seamlessly with transformers' and the broad 'high-performance tokenization' trigger create minor overlap risk with a transformers/AutoTokenizer skill, fitting the anchor-4 example rather than 5.	4 / 5
	Total	17 / 20 Passed

Validation

81%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 13 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (526 lines); consider splitting into references/ and linking	Warning
metadata_version	'metadata.version' is missing	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	13 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/llm-tools/huggingface-tokenizers/SKILL.md
Commit: 4082a2e

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.