huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Quality

76%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/llm-tools/huggingface-tokenizers/SKILL.md

Quality

Content

65%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable body with executable code throughout, but its 507-line length, redundant algorithm explanations, and inline duplication of content that has dedicated reference files hurt conciseness and progressive disclosure, and batch training lacks validation checkpoints.

Suggestions

Remove the inline 'How it works' algorithm explanations and the duplicate BPE training block, pointing instead to references/algorithms.md to reduce token cost and duplication.

Add a validation/verification step to the training workflow (e.g., assert vocab size, test-encode held-out text) so the batch training operation has an explicit checkpoint.

Slim the body to an overview that links out to the four reference files for algorithm mechanics and pipeline-component details, keeping only a single quick-start example inline.

Dimension	Reasoning	Score
Conciseness	At 507 lines the body is mostly efficient, but 'How it works' sections re-explain BPE/WordPiece/Unigram mechanics Claude already knows and the BPE training example is duplicated across Quick start and Algorithms, which could be tightened.	3 / 5
Actionability	Copy-paste-ready, executable Python covers the common cases end to end: load pretrained, train BPE/WordPiece/Unigram, batch encode with padding, alignment tracking, transformers integration, and multi-processing.	5 / 5
Workflow Clarity	The pipeline is given as a conceptual sequence (Normalization -> Pre-tokenization -> Model -> Post-processing) but tokenizer training — a batch operation — has no validation/verification checkpoints, which caps workflow clarity at 3 per the batch-operation rule.	3 / 5
Progressive Disclosure	Four real, well-labeled, one-level-deep reference files are listed, but substantial algorithm and pipeline content that belongs in those references is duplicated inline rather than offloaded to them.	3 / 5
	Total	14 / 20 Passed

Description

88%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that concretely lists capabilities and quantifies performance, with an explicit 'Use when' trigger clause. Minor gaps in trigger-term synonyms and file extensions keep trigger quality and distinctiveness just below the top anchor.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions (tokenize, train custom vocabularies, track alignments, handle padding/truncation, integrate with transformers) plus a quantified performance claim, giving comprehensive capability coverage.	5 / 5
Completeness	Clearly states what it does ('Train custom vocabularies, track alignments, handle padding/truncation') and explicitly when to use it ('Use when you need high-performance tokenization or custom tokenizer training') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Includes natural terms like 'tokenization' and 'custom tokenizer training' and the algorithm names, but misses common synonyms ('tokenize', 'subword', 'vocab') and any file extensions, so coverage is good rather than comprehensive.	4 / 5
Distinctiveness Conflict Risk	The Rust-speed and BPE/WordPiece/Unigram framing carves a clear niche, with only minor overlap risk against a general transformers AutoTokenizer skill.	4 / 5
	Total	18 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (518 lines); consider splitting into references/ and linking	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	14 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/llm-tools/huggingface-tokenizers/SKILL.md
Commit: fcd2e11

Reviewed: 2 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.