huggingface-tokenizers

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Quality

77%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/llm-tools/huggingface-tokenizers/SKILL.md

Quality

Content

65%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-organized body with excellent code coverage, but it is verbose in places (conceptual explanations Claude already knows) and duplicates reference content inline rather than pointing out. Workflow steps lack explicit validation checkpoints.

Suggestions

Remove or compress the 'How it works' algorithm explanations and advantages/trade-offs, or move them entirely to references/algorithms.md, keeping SKILL.md focused on actionable usage.

Avoid duplicating the pipeline and algorithm detail inline when references/pipeline.md and references/algorithms.md already cover them — link out instead.

Add an explicit validation/check step to the training and batch workflows (e.g. verify saved tokenizer round-trips via encode/decode) to strengthen workflow clarity.

Dimension	Reasoning	Score
Conciseness	Mostly efficient with abundant executable code, but the 'How it works' sections re-explain BPE/WordPiece/Unigram merge mechanics and advantages/trade-offs that Claude already knows, adding avoidable tokens.	2 / 3
Actionability	Provides numerous complete, copy-paste-ready code examples covering loading, training, padding, the full pipeline, alignment tracking, and transformers integration.	3 / 3
Workflow Clarity	The pipeline (Normalization → Pre-tokenization → Model → Post-processing) is clearly sequenced, but there are no explicit validation checkpoints or error-recovery feedback loops for the training/batch workflows.	2 / 3
Progressive Disclosure	References are one level deep and well-signaled, but the algorithms and pipeline content duplicated inline also exists in references/algorithms.md and references/pipeline.md, so the split is not clean.	2 / 3
	Total	9 / 12 Passed

Description

90%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description with explicit trigger guidance and good natural-language keywords. The only ding is the second-person 'you need' in the trigger clause, which the rubric penalizes on specificity.

Suggestions

Rewrite the trigger clause in third person to avoid the specificity penalty, e.g. 'Use when high-performance tokenization or custom tokenizer training is needed.'

Dimension	Reasoning	Score
Specificity	Lists several concrete actions ('tokenizes 1GB in <20 seconds', 'Train custom vocabularies, track alignments, handle padding/truncation'), but the trigger clause 'Use when you need' uses second person, which the guidelines penalize by reducing specificity by one.	2 / 3
Completeness	Explicitly states both what it does (fast tokenizers, Rust implementation, algorithm support) and when to use it via the 'Use when you need high-performance tokenization or custom tokenizer training' clause.	3 / 3
Trigger Term Quality	Covers natural terms users would say — 'tokenization', 'custom tokenizer training', 'BPE, WordPiece, Unigram', 'high-performance tokenization' — giving good keyword coverage.	3 / 3
Distinctiveness Conflict Risk	Occupies a clear niche (fast Rust tokenization and custom tokenizer training) with distinct triggers unlikely to collide with other skills.	3 / 3
	Total	11 / 12 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (518 lines); consider splitting into references/ and linking	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	14 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/llm-tools/huggingface-tokenizers/SKILL.md
Commit: 52845c3

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.