huggingface-tokenizers

Fast BPE/WordPiece tokenization and custom vocab training.

Quality

70%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./optional-skills/mlops/huggingface-tokenizers/SKILL.md

Quality

Content

80%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable and well-structured with excellent progressive disclosure to four reference files, but workflow clarity is capped by missing validation/feedback loops on batch training operations, and there is mild redundancy with the algorithm references.

Suggestions

Add explicit validation/feedback checkpoints to batch and training workflows, e.g. verify the saved tokenizer round-trips (encode→decode) and check vocab_size after training before proceeding.

Trim or relocate the inline 'How it works' algorithm step lists to references/algorithms.md since they duplicate the deep-dive file and restate concepts Claude already knows.

Move or condense the inline performance benchmark tables, which are time/hardware-sensitive, into the references or a clearly scoped section to avoid padding the overview.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient with executable, copy-paste code, but repeats algorithm explanations that also live in references/algorithms.md and includes 'How it works' step lists and benchmark tables that partly restate what Claude already knows.	4 / 5
Actionability	Code is fully executable and copy-paste ready across the common cases — loading pretrained tokenizers, training BPE/WordPiece/Unigram, batch encoding with padding, truncation, alignment tracking, and multiprocessing — with concrete imports and expected outputs.	5 / 5
Workflow Clarity	The tokenization pipeline is clearly sequenced (Normalization → Pre-tokenization → Model → Post-processing), but batch/destructive operations like corpus training and multiprocessing lack explicit validation checkpoints or feedback loops, which caps workflow clarity at 3 per the rubric.	3 / 5
Progressive Disclosure	The body is a clear overview with four well-signaled, one-level-deep references (training.md, algorithms.md, pipeline.md, integration.md) that are real files, each summarized in the References section for easy navigation.	5 / 5
	Total	17 / 20 Passed

Description

61%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific and distinct, naming two concrete capabilities, but lacks an explicit 'when to use' trigger clause and omits several capabilities covered in the body. Adding a 'Use when...' clause with natural trigger terms would raise completeness and trigger-term quality.

Suggestions

Add an explicit 'Use when...' clause, e.g. 'Use when training a custom tokenizer, choosing BPE/WordPiece/Unigram, or needing fast subword tokenization with alignment tracking.'

Include broader natural trigger terms and synonyms such as 'subword tokenization', 'tokenizer training', and 'huggingface tokenizers' to improve keyword coverage.

Mention alignment tracking and transformers integration to make the capability list comprehensive rather than minimal.

Dimension	Reasoning	Score
Specificity	Names the domain (tokenization) plus two concrete actions ('BPE/WordPiece tokenization and custom vocab training'), but is not comprehensive — alignment tracking, pipelines, and transformers integration from the body are absent.	3 / 5
Completeness	The 'what' is clearly stated but there is no 'Use when...' clause or equivalent explicit trigger guidance, which per the rubric caps completeness at 3.	3 / 5
Trigger Term Quality	Strong natural terms ('BPE', 'WordPiece', 'tokenization', 'custom vocab training') users would say, though synonyms like 'subword', 'tokenizer training', or library name 'huggingface tokenizers' would round out coverage.	4 / 5
Distinctiveness Conflict Risk	The specific mention of BPE/WordPiece and custom vocab training carves a clear niche with minimal overlap risk against general NLP skills, though it stops short of the fully-distinct explicit-trigger phrasing of a 5.	4 / 5
	Total	14 / 20 Passed

Validation

75%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 12 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (521 lines); consider splitting into references/ and linking	Warning
metadata_version	'metadata.version' is missing	Warning
metadata_field	'metadata' should map string keys to string values	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	12 / 16 Passed

Repository: NousResearch/hermes-agent
Path: optional-skills/mlops/huggingface-tokenizers/SKILL.md
Commit: 6680afb

Reviewed: about 3 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.