huggingface-tokenizers
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training. Also covers SentencePiece (guides in references/).
72%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Fix and improve this skill with Tessl
tessl review fix ./skills/tokenization/huggingface-tokenizers/SKILL.mdW012: Unverifiable external dependency detected (runtime URL that controls agent).
What this means
The skill fetches instructions or code from an external URL at runtime, and the fetched content directly controls the agent’s prompts or executes code. This dynamic dependency allows the external source to modify the agent’s behavior without any changes to the skill itself.
Why it was flagged
The instruction mentions cloning the Google SentencePiece repository from GitHub during setup, establishing a weak/unknown external operational dependency.
- Repository
OpenLAIR/dr-claw
- Revision
3456bb4
- Audited
- Security analysis
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.