CtrlK
BlogDocsLog inGet started
Tessl Logo

sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

69

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Medium

Suggest reviewing before use

SKILL.md
Quality
Evals
Security

Quality

Content

82%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable body with copy-paste code across the main workflows and properly signaled one-level references. Minor gains available from de-duplicating performance specs and adding an explicit model-verification checkpoint after training.

Suggestions

De-duplicate the performance figures (50k sentences/sec, 6MB) that recur in the when-to-use, description-mirrored, and benchmarks sections; state them once.

Add a brief validation step after training, e.g. loading the produced model and asserting the vocab size, to give the workflow an explicit checkpoint.

Move or shorten the inline algorithm and training-config examples that overlap with references/algorithms.md and references/training.md, linking out for detail instead.

DimensionReasoningScore

Conciseness

Mostly lean with executable code and no over-explanation of basic concepts, but performance specs (50k sentences/sec, 6MB) are repeated across the description, when-to-use, and benchmarks sections and could be trimmed.

4 / 5

Actionability

Fully executable, copy-paste ready guidance spanning pip install, spm_train CLI, SentencePieceTrainer.train, encode/decode, T5-style training, and transformers integration, covering the common cases.

5 / 5

Workflow Clarity

Clear Install -> Train -> Encode/Decode sequence with concrete commands, but no explicit validation checkpoint (e.g., verifying the model loaded or vocab size); operations are not destructive so the cap does not apply.

4 / 5

Progressive Disclosure

Good overview structure with two real one-level-deep references (references/training.md, references/algorithms.md) clearly signaled, though some training-config and algorithm detail is inlined that overlaps with those references.

4 / 5

Total

17

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that clearly states capabilities and provides an explicit 'Use when...' trigger. Trigger-term coverage is good but could add common synonyms like 'subword' and 'tokenizer' to reach full natural-phrase coverage.

DimensionReasoningScore

Specificity

Lists several concrete capabilities ('treating text as raw Unicode', 'Supports BPE and Unigram algorithms', 'Train on raw text without pre-tokenization', 'deterministic vocabulary') but they are framed as features rather than a comprehensive action list, leaving minor gaps.

4 / 5

Completeness

Explicitly answers both what the skill does (language-independent tokenizer, BPE/Unigram, fast, lightweight, deterministic) and when to use it via a concrete 'Use when you need multilingual support, CJK languages, or reproducible tokenization' clause.

5 / 5

Trigger Term Quality

Good coverage of natural terms ('multilingual support', 'CJK languages', 'reproducible tokenization', 'BPE', 'Unigram'), but common synonyms like 'subword', 'tokenizer', and 'vocabulary' are absent.

4 / 5

Distinctiveness Conflict Risk

Clear niche naming a specific tool and concrete model users (T5, ALBERT, XLNet, mBART) with distinct triggers, yielding minimal conflict risk with other tokenization skills.

5 / 5

Total

18

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository
synthetic-sciences/openscience
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.