CtrlK
BlogDocsLog inGet started
Tessl Logo

sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

70

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Medium

Suggest reviewing before use

SKILL.md
Quality
Evals
Security

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable skill body with executable code throughout and real, well-signaled one-level-deep references. It is weakened by redundant performance restatements and version-stamped claims that hurt conciseness, and by the absence of validation/feedback checkpoints in the training and inference workflow.

Suggestions

Consolidate the speed/memory figures (stated in three places) into one section and move version-stamped or time-sensitive claims like 'Version: 0.2.0+' and '~1-2 minutes' to a clearly marked version/deprecated area to tighten conciseness.

Add explicit validation checkpoints to the training and encoding workflow — e.g., assert the model loaded and inspect vocab size / sample encodings before using the model downstream — to lift workflow clarity.

Move the detailed training-configuration parameter reference and benchmark tables into references/training.md, keeping the body as a lean overview, to improve token efficiency and progressive disclosure.

DimensionReasoningScore

Conciseness

Mostly efficient with executable examples, but restates speed/memory figures three times (description body, Performance, benchmarks) and carries time-sensitive claims ('Training time: ~1-2 minutes', 'Version: 0.2.0+') outside a deprecated section, so it could be tightened.

2 / 3

Actionability

Every section provides fully executable, copy-paste-ready code or commands — install, spm_train, Python train/encode/decode, subword sampling, T5 pattern, transformers integration — with concrete parameters, matching the 'fully executable' anchor.

3 / 3

Workflow Clarity

The train→encode→decode sequence is ordered, but there are no validation checkpoints or error-recovery feedback loops (e.g., confirm the model loaded, verify vocab size/encoding before downstream use), which caps workflow clarity at 2 per the rubric.

2 / 3

Progressive Disclosure

Clear overview body with well-signaled, one-level-deep references to real files ([Training Guide](references/training.md), [Algorithms](references/algorithms.md)) and descriptive labels, matching the 'clear overview with well-signaled one-level-deep references' anchor.

3 / 3

Total

10

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that concretely states capabilities, performance characteristics, and an explicit 'Use when' trigger with natural keywords. It clearly distinguishes the skill from general tokenizers via model-family and language-specific triggers.

DimensionReasoningScore

Specificity

Names multiple concrete capabilities — 'Supports BPE and Unigram algorithms', 'Train on raw text without pre-tokenization', 'deterministic vocabulary' — rather than vague abstractions, matching the 'lists multiple specific concrete actions' anchor.

3 / 3

Completeness

Explicitly answers both what ('Language-independent tokenizer treating text as raw Unicode...Supports BPE and Unigram') and when ('Use when you need multilingual support, CJK languages, or reproducible tokenization') with an explicit trigger clause.

3 / 3

Trigger Term Quality

Includes natural user-facing terms ('multilingual support', 'CJK languages', 'reproducible tokenization') and recognizable model-family triggers (T5, ALBERT, XLNet, mBART), giving good coverage of phrases a user would actually say.

3 / 3

Distinctiveness Conflict Risk

Occupies a clear niche (language-independent SentencePiece tokenization) with distinct triggers (CJK, multilingual, reproducible tokenization, specific model families), making it unlikely to fire for unrelated skills.

3 / 3

Total

12

/

12

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository
synthetic-sciences/openscience
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.