CtrlK
BlogDocsLog inGet started
Tessl Logo

sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

63

Quality

75%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Medium

Suggest reviewing before use

Fix and improve this skill with Tessl

tessl review fix ./skills/tokenization/sentencepiece/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

68%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with comprehensive executable examples and clear navigation to two real reference files. Weaknesses are noticeable duplication of performance figures, inlined detail that belongs in the reference files, and absent validation checkpoints for the batch training workflow.

Suggestions

Add a validation step to the training workflow (e.g., after training, load the model and verify vocab_size/encode a sample) to satisfy the batch-operation feedback-loop requirement.

Move detailed algorithm comparisons and the full training-configuration parameter reference into references/algorithms.md and references/training.md, keeping the body as a lean overview with links.

De-duplicate the 50k sentences/sec and 6MB figures, stating them once and referencing that section instead of repeating across 'When to use' and 'Performance benchmarks'.

DimensionReasoningScore

Conciseness

Mostly efficient with code examples and tables, but performance figures (50k sentences/sec, 6MB) are repeated across the description, 'When to use', and 'Performance benchmarks' sections, and the 'When to use' section largely restates the frontmatter; could be tightened.

3 / 5

Actionability

Fully executable, copy-paste-ready examples cover the common cases: installation (pip + C++), CLI and Python training, encode/decode, algorithm selection, training config, subword regularization, T5-style training, and transformers integration.

5 / 5

Workflow Clarity

A clear install→train→encode sequence is present, but model training is a batch operation with no validation or verification checkpoints (e.g., confirm vocab size, test-load the model, sanity-check encoding), which caps the score at 3 per the batch-operation feedback-loop rule.

3 / 5

Progressive Disclosure

Good section structure with clearly signaled, one-level-deep references to real files (references/training.md, references/algorithms.md), but algorithm details, training configuration, and benchmarks are inlined in the body rather than split into those references, leaving minor organization gaps.

4 / 5

Total

15

/

20

Passed

Description

82%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that clearly states what the skill does and when to use it, with good trigger terms and a distinct niche. The main weakness is the second-person 'you need' phrasing, which violates the third-person-voice guideline.

Suggestions

Reword 'Use when you need multilingual support...' in third person (e.g., 'Use when working with multilingual text, CJK languages, or when reproducible tokenization is required') to match the rubric's voice convention.

Add a common synonym or file extension (e.g., 'subword tokenization', '.model files') to broaden natural trigger coverage.

DimensionReasoningScore

Specificity

Names the domain and several concrete capabilities (BPE, Unigram, training on raw text, deterministic vocabulary), but the capabilities read more as a feature list than comprehensive concrete actions; the second-person 'Use when you need' triggers the voice penalty (-1).

3 / 5

Completeness

Clearly answers both 'what' (language-independent tokenizer treating text as raw Unicode, supports BPE/Unigram) and 'when' (Use when you need multilingual support, CJK languages, or reproducible tokenization) with concrete trigger phrases.

5 / 5

Trigger Term Quality

Good natural keyword coverage ('multilingual support', 'CJK languages', 'reproducible tokenization', 'BPE', 'Unigram') with a clear 'Use when' trigger, but misses common synonyms/extensions like 'subword', 'spm', or '.model'.

4 / 5

Distinctiveness Conflict Risk

Occupies a clear niche (SentencePiece-style language-independent tokenization) with distinct triggers and named consumer models (T5, ALBERT, XLNet, mBART), giving minimal conflict risk with other skills.

5 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository
OpenLAIR/dr-claw
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.