CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

tinker-training-cost

Calculates training costs for Tinker fine-tuning jobs. Use when estimating costs for Tinker LLM training, counting tokens in datasets, or comparing Tinker model training prices. Tokenizes datasets using the correct model tokenizer and provides accurate cost estimates.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable body — every command, tokenizer ID, data format, and example is executable and copy-paste ready. Its weakness is structure: the large inline pricing and tokenizer-mapping tables (duplicated inside the bundled script) inflate the always-loaded SKILL.md instead of living in a one-level-deep reference file, and a couple of lines over-explain what Claude already knows.

Suggestions

Move the per-model pricing tables and the model-to-tokenizer mapping into a references/pricing.md (one level deep, clearly linked from SKILL.md), keeping only the cost formula, quick-start commands, and one or two example models inline — the same data already exists in the script's MODELS dict, so the tables need not be always in context.

Drop explanations of concepts Claude already knows, e.g., 'different tokenizers produce different token counts for the same text' in Important Notes, and keep only Tinker-specific facts (LoRA-only training, trust_remote_code requirements, VL cost overhead).

Add brief error-recovery guidance to the workflow: what to do when a model name is not in the pricing table, when a JSONL row doesn't match any supported format, or when tokenizer loading fails.

DimensionReasoningScore

Conciseness

The body is largely lean — commands, formula, lookup tables, worked examples — but has minor over-explanation ("different tokenizers produce different token counts for the same text" under Token Counting restates something Claude already knows) and ~60 lines of pricing tables that duplicate the MODELS dict already embedded in scripts/calculate_cost.py, which could be trimmed.

4 / 5

Actionability

Fully executable throughout: copy-paste CLI invocations with real flags (--list-models, --model, --epochs, --json), an explicit cost formula, concrete HuggingFace tokenizer IDs (e.g., meta-llama/Llama-3.1-70B-Instruct), runnable Python, exact JSONL schemas for all three supported formats, and worked numeric examples. Matches the copy-paste-ready top anchor.

5 / 5

Workflow Clarity

The sequence is clear and unambiguous (list models → choose model/tokenizer → run script on dataset), and the script's internal 3-step pipeline is enumerated. Not 5 because there are minor checkpoint gaps: no guidance on what to do when a model name isn't in the table, a JSONL row fails to parse, or a tokenizer download fails. Not 3 because the operation is read-only estimation (not destructive/batch-mutating), so the validation cap doesn't apply and the sequence itself has no real gaps.

4 / 5

Progressive Disclosure

Sections are well-headed and the one bundle file (scripts/calculate_cost.py) is real and correctly referenced, but this is a ~175-line SKILL.md whose bulk is lookup material — the pricing tables and the model-to-tokenizer mapping (together ~75 lines, duplicated in the script's own MODELS dict) are exactly the kind of content the rubric says belongs in a references/ file, loaded only when needed. Matches 'content that should be separate is inline' better than the 'most content appropriately placed' anchor.

3 / 5

Total

16

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete actions, an explicit 'Use when...' clause with multiple concrete triggers, and a distinctive niche anchored by the Tinker name. The only weaknesses are a mildly redundant closing clause and a couple of missing natural phrasings.

DimensionReasoningScore

Specificity

Quotes several concrete actions — "Calculates training costs for Tinker fine-tuning jobs", "Tokenizes datasets using the correct model tokenizer", "counting tokens in datasets", "comparing Tinker model training prices" — but coverage has minor gaps (no mention of listing models/pricing or output formats) and "provides accurate cost estimates" restates the opening claim without adding a new capability, so it sits just below the comprehensive anchor.

4 / 5

Completeness

Explicitly answers both: "what" ("Calculates training costs ... Tokenizes datasets using the correct model tokenizer") and "when" with concrete trigger phrases ("Use when estimating costs for Tinker LLM training, counting tokens in datasets, or comparing Tinker model training prices"). This matches the top anchor's what+when structure with concrete trigger clauses.

5 / 5

Trigger Term Quality

Good natural-term coverage: "estimating costs", "counting tokens in datasets", "comparing ... training prices", "fine-tuning jobs" — phrases users would plausibly say. A few natural variants are missing (e.g., "fine-tuning cost estimate", "how much does it cost to train", "pricing"), keeping it below the comprehensive-synonym anchor.

4 / 5

Distinctiveness Conflict Risk

"Tinker" is a named platform appearing in every clause, carving out a clear niche (Tinker fine-tuning cost estimation) with triggers that would not plausibly fire for unrelated skills — minimal conflict risk.

5 / 5

Total

18

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

synthetic-sciences/openscience

Path

backend/cli/skills/cloud-compute/tinker-training-cost/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.