model-economics

Cost modeling and ROI analysis for specialized LLM development. Use when deciding whether to train a custom model, estimating total cost, or calculating break-even vs frontier APIs. Covers training costs, inference costs, and time-to-ROI projections.

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A data-rich, actionable skill with executable calculators and concrete cost tables. Its main weaknesses are time-sensitive inlined pricing that should be flagged or split out and the absence of a progressive-disclosure file structure.

Suggestions

Move the dated model pricing tables into a separate references file (e.g., references/frontier-pricing.md) and keep only a concise tier summary plus a pointer in SKILL.md, both to improve progressive disclosure and to isolate time-sensitive data.

Add a 'last verified' note or deprecated/old-patterns framing around the specific model names and prices so stale data does not mislead and does not penalize conciseness.

Add an explicit validation/checkpoint step after the break-even calculator (e.g., 'If viable=False, stop and report the reason; otherwise continue to the ROI timeline template') to close the feedback loop.

Dimension	Reasoning	Score
Conciseness	Mostly efficient with dense, useful tables, but inlines time-sensitive model-specific pricing ('Reference tiers as of early 2026', specific dated model names and dollar figures) without a deprecated/old-patterns framing, which the rubric penalizes, alongside some explanatory prose that could be trimmed.	3 / 5
Actionability	Provides two fully executable Python functions with docstrings plus a concrete example call showing inputs and expected output ('Break-even in ~1.1 months, $52K annual savings'), and tables of concrete cost numbers — copy-paste ready covering common cases.	5 / 5
Workflow Clarity	The Quick Assessment Questions give a sequenced triage (spend, task constraint, data, quality bar) leading into the decision framework and break-even calculator, but validation checkpoints on the calculator output (e.g., handling viable=False) are only implicit rather than explicit feedback loops.	4 / 5
Progressive Disclosure	Well sectioned with clear headers, but no bundle files exist and substantial inlined reference pricing tables that would benefit from a separate references file remain inline; the one external pointer ('Load `tinker-training-cost` skill') points to another skill rather than a local reference, so navigation is not clearly signaled.	3 / 5
	Total	15 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that clearly states both the capability and the trigger conditions. It names concrete actions and natural trigger phrases with minimal conflict risk.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — 'Cost modeling and ROI analysis', 'deciding whether to train a custom model', 'estimating total cost', 'calculating break-even vs frontier APIs', 'Covers training costs, inference costs, and time-to-ROI projections' — giving comprehensive coverage of the domain.	5 / 5
Completeness	Explicitly answers both what ('Cost modeling and ROI analysis for specialized LLM development') and when ('Use when deciding whether to train a custom model, estimating total cost, or calculating break-even vs frontier APIs') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Includes natural trigger phrases users would say ('deciding whether to train a custom model', 'estimating total cost', 'calculating break-even vs frontier APIs') with good synonym coverage, but is missing a few common variations such as 'fine-tuning cost'.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (specialized model training economics and break-even vs frontier APIs) with distinct triggers, creating minimal overlap risk with other skills.	5 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/model-economics/SKILL.md
Commit: 3a6c3a9

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.