update-llm-model-list

Audit and update the supported LLM model list in assets.py against litellm's registry (models.litellm.ai). Use when adding new models, pruning outdated ones, or verifying the list is correct.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A dense, executable body with real domain expertise and a clear step sequence, held back mainly by structure: the two audit scripts should live as bundled script files rather than inline heredocs. Workflow validation is present but lacks an explicit fix-retry loop.

Suggestions

Move the two inline heredoc scripts (/tmp/check_agenta_models.py and /tmp/find_missing.py) into bundled scripts/ files and reference them by path, keeping SKILL.md as a lean overview.

Add an explicit fix-and-revalidate loop after Step 4 (e.g. "If the check fails, fix the listed models per Key rules and rerun until it passes") to close the workflow feedback gap.

Make the scripts fully copy-paste ready by having them import supported_llm_models directly (or read assets.py) instead of the 'paste here' / 'fill in the AGENTA_* sets' placeholders.

Dimension	Reasoning	Score
Conciseness	Mostly lean and dense with genuinely non-obvious domain knowledge (prefix-stripping rules, LITELLM_LOCAL_MODEL_COST_MAP pinning), but the two long inline heredoc scripts and a few asides ("no local install needed") carry minor padding that could be trimmed.	4 / 5
Actionability	Copy-paste-ready uvx/pytest/ruff commands with self-contained PEP-723 script headers, but placeholders like "paste supported_llm_models here or import it" and "Fill in the AGENTA_* sets" leave small gaps the operator must resolve before execution.	4 / 5
Workflow Clarity	Steps 1-4 are clearly sequenced with explicit validation ("All checks must pass before committing", sys.exit(1) naming each missing model), but there is no explicit fix-and-revalidate retry loop or checklist, so it sits at the 4 rather than the 5 anchor.	4 / 5
Progressive Disclosure	No bundle files exist and both audit scripts (30-50 lines each) are inlined as heredocs in SKILL.md — content that clearly belongs in a scripts/ file is inline — despite good section headers and a Related files table.	3 / 5
	Total	15 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete what and when, specific artifacts, and natural trigger phrases in third-person voice. Only minor headroom in trigger synonym coverage.

Dimension	Reasoning	Score
Specificity	"Audit and update the supported LLM model list in assets.py against litellm's registry" names concrete artifacts (assets.py, litellm registry) and multiple specific actions — audit, update, prune, verify — giving comprehensive coverage of the task lifecycle.	5 / 5
Completeness	Explicitly answers what ("Audit and update the supported LLM model list in assets.py against litellm's registry") and when ("Use when adding new models, pruning outdated ones, or verifying the list is correct") with concrete trigger phrases.	5 / 5
Trigger Term Quality	"adding new models, pruning outdated ones, or verifying the list is correct" gives good natural trigger phrases, but a few common variations are missing (e.g. "model list is stale", "new provider", file-name triggers).	4 / 5
Distinctiveness Conflict Risk	Anchored to one specific file (assets.py) and one external source (litellm's registry / models.litellm.ai), a clear niche with minimal overlap risk against other skills.	5 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: Agenta-AI/agenta
Path: .agents/skills/update-llm-model-list/SKILL.md
Commit: 7a975fb

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.