llm-evaluation

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

1.75x

Quality

74%

Does it follow best practices?

Impact

91%

1.75x

Average score across 3 eval scenarios

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./tests/ext_conformance/artifacts/agents-wshobson/llm-application-dev/skills/llm-evaluation/SKILL.md

The canonical home for this skill is llm-evaluation in wshobson/agents

Quality

Content

65%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, code-rich reference for LLM evaluation that scores well on actionability but is weakened by token spent restating concepts Claude already knows and by a monolithic single-file structure with no bundle references or validation-gated workflows.

Suggestions

Trim glossary-style definitions of well-known metrics (BLEU, ROUGE, perplexity, precision/recall) and section tag-lines; assume Claude's knowledge and keep only the executable code and any non-obvious guidance.

Move the longer integration sections (LangSmith, full metric implementations, benchmarking) into separate reference files under references/ and link to them from SKILL.md so the body is a concise overview.

Add explicit validation/feedback-loop steps to the batch-oriented workflows (e.g., regression detection and benchmarking) so failures are caught before results are reported.

Dimension	Reasoning	Score
Conciseness	The body is mostly code-forward and efficient, but it spends tokens explaining concepts Claude already knows (e.g., glossary-style metric definitions like 'BLEU: N-gram overlap (translation)', 'Perplexity: Language model confidence') and section tag-lines such as 'Fast, repeatable, scalable evaluation using computed scores.'.	3 / 5
Actionability	Provides fully executable, copy-paste-ready code across the common cases (BLEU, ROUGE, BERTScore, custom metrics, LLM-as-judge pointwise/pairwise/reference, inter-rater agreement, A/B testing, regression detection, LangSmith integration, benchmarking) with concrete function signatures and example usage.	5 / 5
Workflow Clarity	Content is organized as a topical reference of code snippets rather than a sequenced multi-step process; the Quick Start and Regression Detection sections give a rough sequence but there are no explicit validation checkpoints or feedback loops, and batch/destructive evaluation workflows lack validation steps.	3 / 5
Progressive Disclosure	No bundle files exist (references/, scripts/, assets/ are absent), so the entire ~690-line body is inlined monolithically in SKILL.md; section headers provide some structure, but substantial content that could live in separate reference files (LangSmith integration, full metric catalog) is inlined with only external links under Resources.	3 / 5
	Total	14 / 20 Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that clearly states both capabilities and usage triggers in third person, with concrete natural-language trigger phrases. Minor room to improve by enumerating more granular actions and adding synonym variants of the trigger terms.

Suggestions

Replace umbrella categories ('automated metrics, human feedback, and benchmarking') with more granular concrete actions (e.g., 'compute BLEU/ROUGE/BERTScore, run LLM-as-judge pairwise comparisons, run A/B tests with statistical significance').

Add common synonym trigger terms (e.g., 'evals', 'model evaluation', 'AI quality testing') to broaden natural-keyword coverage.

Dimension	Reasoning	Score
Specificity	Names the LLM-evaluation domain and several concrete capability areas ('automated metrics, human feedback, and benchmarking'), but the actions are umbrella categories rather than the granular, enumerated concrete actions in the anchor-5 example.	4 / 5
Completeness	Explicitly answers both 'what' ('Implement comprehensive evaluation strategies...') and 'when' ('Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Includes natural phrases a user would say ('testing LLM performance', 'measuring AI application quality', 'establishing evaluation frameworks'); good coverage but missing common synonyms and variations that would push it to 5.	4 / 5
Distinctiveness Conflict Risk	The LLM-evaluation niche has distinct triggers and low conflict risk, but 'evaluation' is broad enough to carry minor overlap risk with general testing or QA skills, keeping it below 5.	4 / 5
	Total	17 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
skill_md_line_count	SKILL.md is long (696 lines); consider splitting into references/ and linking	Warning

	Total	15 / 16 Passed

Repository: Dicklesworthstone/pi_agent_rust
Path: tests/ext_conformance/artifacts/agents-wshobson/llm-application-dev/skills/llm-evaluation/SKILL.md
Commit: 6453c0b

Reviewed: about 8 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.