Name: llm-evaluation
Rating: 52.8 (1 reviews)
Author: administrakt0r

llm-evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

Quality

59%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./plugins/AI-Agents-Safe-Coding-Skills-claude/skills/llm-evaluation/SKILL.md

The canonical home for this skill is llm-evaluation in administrakt0r/AI-Agents-Safe-Coding-Skills

Quality

Content

57%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body provides broad, mostly-executable code across LLM evaluation techniques, but it inlines reference-level detail into the main skill file while citing bundle files that do not exist, and it lacks a concrete sequenced evaluation workflow with validation checkpoints. Tightening into an overview plus real reference files, and fixing the non-executable Quick Start, would raise the weaker dimensions.

Suggestions

Move the detailed metric, A/B-testing, regression, and benchmarking code into the referenced files (references/*.md, assets/evaluation-framework.py) and actually create those files, keeping SKILL.md a concise overview with one-level-deep pointers.

Add a concrete sequenced evaluation workflow with validation checkpoints (e.g., 1. define metrics + test set, 2. run baseline, 3. validate scores/variance, 4. compare variant with statistical test) instead of the generic three-line Instructions.

Fix the Quick Start to use real, installable libraries (or remove the fictional `from llm_eval import EvaluationSuite, Metric` harness) and update the LLM-judge example to the current OpenAI SDK so the headline example is copy-paste runnable.

Dimension	Reasoning	Score
Conciseness	The body is ~480 lines with ~9 substantial inlined code sections (BLEU, ROUGE, BERTScore, custom metrics, LLM-judge, human eval, A/B testing, regression, benchmarking) that the skill's own Resources list says belong in separate reference files, so the main file duplicates content it claims to offload.	3 / 5
Actionability	Most metric implementations use real libraries (nltk, rouge_score, bert_score, detoxify) and are executable, but the Quick Start relies on a fictional `llm_eval` package, `calculate_factuality` is a stub, and the LLM-judge code uses the deprecated `openai.ChatCompletion.create` style with a 'gpt-5' model string.	4 / 5
Workflow Clarity	The Instructions offer only a vague generic sequence ('Clarify goals... Apply best practices... Provide steps') with no concrete evaluation workflow, and there are no validation checkpoints for batch evaluation/A-B testing runs, which the rubric caps at 3.	3 / 5
Progressive Disclosure	The file is well-sectioned and lists a clear Resources section, but the referenced bundle files (references/metrics.md, assets/evaluation-framework.py, scripts/evaluate-model.py, resources/implementation-playbook.md) do not exist, and the detailed content they would hold is inlined in SKILL.md, leaving navigation broken.	3 / 5
	Total	13 / 20 Passed

Description

61%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description clearly identifies a distinct LLM-evaluation niche with decent natural trigger terms, but it lacks a 'Use when...' clause and leans on category labels rather than concrete actions. Adding explicit trigger guidance and specific verbs would lift the capped completeness and specificity scores.

Suggestions

Append a 'Use when...' clause with concrete trigger phrases (e.g., 'Use when measuring LLM application performance, comparing models or prompts, or detecting regressions before deployment') to satisfy the completeness requirement.

Replace the generic 'Master comprehensive evaluation strategies' verb with concrete actions such as 'compute automated metrics (BLEU, ROUGE, BERTScore), run human annotation, and A/B test models'.

Add common natural trigger terms users say, such as 'LLM-as-judge', 'benchmarking', and 'regression testing'.

Dimension	Reasoning	Score
Specificity	The description names the domain ('evaluation strategies for LLM applications') and three evaluation modalities ('automated metrics', 'human evaluation', 'A/B testing'), but 'Master comprehensive evaluation strategies' is a generic verb with no concrete actions like 'compute BLEU' or 'build annotation tasks'.	3 / 5
Completeness	It clearly states 'what' (master evaluation strategies across metrics/human/A-B testing) but has no 'Use when...' or equivalent trigger guidance, which per the rubric caps completeness at 3.	3 / 5
Trigger Term Quality	Natural terms a user would say are present ('LLM applications', 'automated metrics', 'human evaluation', 'A/B testing'), though common variations like 'LLM-as-judge', 'benchmarking', and 'regression testing' are missing.	4 / 5
Distinctiveness Conflict Risk	'LLM evaluation strategies' is a fairly distinct niche with minor overlap risk against general ML-testing skills, though the lack of explicit trigger phrases keeps it just below a 5.	4 / 5
	Total	14 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning
referenced_paths_exist	Referenced path issues: 8 missing	Warning

	Total	14 / 16 Passed

Repository: administrakt0r/AI-Agents-Safe-Coding-Skills
Path: plugins/AI-Agents-Safe-Coding-Skills-claude/skills/llm-evaluation/SKILL.md
Commit: 57c135b

Reviewed: 4 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.