langsmith-evaluator

INVOKE THIS SKILL when building evaluation pipelines for LangSmith. Covers three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2) Defining Run Functions - how to capture outputs and trajectories from your agent; (3) Running Evaluations - locally with evaluate() or auto-run via LangSmith. Uses the langsmith CLI tool.

Quality

75%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./.agents/skills/langsmith-evaluator/SKILL.md

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable with executable bilingual examples and real validation feedback loops, weakened by repetition of the inspect-first message and no progressive disclosure into reference files for a skill this large.

Suggestions

Consolidate the repeated 'inspect before you implement' guidance into the single crucial_requirement block and reference it rather than restating it in code_evaluators, run_functions, and troubleshooting.

Split the detailed per-evaluator code examples into a references/ file (e.g., EVALUATOR_EXAMPLES.md) and keep SKILL.md as a lean overview with one-level-deep links, improving progressive disclosure.

Add a single end-to-end numbered workflow (inspect → define run function → write evaluators → upload/run) to unify the fragmented sequences.

Dimension	Reasoning	Score
Conciseness	Mostly efficient code-focused content, but the 'inspect before you implement' Golden Rule is restated across crucial_requirement, code_evaluators, run_functions, and troubleshooting, and prose like 'Try your hardest to match your run function output to your dataset schema' could be tightened.	3 / 5
Actionability	Fully executable copy-paste Python and TypeScript examples plus concrete CLI commands cover the common cases (LLM judge, code evaluator, run functions, upload, evaluate); placeholders like YOUR_TRAJECTORY_FIELD are explicitly justified by the inspect-first rule.	5 / 5
Workflow Clarity	Clear inspect-first sequence and a debugging workflow with validate→fix→retry feedback loops plus a best-practices checklist, but the end-to-end flow is fragmented across tagged sections rather than one coherent numbered pipeline.	4 / 5
Progressive Disclosure	Well-organized into tagged sections, but at ~360 lines all content is inlined in a single SKILL.md with no bundle/reference files to offload detail; the simple-skill exception does not apply given the length.	3 / 5
	Total	15 / 20 Passed

Description

80%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description with explicit invocation guidance and a clear LangSmith evaluation niche. Its main gap is a single trigger phrase rather than a richer set of natural synonyms and trigger conditions.

Suggestions

Expand the 'when' clause to list multiple natural trigger phrases (e.g., 'when the user wants to evaluate an agent, grade outputs, set up evals, or compare runs in LangSmith').

Add common synonyms like 'evals', 'grade agent outputs', or 'test my agent' to improve trigger-term coverage.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions across three named components — 'Creating Evaluators - LLM-as-Judge, custom code', 'Defining Run Functions - capture outputs and trajectories', 'Running Evaluations - locally with evaluate() or auto-run' — giving comprehensive coverage.	5 / 5
Completeness	Clearly answers 'what' (three core components enumerated) and has an explicit 'when' ('INVOKE THIS SKILL when building evaluation pipelines for LangSmith'), but the 'when' is a single trigger condition rather than the multiple concrete trigger phrases of the 5 anchor.	4 / 5
Trigger Term Quality	Good natural-term coverage ('building evaluation pipelines', 'LangSmith', 'LLM-as-Judge', 'evaluators', 'evaluate()') but misses common synonyms a user might say like 'grade my agent', 'test my agent', or 'evals'.	4 / 5
Distinctiveness Conflict Risk	The LangSmith-specific evaluation-pipeline niche is mostly distinct with clear triggers, though there is minor overlap risk with broader LangSmith tracing or general agent-evaluation skills.	4 / 5
	Total	17 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: hefgi/dotfiles
Path: .agents/skills/langsmith-evaluator/SKILL.md
Commit: e6f304f

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.