CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

langsmith-evaluator

INVOKE THIS SKILL when building evaluation pipelines for LangSmith. Covers three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2) Defining Run Functions - how to capture outputs and trajectories from your agent; (3) Running Evaluations - locally with evaluate() or auto-run via LangSmith. Uses the langsmith CLI tool.

68

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable content with executable dual-language evaluator examples, precise CLI commands, and a well-enforced inspect-before-implement workflow including safety gating for destructive operations. The main weaknesses are repeated restatement of the same rules (Golden Rule, RunTree-vs-dict handling), a few garbled sentences, and no progressive disclosure at all — all detail lives inline in one large SKILL.md.

Suggestions

State the Golden Rule and the RunTree-vs-dict handling once each (e.g., in crucial_requirement and the evaluator_format table) and cross-reference them from other sections instead of repeating them four to five times.

Fix the garbled sentences ('Query LangSmith traces to when outputs don't contain needed data...', 'verify against trace to output contains the right data') and the duplicated step number 4 in the run-function debugging workflow.

Move one language's code variants (e.g., TypeScript examples) and the trajectory-capture deep dive into reference files (e.g., references/examples.typescript.md, references/trajectories.md), keeping SKILL.md a lean overview with clearly signaled one-level-deep references.

DimensionReasoningScore

Conciseness

Mostly efficient but with real redundancy: the 'inspect before you implement' Golden Rule is restated in crucial_requirement, code_evaluators, run_functions, the trajectory section, and troubleshooting; the RunTree-vs-dict pattern appears four times (evaluator_format table, both evaluator code sections, troubleshooting). Garbled sentences like 'Query LangSmith traces to when outputs don't contain needed data to understand how to extract from execution' add noise. The repetition goes beyond the 'minor instances' of the 4 anchor, but the body is far from the padded over-explanation of a 2.

3 / 5

Actionability

Fully executable, copy-paste-ready code covering the common cases: complete Python and TypeScript LLM-judge evaluators (with structured-output setup), exact CLI commands ('langsmith evaluator upload ... --dataset "My Dataset" --replace'), and evaluate() invocations in both languages. Placeholders like 'YOUR_TRAJECTORY_FIELD' are explicitly justified ('Replace these placeholders with your actual field names') because output shapes genuinely vary, matching the rubric's flexibility exception.

5 / 5

Workflow Clarity

Clear sequences with most checkpoints present: the Golden Rule (run → inspect → implement), a debugging workflow with print/query/verify steps, 'Test evaluators independently', and safety gating for destructive CLI operations ('NEVER use --yes flag unless the user explicitly requests it'). Falls short of 5 due to the duplicated step number 4/4 in the debugging workflow and garbled checkpoint sentences ('verify against trace to output contains the right data') that muddy the validation steps.

4 / 5

Progressive Disclosure

Good in-file section structure (setup → golden rule → formats → types → examples → upload → run → troubleshoot), but it is a ~360-line monolithic body with no reference files at all, and every example is duplicated per-language (Python + TypeScript blocks in six sections), which is bulk that belongs in reference files. Matches the 3 anchor ('content that should be separate is inline') rather than 4, whose content would be mostly appropriately placed.

3 / 5

Total

15

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it enumerates the skill's three core components with concrete technical specifics, uses an explicit 'INVOKE THIS SKILL when...' trigger clause, and anchors itself firmly in the LangSmith niche. The only weakness is missing colloquial trigger variations such as 'evals', 'grading', or 'benchmarking' that users commonly say.

DimensionReasoningScore

Specificity

Lists multiple specific concrete actions covering the skill comprehensively: 'Creating Evaluators - LLM-as-Judge, custom code', 'Defining Run Functions - how to capture outputs and trajectories from your agent', 'Running Evaluations - locally with evaluate() or auto-run via LangSmith', plus 'Uses the langsmith CLI tool'. It exceeds the 4 anchor because coverage of the skill's core components is complete rather than having minor gaps.

5 / 5

Completeness

Explicitly answers both: what ('Covers three core components: (1) Creating Evaluators... (2) Defining Run Functions... (3) Running Evaluations...') and when ('INVOKE THIS SKILL when building evaluation pipelines for LangSmith'). The when-clause is a concrete, explicit trigger phrase, so it matches the 5 anchor rather than the 4 anchor whose 'when' could be more explicit.

5 / 5

Trigger Term Quality

Good natural keyword coverage: 'evaluation pipelines', 'LLM-as-Judge', 'Evaluators', 'trajectories', 'LangSmith', 'evaluate()'. Falls short of the 5 anchor because common colloquial variations users actually say ('evals', 'grading agent outputs', 'benchmarking') are missing, but exceeds 3 since the included terms are highly relevant and domain-specific.

4 / 5

Distinctiveness Conflict Risk

Clear niche with distinct triggers: scoped to 'evaluation pipelines for LangSmith', naming 'LLM-as-Judge', 'evaluate()', and the 'langsmith CLI tool'. Minimal conflict risk with other skills since the domain, product, and API surface are all uniquely identified.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

hefgi/dotfiles

Path

.agents/skills/langsmith-evaluator/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.