langsmith-dataset

INVOKE THIS SKILL when creating evaluation datasets, uploading datasets to LangSmith, or managing existing datasets. Covers dataset types (final_response, single_step, trajectory, RAG), CLI management commands, SDK-based creation, and example management. Uses the langsmith CLI tool.

Quality

83%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with executable commands and code across Python and TypeScript, and is cleanly sectioned for navigation. The main weakness is redundancy — CLI usage is repeated across multiple sections — which inflates token cost without adding guidance.

Suggestions

Consolidate the duplicated CLI command listings in <usage>, <script_usage>, and <example_workflow> into one reference section, linking from the others.

Add an explicit validate-before-destructive-action checkpoint (e.g., run `langsmith dataset get` to confirm the target before `dataset delete`) inside the workflow rather than relying solely on CLI prompts.

Trim or factor out the repeated dataset-structure examples if they duplicate the <dataset_types_overview> descriptions.

Dimension	Reasoning	Score
Conciseness	Mostly efficient with concrete commands and code, but it repeats CLI commands across <usage>, <script_usage>, and <example_workflow> sections and includes some restated guidance, padding the token budget beyond what a single pass would need.	3 / 5
Actionability	Provides copy-paste ready bash commands, complete Python and TypeScript code blocks, concrete JSON dataset structures per type, and explicit flags — fully executable guidance covering the common cases.	5 / 5
Workflow Clarity	The <example_workflow> section lays out a clear numbered sequence (export traces → process → upload → verify → run experiments) with a verification step, and troubleshooting covers failure modes, but destructive operations rely on the CLI's own prompts rather than an explicit validate-then-proceed checkpoint inside the workflow.	4 / 5
Progressive Disclosure	Content is well organized into clearly signaled XML-tagged sections (<setup>, <usage>, <creating_datasets>, <dataset_structures>, etc.) with no nested references and no bundle files, though several sections overlap in content rather than being split into separate referenced files.	4 / 5
	Total	16 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is comprehensive, specific, and explicitly signals when to invoke the skill, covering concrete actions and dataset types. Minor room for improvement lies in adding more colloquial synonyms users might naturally say.

Suggestions

Add natural synonyms a user might say (e.g., 'eval sets', 'test cases', 'golden examples') alongside the formal 'evaluation datasets' phrasing.

Consider trimming the parenthetical enumeration of dataset types if brevity is preferred, though it currently aids specificity.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — 'creating evaluation datasets, uploading datasets to LangSmith, or managing existing datasets' — plus specific coverage of dataset types (final_response, single_step, trajectory, RAG), CLI commands, SDK-based creation, and example management.	5 / 5
Completeness	Clearly answers 'what' (create/manage/upload evaluation datasets, covering four dataset types and CLI/SDK/example management) and 'when' via the explicit 'INVOKE THIS SKILL when creating evaluation datasets, uploading datasets to LangSmith, or managing existing datasets' trigger clause.	5 / 5
Trigger Term Quality	Includes natural terms users would say ('creating evaluation datasets', 'uploading datasets', 'managing existing datasets') and file/tool references (LangSmith, langsmith CLI), but lacks common synonyms or shorthand variations a user might utter (e.g., 'eval sets', 'test cases', 'examples').	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (LangSmith evaluation datasets) with distinct triggers tied to LangSmith and dataset lifecycle tasks, making collision with unrelated skills unlikely.	5 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: hefgi/dotfiles
Path: .agents/skills/langsmith-dataset/SKILL.md
Commit: e6f304f

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.