CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

pxi-eval-dataset

Generate synthetic evaluation datasets for the PXI eval harness (evals/pxi/). Use whenever the user asks to create, author, draft, expand, or audit an eval dataset for a PXI tool, skill, or behavior — including phrases like "write evals for <tool>", "test PXI behavior", "synthetic dataset for PXI", "cover this tool with eval examples", or "find gaps in our PXI eval coverage". Inspects whichever evaluators currently live under evals/pxi/evaluators/ at use time and pauses to recommend a new evaluator if the behavior under test can't be scored by what already exists.

72

Quality

91%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

81%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, deeply actionable workflow skill: every step has concrete commands, paths, and examples, and validation/triage feedback loops are exemplary. The main costs are rationale-heavy passages that pad token budget and large inline blocks (the annotation prompt template, matching-semantics reference) that belong in one-level-deep reference files.

Suggestions

Trim the 'why' rationale blocks (context-rot explanation in step 6, 'Why this matters:' after the agreement rule, and the paragraph on framing the task as a goal with constraints) to one sentence each — Claude needs the rule, not the design-history justification.

Move the verbatim annotation task-instruction prompt template (step 6, 'Task instruction') into a reference file (e.g. references/annotation_prompt_template.md) and keep a short 'include verbatim' pointer plus the schema slot in the body.

Move the evaluator matching-semantics section ('Subset match', 'Variant match', 'Any-of match', 'Order-independent conjunctions') into references/matching_semantics.md next to the dataset schema reference, leaving a one-line pointer in SKILL.md.

DimensionReasoningScore

Conciseness

The bulk is dense, harness-specific instruction Claude cannot infer (evaluator discovery rules, annotation fan-out protocol, triage categories, YAML gotchas), but there are several passages of rationale that could be trimmed: the context-rot explanation in step 6, 'Why this matters:' after the agreement rule, and meta-commentary like 'This frames the task as a goal with constraints, not a procedural recipe. Cross-model fan-out is only useful if...'. Not 2 because almost none of it re-teaches concepts Claude already knows (no 'what is YAML' padding); not 4 because the repeated just-it-yourself-rationale blocks appear in at least three places and read as over-explanation of the author's design choices.

3 / 5

Actionability

Fully executable throughout: exact commands ('uv run python -c "from evals.pxi.harness.datasets import load_dataset; load_dataset(\'<name>\')"', 'uv run python -m evals.pxi.harness.run_experiment --dataset <name>'), a per-model invocation table, a working bash example for the Codex wrapper, concrete YAML blocks for variant lists and metadata, and specific file paths for every artifact. Not 4 because the concrete examples cover the common cases end-to-end with nothing left pseudocode.

5 / 5

Workflow Clarity

A numbered 7-step workflow with explicit validation checkpoints (step 7: 'Parse + validate schema', 'Run the experiment end-to-end and triage every failure before calling the dataset done'), a three-bucket failure triage table, a fix-order rule ('harness/evaluator issues first, dataset issues second'), and a re-run loop ('Re-run the full experiment after each batch of fixes. Stop when...'). This is a batch operation and validation is present, so the cap does not apply. Not 4 because error-recovery feedback loops and checklists (coverage dimensions in step 4) are explicit, not implied.

5 / 5

Progressive Disclosure

Good structure: clearly headed sections, a verified one-level-deep script reference ('scripts/annotate_via_codex.sh' exists and its flags in the body match the file), and pointers to live repo files instead of duplicated schemas ('Do not copy a schema from this skill into prompts'). Not 5 because some content that would sit better in a reference file is inlined in SKILL.md — notably the verbatim ~40-line annotation task-instruction prompt template and the evaluator matching-semantics reference — making the body a ~575-line near-monolith; not 3 because what is inlined is well-signaled with headers and the only bundle reference is real and clearly linked.

4 / 5

Total

17

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: third-person, concrete actions, explicit 'Use whenever...' trigger guidance with natural quoted phrasings, and a tightly scoped niche with boundary behavior stated. No vague fluff or over-claims.

DimensionReasoningScore

Specificity

Multiple concrete actions are named: 'Generate synthetic evaluation datasets for the PXI eval harness (evals/pxi/)', 'Inspects whichever evaluators currently live under evals/pxi/evaluators/', and 'pauses to recommend a new evaluator'. Coverage is comprehensive for the skill's niche. Not 4 because no meaningful action within scope is left out — even the boundary behavior (recommending a new evaluator on a gap) is explicit.

5 / 5

Completeness

Explicitly answers both questions: the 'what' is 'Generate synthetic evaluation datasets for the PXI eval harness (evals/pxi/)' and the 'when' is 'Use whenever the user asks to create, author, draft, expand, or audit an eval dataset... including phrases like...'. Both are concrete and paired in one tight description. Not 4 because the 'when' clause is fully explicit with trigger phrases, not merely present.

5 / 5

Trigger Term Quality

Quotes natural user phrasings users would actually say: 'write evals for <tool>', 'test PXI behavior', 'synthetic dataset for PXI', 'cover this tool with eval examples', 'find gaps in our PXI eval coverage', plus verb synonyms (create, author, draft, expand, audit) and the concrete path trigger evals/pxi/. Not 4 because both synonyms and concrete phrase-level triggers are present, matching the anchor-5 pattern of PDF files/PDFs/forms/.pdf.

5 / 5

Distinctiveness Conflict Risk

The niche is sharply delimited — PXI eval datasets under evals/pxi/ with named evaluator directories — so it is unlikely to fire for adjacent skills (general eval writing, dataset tooling outside PXI). Not 4 because every trigger phrase is PXI-qualified, leaving minimal overlap risk with closely related skills.

5 / 5

Total

20

/

20

Passed

Validation

87%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

skill_md_line_count

SKILL.md is long (598 lines); consider splitting into references/ and linking

Warning

metadata_field

'metadata' should map string keys to string values

Warning

Total

14

/

16

Passed

Repository

Arize-ai/phoenix

Path

.agents/skills/pxi-eval-dataset/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.