ml-benchmark-evaluation

Rigorous methodology for evaluating ML models on established benchmarks. Covers proper train/val/test splits, baseline verification from original papers, exact metric formula discrepancies, data-leak detection checklist, multi-seed robustness, and honest reporting templates. Use when claiming to beat published baselines, writing methods papers, or auditing existing results.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable content with a clear sequenced workflow and strong validation checkpoints, but it is a long single-file document with no progressive disclosure, leaving some detailed reference material inlined rather than split out.

Suggestions

Move the detailed metric formula discrepancy comparison (Formulas A/B/C) and the physics-informed validation table into a reference file (e.g., references/metrics.md, references/physics-validation.md) and link to them from SKILL.md to reduce the body length and improve navigation.

Trim or remove the 'Common Pitfalls' table, since each row restates guidance already covered in sections 1-5; keeping only pitfalls not already discussed would cut redundancy.

Add a brief 'Quick start' or 'Minimum viable evaluation' summary at the top pointing to the detailed sections, so a reader can grasp the core protocol before the full methodology.

Dimension	Reasoning	Score
Conciseness	Mostly efficient with executable code and domain-specific guidance that earns its place, but the 'Common Pitfalls' table repeats points already made in sections 1-5 and a few explanatory sentences ('Single-seed results can be lucky') could be trimmed.	4 / 5
Actionability	Fully executable, copy-paste-ready code throughout — train/val/test slicing, assert-based leak checks, a complete nRMSE function, a multi-seed loop, and a concrete reporting template cover the common cases.	5 / 5
Workflow Clarity	A numbered 1-7 sequence with an explicit validation gate ('Run these 6 checks before reporting any result'), assert-based checkpoints, and feedback loops ('If in doubt, report BOTH formulas', re-validate) satisfies the checklist-and-feedback anchor.	5 / 5
Progressive Disclosure	Well-sectioned internally but a ~190-line monolithic file with zero external references; detailed material like the A/B/C metric formula comparison and the physics-informed validation table is inlined where a one-level-deep reference file would keep SKILL.md leaner.	3 / 5
	Total	17 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that clearly states both capabilities and trigger conditions with concrete, domain-appropriate language. The only gap is moderate synonym coverage in the trigger terms.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — 'train/val/test splits, baseline verification from original papers, exact metric formula discrepancies, data-leak detection checklist, multi-seed robustness, and honest reporting templates' — giving comprehensive coverage of the skill's capabilities.	5 / 5
Completeness	Explicitly answers both 'what' (rigorous methodology for evaluating ML models on benchmarks with the listed coverage) and 'when' ('Use when claiming to beat published baselines, writing methods papers, or auditing existing results') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Natural triggers like 'claiming to beat published baselines', 'writing methods papers', and 'auditing existing results' read well, but common synonyms such as 'reproducibility', 'benchmarking', or 'comparing models' are missing.	4 / 5
Distinctiveness Conflict Risk	Carves a clear niche (ML benchmark evaluation rigor) with distinct triggers unlikely to fire for unrelated skills; minimal overlap risk with generic training or analysis skills.	5 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/ml-benchmark-evaluation/SKILL.md
Commit: 3a6c3a9

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.