Name: huggingface-community-evals
Rating: 68 (1 reviews)
Author: huggingface

huggingface-community-evals

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill body with copy-paste commands and a clear workflow, weakened mainly by repeated handoff guidance and a dangling examples/USAGE_EXAMPLES.md reference. Tightening redundancy and creating (or removing) the examples file would lift its weakest dimensions.

Suggestions

Consolidate the 'hand off to hugging-face-jobs' guidance into one place (e.g., the Remote Execution Boundary section) instead of repeating it in Overview, Prerequisites, Core Workflow, Hardware Guidance, and Troubleshooting.

Create examples/USAGE_EXAMPLES.md or remove the dangling references to it in the script-selection table and Examples section so all cited paths resolve.

Explicitly wire the smoke-test failure loop into Core Workflow (e.g., 'if the smoke test fails, debug via Troubleshooting, then re-run before scaling up') to close the validation gap.

Dimension	Reasoning	Score
Conciseness	The body assumes Claude's competence (no 'vLLM is a…' concept explanations) and is mostly lean commands and tables, but the 'hand off to hugging-face-jobs' guidance is repeated in ~4-5 places and Backend Selection restates backend choices already given in Quick Start, so it could be tightened rather than being fully efficient.	3 / 5
Actionability	Copy-paste-ready 'uv run scripts/…' commands with complete argument sets (--model, --task, --limit, --tasks, --backend, --trust-remote-code, --use-chat-template, --max-samples) cover all three scripts plus their Transformers/accelerate fallbacks and troubleshooting fixes, matching the fully-executable-covers-common-cases anchor.	5 / 5
Workflow Clarity	The 5-step Core Workflow is clearly sequenced with a smoke-test→scale-up checkpoint and GPU-verify conditional, but the failure-recovery loop (smoke fails → debug via Troubleshooting → re-run → scale) is not explicitly wired into the workflow steps, leaving a minor validation gap below the explicit-feedback-loop 5 anchor.	4 / 5
Progressive Disclosure	Well-sectioned overview with an excellent use-case→script table and one-level-deep references to the three real scripts/ files, but 'examples/USAGE_EXAMPLES.md' is referenced in two places though no examples/ directory exists — a dangling reference that keeps it below the fully-navigable 5 anchor.	4 / 5
	Total	16 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that cleanly states what it does, when to use it, and what to avoid, with named frameworks and backends. Its only weak spot is missing common synonyms (benchmark, evals) in the trigger terms.

Dimension	Reasoning	Score
Specificity	Names the domain (HF Hub model evals on local hardware) and lists several concrete actions with specific tools — 'using inspect-ai and lighteval', 'backend selection', 'choosing between vLLM / Transformers / accelerate' — with only minor coverage gaps, fitting the 'lists several specific actions' anchor rather than the 1-2-action anchor below.	4 / 5
Completeness	Explicitly answers both what ('Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware') and when via a concrete 'Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate' clause plus a 'Not for…' anti-trigger list, matching the explicit-what-and-when-with-concrete-triggers anchor.	5 / 5
Trigger Term Quality	Good natural-keyword coverage ('evaluations', 'Hugging Face Hub models', 'local GPU evals', 'vLLM', 'inspect-ai', 'lighteval'), but common synonyms users actually say like 'benchmark'/'benchmarks' and 'evals' are missing, so it falls short of the comprehensive-synonym 5 anchor.	4 / 5
Distinctiveness Conflict Risk	The 'on local hardware' constraint carves a clear niche and the 'Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation' exclusion minimizes overlap with sibling skills, matching the clear-niche-minimal-conflict anchor.	5 / 5
	Total	18 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: huggingface/skills
Path: skills/huggingface-community-evals/SKILL.md
Commit: 97862b0

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.