CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

huggingface-community-evals

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

Quality

84%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-organized body with executable commands and a sensible smoke-test-first workflow. Its main defects are a dangling reference to a nonexistent `examples/USAGE_EXAMPLES.md` and redundant repetition of the remote-handoff and backend-selection guidance.

Suggestions

Fix or remove the dangling `examples/USAGE_EXAMPLES.md` reference (cited in the script table and the Examples section) — either ship the file or delete both mentions, since following the reference fails today.

Consolidate the remote-handoff guidance: keep the dedicated "Remote Execution Boundary" section and drop the repeated mentions in the Overview and Core Workflow step 5, cutting ~15 lines of redundancy.

Specify what a passing smoke test means (e.g., check the eval log/scores, confirm the task ran with no backend errors) so the 'scale up only after it passes' checkpoint is an explicit validation step.

DimensionReasoningScore

Conciseness

The body is dense and useful — tables, copy-paste commands, no tutorials on concepts Claude already knows — but it is more than minor-trimmable: the hugging-face-jobs handoff is stated four times (Overview twice, Core Workflow step 5, and a dedicated "Remote Execution Boundary" section), and "Backend Selection" largely restates the Quick Start options. That lands at 4 (efficient with over-explanation that could be trimmed) rather than 3, since almost all tokens carry actionable guidance.

4 / 5

Actionability

Every path is copy-paste executable: complete `uv run scripts/... --model ... --task ...` commands for all three options and both fallbacks, concrete prerequisite checks (`nvidia-smi`, `printenv HF_TOKEN`), concrete smoke-test flags (`--limit 20`, `--max-samples 20`), and troubleshooting with exact flags (`--batch-size`, `--gpu-memory-utilization`, `--backend hf`, `--trust-remote-code`). This matches the 5 anchor fully.

5 / 5

Workflow Clarity

The Core Workflow sequences framework choice → backend choice → smoke test → scale up, with an environment validation step in Prerequisites and a troubleshooting section for error recovery. It stays at 4 rather than 5 because "Scale up only after the smoke test passes" never defines what passing means (no check of eval output/scores) — a minor validation gap, not the missing checkpoints of anchor 3.

4 / 5

Progressive Disclosure

The three referenced scripts all exist in the bundle and are well-signaled via the use-case table, but `examples/USAGE_EXAMPLES.md` is referenced twice (script-selection table and Examples section) and does not exist — a broken navigation target. Combined with ~200 lines where task lists and usage patterns arguably belong in that (missing) separate file, this fits the 3 anchor (structure present, references present but flawed/ineffectively organized) rather than 4.

3 / 5

Total

16

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that names concrete actions, explicit triggers, and clear exclusions. The main improvement is making the 'when' clause match concrete user phrasing (e.g., "benchmark", "evaluate locally") rather than abstract use cases.

DimensionReasoningScore

Specificity

"Run evaluations for Hugging Face Hub models using inspect-ai and lighteval" names the domain plus multiple concrete actions, and "backend selection... choosing between vLLM / Transformers / accelerate" enumerates the specific choices the skill makes — comprehensive coverage of the skill's core actions, so it fits the 5 anchor rather than 4 (no meaningful action gaps for this domain).

5 / 5

Completeness

Both parts are explicit: a clear what ("Run evaluations... using inspect-ai and lighteval on local hardware") and an explicit when ("Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate") plus a "Not for" exclusion list. It sits at 4 rather than 5 because the 'when' clause is framed as abstract use cases rather than concrete user-sayable trigger phrases (e.g., "use when the user asks to evaluate or benchmark a Hub model locally"), and above 3 because 'when' is present and specific, not weakly implied.

4 / 5

Trigger Term Quality

Natural terms like "evaluations", "Hugging Face Hub models", "local GPU evals", "inspect-ai", "lighteval", "vLLM / Transformers / accelerate" cover what users would say, but common synonyms such as "benchmark", "evaluate a model", or "run a leaderboard eval" are missing, which matches the 4 anchor (good coverage, a few natural terms missing) rather than 5.

4 / 5

Distinctiveness Conflict Risk

The skill occupies a clear niche (local-hardware evals) and the "Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation" clause explicitly fences off the adjacent skills it could collide with — minimal conflict risk, matching the 5 anchor.

5 / 5

Total

18

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

huggingface/skills

Path

skills/huggingface-community-evals/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.