spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Quality

—

Does it follow best practices?

Impact

—

No eval scenarios have been run

Securityby

Passed

No known issues

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable skill body with good progressive disclosure into references/details.md. It loses points for explaining basic Spark execution concepts Claude already knows and for lacking explicit validation checkpoints in its optimization workflow.

Suggestions

Remove or condense the "Spark Execution Model" diagram and Key Performance Factors table — these restate Spark fundamentals Claude already knows; keep only non-obvious tuning guidance.

Add a short "Verify results" step to the optimization flow, e.g. inspect the Spark UI for skew/spills/GC after running, and re-measure job time before and after each change.

Tighten the intro paragraph and "When to Use" list, which overlap with the frontmatter description, to reclaim tokens.

Dimension	Reasoning	Score
Conciseness	Mostly efficient (lean Quick Start, actionable do/don't list), but the "Spark Execution Model" diagram and the Key Performance Factors table explain concepts Claude already knows and could be trimmed.	2 / 3
Actionability	The Quick Start provides fully executable, copy-paste-ready PySpark code with concrete config keys, and best-practice bullets give specific values (e.g. "128MB - 256MB per partition").	3 / 3
Workflow Clarity	Content is well-organized by topic but there is no sequenced optimization workflow with validation/verification checkpoints (e.g. check Spark UI for skew after running), which the rubric expects for batch-oriented operations.	2 / 3
Progressive Disclosure	SKILL.md is a concise overview that clearly signals one-level-deep detail in references/details.md (verified to exist), with easy navigation and appropriate content splitting.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A concise, well-structured description that names specific capabilities, provides explicit trigger guidance, and carves out a distinct Spark-performance niche. No notable weaknesses.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — "partitioning, caching, shuffle optimization, and memory tuning" — mirroring the score-3 example of distinct named capabilities rather than vague abstractions.	3 / 3
Completeness	Explicitly answers both what ("Optimize Apache Spark jobs with...") and when via a dedicated "Use when..." clause, matching the score-3 anchor.	3 / 3
Trigger Term Quality	Natural trigger terms a user would say — "improving Spark performance", "debugging slow jobs", "scaling data processing pipelines" — give good coverage of common phrasings.	3 / 3
Distinctiveness Conflict Risk	Apache Spark optimization is a clear niche with Spark-specific triggers, making conflict with unrelated skills unlikely.	3 / 3
	Total	12 / 12 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: wshobson/agents
Commit: 5cc2549

Reviewed: about 4 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.