CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

64

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

65%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured overview with an excellent progressive-disclosure setup and a copy-paste-ready Quick Start. The main costs are re-taught Spark fundamentals (execution model diagram, concept table) and the absence of a sequenced diagnostic workflow with monitoring/validation checkpoints — optimization guidance is presented as patterns to know rather than a process to follow.

Suggestions

Cut the 'Spark Execution Model' section and trim the Key Performance Factors table to only non-obvious guidance; Claude already knows the driver/job/stage/task model.

Add a short numbered diagnostic workflow (e.g., check Spark UI for skew/spills/GC → apply the matching pattern from references/details.md → re-run and verify improvement) to give the patterns a validation loop.

Inline one or two-line code snippets for the highest-value best practices (broadcast join, salting) so the main file is self-sufficient for the most common fixes.

DimensionReasoningScore

Conciseness

Mostly efficient — the Quick Start, factor table, and do's/don'ts are tight — but the 'Spark Execution Model' ASCII diagram and the explanatory factor table re-teach concepts (driver/job/stage/task, what shuffle and skew are) that Claude already knows. Not 4 because a whole section plus part of the table is trimmable over-explanation; not 2 because there is no padded prose or library introductions.

3 / 5

Actionability

The Quick Start is complete, executable PySpark with real config keys (AQE, Kryo, shuffle partitions) and an end-to-end read/transform/write example, and best practices include concrete numbers ('128MB - 256MB per partition'). Not 5 because several recommendations ('Salting, broadcast joins', 'Manual salting') have no inline example — the code lives only in the reference file.

4 / 5

Workflow Clarity

Content is organized by concept rather than as a sequenced process; there is no diagnostic workflow (e.g., inspect Spark UI → identify skew/spill → apply pattern → re-check) and no validation checkpoints, only a one-line 'Monitor Spark UI' bullet. It exceeds 2 because the section flow (when-to-use → concepts → quick start → details → best practices) gives a rough order, but the anchor for 3 — steps present but checkpoints missing — is the best fit.

3 / 5

Progressive Disclosure

SKILL.md is a lean overview (~96 lines) with one clearly signaled, one-level-deep reference: 'Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.' The bundle matches — references/details.md exists (321 lines) and contains no nested .md references — so navigation is easy and appropriately split.

5 / 5

Total

15

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, well-formed description: third-person voice, concrete capability list, and an explicit 'Use when...' trigger clause covering the main use cases. Only minor gaps — a few natural synonyms (PySpark, tuning) are missing, and sub-areas like skew/join optimization aren't mentioned.

DimensionReasoningScore

Specificity

Names the domain ('Apache Spark jobs') and four concrete action areas — 'partitioning, caching, shuffle optimization, and memory tuning' — matching the anchor for several specific actions with minor gaps (join strategy, skew handling, and serialization are absent from the description). Not 5 because coverage stops short of comprehensive; not 3 because far more than 1-2 actions are listed.

4 / 5

Completeness

Explicitly answers both: what — 'Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning'; when — 'Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.' This matches the anchor that clearly and concretely answers both with trigger phrases, and the 'Use when...' clause is present.

5 / 5

Trigger Term Quality

'improving Spark performance', 'debugging slow jobs', 'scaling data processing pipelines' are natural phrases a user would say, giving good keyword coverage. A few natural terms are missing (e.g., 'PySpark', 'tuning', 'slow Spark job'), so it falls just short of the comprehensive synonym/extension coverage of a 5.

4 / 5

Distinctiveness Conflict Risk

The description is anchored to a clear niche (Apache Spark performance) with distinct triggers ('Spark jobs', 'debugging slow jobs') and minimal overlap risk with generic data-processing skills. Not 4 because nothing in the wording could plausibly route a non-Spark request to this skill.

5 / 5

Total

18

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

wshobson/agents

Path

plugins/data-engineering/skills/spark-optimization/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.