spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

1.28x

Quality

78%

Does it follow best practices?

Impact

77%

1.28x

Average score across 3 eval scenarios

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./tests/ext_conformance/artifacts/agents-wshobson/data-engineering/skills/spark-optimization/SKILL.md

The canonical home for this skill is spark-optimization in wshobson/agents

Quality

Content

65%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with excellent executable code coverage, but it carries some concept over-explanation, lacks validation checkpoints around its destructive write operations, and keeps all detail in a single inlined file.

Suggestions

Trim or remove the 'Spark Execution Model' diagram and 'Storage levels explained' block — Claude already knows these concepts — to improve token efficiency.

Add a validation/verification checkpoint for destructive writes (e.g. verify output row counts or schema before and after '.write.mode("overwrite")') to raise workflow clarity above 3.

Move the detailed pattern code or the full configuration cheat sheet into a separate reference file (e.g. references/CONFIG.md) and link to it from the overview to enable progressive disclosure.

Dimension	Reasoning	Score
Conciseness	Mostly efficient and dominated by actionable code, but includes concept explanations Claude already knows (the 'Spark Execution Model' diagram, the 'Storage levels explained' block) and a redundant 'When to Use This Skill' list that restates the description.	3 / 5
Actionability	Provides copy-paste ready, executable PySpark code with specific config keys and functions across all seven patterns, covering the common optimization cases comprehensively.	5 / 5
Workflow Clarity	This is a pattern catalog rather than a sequenced workflow, and the destructive/batch write examples (e.g. '.write.mode("overwrite")') lack validation or verification checkpoints, which caps workflow clarity per the rubric.	3 / 5
Progressive Disclosure	Sections are clearly organized with headers, but the ~415-line body is monolithic with all patterns, the config cheat sheet, and best practices inlined and no one-level-deep reference files to offload detail.	3 / 5
	Total	14 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is strong: it names the domain, lists concrete optimization actions, and provides explicit 'Use when' trigger guidance. The only minor gap is keyword synonym coverage in the trigger terms.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — 'partitioning, caching, shuffle optimization, and memory tuning' — giving comprehensive coverage of the optimization domain rather than vague language.	5 / 5
Completeness	Clearly states what it does ('Optimize Apache Spark jobs with...') and when to use it ('Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Natural phrases like 'improving Spark performance', 'debugging slow jobs', 'scaling data processing pipelines' are present, but a few common synonyms/variations are missing, so it is not quite comprehensive.	4 / 5
Distinctiveness Conflict Risk	'Apache Spark jobs' is a clear niche with distinct triggers, and the listed performance scenarios are unlikely to overlap with unrelated skills.	5 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: Dicklesworthstone/pi_agent_rust
Path: tests/ext_conformance/artifacts/agents-wshobson/data-engineering/skills/spark-optimization/SKILL.md
Commit: 6453c0b

Reviewed: about 9 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.