spark-engineer

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.

Quality

96%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Failed to scan

The risk profile of this skill

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is a well-structured Spark skill: executable code, a validation-backed workflow with a retry loop, and a clean reference table pointing to real one-level-deep files. The only weakness is minor verbosity in the restated opener and the term-dump Knowledge Reference section.

Suggestions

Remove the restated "Senior Apache Spark engineer specializing in..." opener since the frontmatter description already covers it.

Delete or condense the "Knowledge Reference" paragraph, which lists concepts (catalyst optimizer, tungsten, accumulators) Claude already knows.

Replace the ellipsis placeholders in the caching example (df.filter(...).withColumn(...)) with concrete expressions so it is fully copy-paste-ready.

Dimension	Reasoning	Score
Conciseness	Code examples and comments are lean and actionable, but the opening "Senior Apache Spark engineer specializing in..." restates the description and the "Knowledge Reference" paragraph dumps terms Claude already knows (catalyst optimizer, tungsten execution engine); both could be trimmed. Not a 3 because the padding is minor relative to the tight code and constraints.	4 / 5
Actionability	Provides copy-paste-ready PySpark for a mini-pipeline, broadcast join, skew salting, and caching, with specific configs (spark.sql.shuffle.partitions=400, AQE) and concrete MUST DO / MUST NOT DO constraints covering the common cases.	5 / 5
Workflow Clarity	The five-step Core Workflow ends with explicit validation ("Check Spark UI for shuffle spill", "verify partition count with df.rdd.getNumPartitions()") and a feedback loop ("if spill or skew detected, return to step 4"), satisfying the batch-operation feedback-loop requirement.	5 / 5
Progressive Disclosure	A Reference Guide table signals five one-level-deep reference files, each with a "Load When" condition; all five referenced files exist in references/ and the overview keeps detailed guidance off the main page. Matches the score-5 anchor of well-signaled, one-level-deep references.	5 / 5
	Total	19 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is exemplary: it pairs an explicit "Use when" trigger clause with a comprehensive list of concrete Spark actions, natural synonyms, and a file extension. It clearly answers both what the skill does and when to invoke it with minimal conflict risk.

Dimension	Reasoning	Score
Specificity	Lists eight concrete actions ("write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics") giving comprehensive coverage of the Spark domain.	5 / 5
Completeness	Explicitly answers "when" ("Use when writing Spark jobs, debugging performance issues, or configuring cluster settings...") and "what" (the "Invoke to..." clause listing concrete actions) with concrete trigger phrases.	5 / 5
Trigger Term Quality	Covers natural user terms including file extensions (".parquet files"), synonyms ("big data workloads", "distributed data processing pipelines"), and specific APIs ("Spark SQL", "RDD", "structured streaming").	5 / 5
Distinctiveness Conflict Risk	Apache Spark is a clear, specific niche with distinct triggers; minimal overlap risk despite adjacent data-engineering skills.	5 / 5
	Total	20 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: jeffallan/claude-skills
Path: skills/spark-engineer/SKILL.md
Commit: 882ef55

Reviewed: 4 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.