CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

apache-spark-data-processing

Complete guide for Apache Spark data processing including RDDs, DataFrames, Spark SQL, streaming, MLlib, and production deployment

Quality

60%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./plugins/luxor-data-engineering/skills/apache-spark-data-processing/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

38%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is a comprehensive Spark textbook with abundant executable code, but it is massively over-verbose for a skill: it re-explains well-known concepts, inlines all deep-dive material that belongs in separate reference files, and includes a dangling EXAMPLES.md reference. Actionable content is strong; token efficiency and file structure are the weak points.

Suggestions

Split the deep dives (Structured Streaming, MLlib, Production Deployment, Troubleshooting) into separate reference files under references/ with clearly signaled links (e.g., '**Streaming**: See [STREAMING.md](references/STREAMING.md)'), keeping SKILL.md to an overview plus quick-start examples.

Delete or drastically compress concept explanations Claude already knows (RDD/DataFrame definitions, lazy evaluation, what Parquet/CSV are, memory and locality basics), keeping only Spark-specific guidance, heuristics, and gotchas.

Fix the dangling EXAMPLES.md reference (either create the file or remove the claim) and replace placeholder snippets ('agg(...)', 'process_with_salting', 'dimension_df["*"]') with complete, runnable code.

DimensionReasoningScore

Conciseness

The ~1,700-line body extensively re-teaches concepts Claude already knows — 'RDDs are Spark's fundamental data abstraction', 'DataFrames are distributed collections of data organized into named columns - similar to a database table', explanations of what Parquet/ORC/CSV are, memory-region breakdowns, and data-locality definitions. The padding is pervasive across every major section, matching the severely-verbose anchor rather than the 'several unnecessary explanations' of score 2.

1 / 5

Actionability

Most code is concrete and executable (DataFrame creation, streaming query setup, spark-submit commands, ML pipelines), but several snippets contain bare placeholders or undefined functions: 'df_salted.groupBy("salted_key").agg(...)', 'process_with_salting(skewed_df)', invalid 'dimension_df["*"]' syntax, and a streaming example referencing an undefined 'schema' variable. This fits 'mostly executable with minor gaps' — well above pseudocode-level, but not copy-paste ready throughout.

4 / 5

Workflow Clarity

A rough topical arc exists (When to Use → Concepts → Deep dives → Tuning → Deployment → Troubleshooting) plus a 'Development Workflow' list ('Start Small... Test at Scale'), but there is no sequenced procedure with validation checkpoints — the ETL and deployment sections never state how to verify a pipeline works before shipping. This matches 'sequence present but checkpoints missing or implicit'; it is not 4 because validation is wholly absent, not just minor-gapped.

3 / 5

Progressive Disclosure

This is a monolithic SKILL.md with no references/, scripts/, or assets/ directories — deep-dive material (Structured Streaming, MLlib, production deployment) that clearly belongs in separate reference files is entirely inlined, matching the 'content that clearly belongs in separate files is inlined' anchor. It is worse than score 3's 'some structure' because there is no bundle structure at all, and the single reference it makes — 'All code snippets in the EXAMPLES.md file' — points to a file that does not exist.

2 / 5

Total

10

/

20

Passed

Description

65%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description clearly identifies its Spark niche with strong, distinctive keywords, but it reads as a table-of-contents rather than a capability statement: no action verbs and no 'Use when...' trigger guidance. Adding an explicit trigger clause and concrete actions would raise both completeness and specificity.

Suggestions

Add a 'Use when...' clause (e.g., 'Use when processing large-scale datasets with PySpark, building ETL pipelines, or running distributed analytics') to satisfy the missing 'when' half of completeness.

Replace the passive topic list with concrete actions (e.g., 'Build and tune distributed data pipelines, run Spark SQL queries, train ML models with MLlib, deploy streaming jobs').

Include natural synonyms users actually say, especially 'PySpark', 'big data', and 'ETL', to broaden trigger-term coverage.

DimensionReasoningScore

Specificity

The description enumerates six concrete subtopics ('RDDs, DataFrames, Spark SQL, streaming, MLlib, and production deployment') but uses no action verbs — 'Complete guide for... including...' names domains rather than concrete actions, matching the anchor for a named domain with concrete items but no stated capabilities. It is below the 'several specific actions' anchor because nothing describes what is actually done, and above the generic anchor because the enumeration is specific.

3 / 5

Completeness

The 'what' is clear ('Complete guide for Apache Spark data processing including...') but there is no 'Use when...' clause or equivalent trigger guidance anywhere, which per the judging guidelines caps completeness at 3. It is not 4 because 'when' is entirely absent rather than merely under-specified, and not 2 because the 'what' is concrete and multi-part.

3 / 5

Trigger Term Quality

Terms like 'RDDs', 'DataFrames', 'Spark SQL', 'streaming', and 'MLlib' are exactly what a Spark user would say, giving good keyword coverage. It falls short of comprehensive coverage because natural variations such as 'PySpark', 'big data', or 'ETL' are missing, but it clearly exceeds the 'some relevant keywords missing common variations' anchor.

4 / 5

Distinctiveness Conflict Risk

'Apache Spark', 'RDDs', and 'MLlib' define a clear niche with distinct triggers and minimal conflict risk — a user mentioning these terms unambiguously needs this skill. It is not 4 because the named technologies are specific enough that even closely related skills (e.g., generic data-processing or pandas skills) would rarely be confused with it.

5 / 5

Total

15

/

20

Passed

Validation

87%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

skill_md_line_count

SKILL.md is long (1721 lines); consider splitting into references/ and linking

Warning

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

14

/

16

Passed

Repository

manutej/luxor-claude-marketplace

Path

plugins/luxor-data-engineering/skills/apache-spark-data-processing/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.