spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

1.40x

Quality

71%

Does it follow best practices?

Impact

73%

1.40x

Average score across 6 eval scenarios

Securityby

Passed

No known issues

Optimize this skill with Tessl

npx tessl skill review --optimize ./plugins/data-engineering/skills/spark-optimization/SKILL.md

Evaluation results

92%

52%

Sales Analytics Pipeline Optimization

Join optimization and session configuration

Criteria

Without context

With context

AQE enabled

100%

AQE coalesce partitions

100%

AQE skew join

100%

Kryo serializer

100%

Shuffle partitions

Broadcast join hint

100%

Broadcast threshold config

100%

Shuffle compression

100%

Compression codec lz4

100%

mergeSchema false

100%

Column pruning

100%

maxPartitionBytes config

100%

82%

13%

Multi-Feature ML Feature Engineering Pipeline

Caching, persistence, and iterative pipeline patterns

Criteria

Without context

With context

MEMORY_AND_DISK cache

50%

100%

Cache before multiple actions

100%

Unpersist after use

100%

Checkpoint used

Checkpoint dir set

approx_count_distinct used

100%

No Python UDFs

100%

No large collect

100%

No count for existence

100%

AQE enabled

100%

Kryo serializer

100%

78%

E-Commerce Order Attribution Pipeline with Skewed Data

Data skew handling and partitioning strategy

Criteria

Without context

With context

Skew detection

100%

Salt column added

100%

80%

Salted key column

60%

80%

Other side exploded

100%

80%

AQE skew factor

100%

AQE skew threshold

100%

Coalesce not repartition

Repartition with key

Write with partitionBy

100%

Memory configuration

100%

Partition count config

100%

Parquet snappy

100%

38%

22%

Recurring Order-Customer Analytics Pipeline

Bucket joins and Delta Lake optimization

Criteria

Without context

With context

bucketBy used

sortBy used

saveAsTable used

Matching bucket count

Delta optimizeWrite

100%

Delta autoCompact

100%

ZORDER BY applied

100%

Parquet block size

AQE enabled

100%

Kryo serializer

100%

mergeSchema false

Column pruning

57%

23%

Clickstream Event Aggregation Service

Shuffle pre-aggregation and Arrow optimization

Criteria

Without context

With context

Local pre-aggregation

16%

Global aggregation on partial sum

Arrow enabled

100%

approx_count_distinct used

100%

openCostInBytes config

Shuffle compression

100%

Compression codec lz4

100%

AQE shuffle auto

100%

No large collect

100%

Coalesce for output

Kryo serializer

100%

mergeSchema false

92%

10%

Spark Performance Diagnostic Toolkit

Performance monitoring and query plan analysis

Criteria

Without context

With context

explain() used

100%

explain cost mode

50%

100%

spark_partition_id skew check

100%

Skew ratio calculation

62%

100%

statusTracker used

100%

Stage metrics printed

100%

Executor memory monitoring

70%

100%

Memory formatted GB

100%

wholeStage codegen

AQE enabled

100%

calculate_partitions function

100%

Kryo serializer

100%

Repository: wshobson/agents
Commit: 70444e5

Evaluated: 29 days ago
Agent: Claude Code
Model: Claude Sonnet 4.6

Table of Contents

Sales Analytics Pipeline Optimization Multi-Feature ML Feature Engineering Pipeline E-Commerce Order Attribution Pipeline with Skewed Data Recurring Order-Customer Analytics Pipeline Clickstream Event Aggregation Service Spark Performance Diagnostic Toolkit

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.