spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Install with Tessl CLI

npx tessl i github:Dicklesworthstone/pi_agent_rust --skill spark-optimization

What are skills?

Review — 82%

Does it follow best practices?

If you maintain this skill, you can automatically optimize it using the tessl CLI to improve its score:

npx tessl skill review --optimize ./path/to/skill

Learn more

Validation — 11 / 11 Passed

Validation for skill structure

SKILL.md

Review

Evals

Evaluation results

100%

24%

Sales Analytics Pipeline Optimization

Join optimization and session configuration

Criteria

Without context

With context

AQE enabled

100%

AQE coalesce partitions

100%

AQE skew join

100%

Kryo serializer

100%

Shuffle partitions

100%

Broadcast join hint

100%

Broadcast threshold config

100%

Shuffle compression

100%

Compression codec lz4

100%

mergeSchema false

100%

Column pruning

100%

maxPartitionBytes config

100%

Without context: $0.3165 · 1m 32s · 12 turns · 54 in / 5,112 out tokens

With context: $0.4619 · 2m 8s · 16 turns · 254 in / 6,455 out tokens

77%

Multi-Feature ML Feature Engineering Pipeline

Caching, persistence, and iterative pipeline patterns

Criteria

Without context

With context

MEMORY_AND_DISK cache

100%

Cache before multiple actions

100%

Unpersist after use

100%

50%

Checkpoint used

Checkpoint dir set

approx_count_distinct used

100%

No Python UDFs

100%

No large collect

100%

No count for existence

100%

AQE enabled

100%

Kryo serializer

100%

Without context: $0.2389 · 1m 14s · 10 turns · 10 in / 3,789 out tokens

With context: $0.5522 · 2m 6s · 22 turns · 20 in / 6,428 out tokens

54%

-17%

E-Commerce Order Attribution Pipeline with Skewed Data

Data skew handling and partitioning strategy

Criteria

Without context

With context

Skew detection

100%

Salt column added

100%

Salted key column

50%

Other side exploded

100%

AQE skew factor

100%

AQE skew threshold

100%

Coalesce not repartition

Repartition with key

100%

Write with partitionBy

100%

Memory configuration

100%

Partition count config

100%

Parquet snappy

100%

Without context: $0.5846 · 2m 56s · 20 turns · 21 in / 9,796 out tokens

With context: $0.8312 · 3m 49s · 25 turns · 24 in / 12,488 out tokens

Evaluated: 4 days ago
Agent: Claude Code

Table of Contents

Sales Analytics Pipeline Optimization Multi-Feature ML Feature Engineering Pipeline E-Commerce Order Attribution Pipeline with Skewed Data

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.