CtrlK
BlogDocsLog inGet started
Tessl Logo

spark-python-data-source

Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.

76

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

87%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable skill body with excellent progressive disclosure and token discipline. The main gap is the absence of an explicit validation/feedback loop in the build-deploy workflow for batch writer operations.

Suggestions

Add an explicit validate-then-proceed checkpoint to the build workflow (e.g., 'run uv run pytest and only proceed to uv build when tests pass') to close the feedback-loop gap for batch/destructive writer operations.

In the Testing section, show a failure->fix->re-run loop so the verification step reads as a real feedback loop rather than a single assertion block.

Briefly note in the main body when to consult error-handling.md before shipping a writer, so retry/circuit-breaker validation is part of the build sequence rather than only discoverable in the reference.

DimensionReasoningScore

Conciseness

Lean and efficient: it states Spark-specific constraints Claude is unlikely to know (executor serialization, importing inside executor methods, flat inheritance) and explicitly declines to repeat general Python best practices, with tight code and bullet summaries pointing to references.

3 / 3

Actionability

Provides copy-paste-ready setup commands, a concrete project layout, executable pytest fixtures, and specific lint/build commands alongside real reference implementations.

3 / 3

Workflow Clarity

The build sequence (setup -> architecture -> decisions -> test) is clearly listed and the testing section acts as verification, but there is no explicit validate-then-proceed feedback loop for batch writer operations, which the rubric caps at 2 for destructive/batch workflows.

2 / 3

Progressive Disclosure

SKILL.md is a concise overview with eight well-signaled one-level-deep references, each given an inline 'See X for Y' pointer plus a bottom References section with 'Read when...' guidance; all referenced files exist and content is appropriately split.

3 / 3

Total

11

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description with explicit use-when triggers and good coverage of natural user phrasing. It is slightly long, but each clause adds a distinct trigger rather than padding.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'Build custom Python data sources', 'batch and streaming readers/writers', 'implement a DataSourceReader or DataSourceWriter', 'pull data from or push data to a system via Spark' — matching the top anchor.

3 / 3

Completeness

Clearly states what the skill does (build custom Python data sources for batch and streaming) and gives an explicit 'Use this skill whenever someone wants to...' trigger clause addressing both what and when.

3 / 3

Trigger Term Quality

Covers natural user phrasing such as 'connect Spark to an external system', 'build a Spark connector or plugin in Python', and the explicit 'read from X in Spark' / 'write DataFrame to Y' variants users would actually say.

3 / 3

Distinctiveness Conflict Risk

Niche is clearly scoped to the PySpark DataSource API and custom Python connectors, with triggers unlikely to fire for unrelated skills.

3 / 3

Total

12

/

12

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository
databricks-solutions/ai-dev-kit
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.