CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

databricks-spark-structured-streaming

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, writing custom stateful logic with transformWithState (timers, TTL, per-key state; preferred over applyInPandasWithState and flatMapGroupsWithState for new code), optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

This is a well-executed progressive-disclosure hub: a compact quick start, clearly signaled one-level-deep references that all exist, and a concrete production checklist, with the bulk of detail correctly pushed to reference files. The main headroom is minor tightening — removing the duplicated one-row table and meta sentence, defining the `schema` placeholder in the quick start, and adding a brief error-recovery pointer for failed or lagging streams.

Suggestions

Drop the one-row '## Best Practices' table (it only points to streaming-best-practices.md, which the checklist and other tables already cover) and cut the meta sentence 'This skill provides navigation to detailed patterns and best practices' — the table structure already communicates this.

Make the Quick Start fully copy-paste ready by defining the `schema` variable (or an explicit comment noting it must be supplied), and add a one-line error-recovery pointer (e.g., where to look when a stream fails or lag grows) to strengthen the feedback loop.

Consolidate the RTM mentions that currently appear in three different table rows, and move version-gated details like '(DBR 18+)' and '(DBR 18.3+)' into the target reference files to keep the index lean and less time-sensitive.

DimensionReasoningScore

Conciseness

The body is lean — a quick-start code block, three themed routing tables, and a checklist with no concept explanations Claude already knows — matching the score-4 anchor ('efficient; minor instances that could be trimmed'). It is not a 5 because of small trims available: the meta sentence 'This skill provides navigation to detailed patterns and best practices', a one-row 'Best Practices' table that duplicates the '## Production Checklist' section below it, RTM appearing in three separate table rows, and version parentheticals ('DBR 18+', 'DBR 18.3+') that add time-sensitive detail to what should be a lean index.

4 / 5

Actionability

The Quick Start is nearly copy-paste-ready executable PySpark (readStream from Kafka, from_json parse, writeStream with checkpoint and trigger) and the checklist items are concrete and verifiable ('txnVersion/txnAppId', 'UC volumes, not DBFS'), fitting the score-4 anchor 'mostly executable guidance with minor gaps'. It falls short of 5 because the code block relies on an undefined `schema` variable without noting it must be defined, and the body itself is deliberately a routing hub, so most executable guidance lives in the references rather than the main file.

4 / 5

Workflow Clarity

The Quick Start provides a concrete sequence (ingest, parse, write with trigger/checkpoint) and the Production Checklist supplies real validation checkpoints for a risky continuous workload (persistent checkpoints, monitoring of 'input rate, lag, batch duration', exactly-once verification), which fits score 4's 'clear sequence with most checkpoints present'. It is not 5 because there is no explicit error-recovery/feedback loop in the body (e.g., what to do when a stream fails or lag grows — that presumably lives in references), and guidance on when to consult which reference in what order is only implicit in the table organization.

4 / 5

Progressive Disclosure

The body is a clean overview hub: three themed tables map every pattern/topic to a one-level-deep reference with a one-line description of its contents, and all 12 referenced files (kafka-streaming.md, real-time-mode.md, transform-with-state.md, etc.) actually exist in references/. Spot-checking the bundle shows references link only laterally to siblings rather than nesting content two-plus levels deep, matching the score-5 anchor 'clear overview with well-signaled one-level-deep references; content appropriately split; easy navigation'.

5 / 5

Total

17

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: it states what the skill covers, gives an explicit and richly detailed 'Use when' clause with natural trigger terms including abbreviations and API names, and is tightly scoped to Spark Structured Streaming so conflict risk with sibling skills is minimal. Third-person voice is used throughout and there is no fluff despite the length — every clause adds a trigger or capability.

DimensionReasoningScore

Specificity

The description enumerates many concrete actions — 'building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, writing custom stateful logic with transformWithState (timers, TTL, per-key state), optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks' — which is comprehensive, concrete coverage of the domain, matching the score-5 anchor. It goes beyond the score-4 anchor ('several specific actions; minor gaps') because no major streaming capability area is left unlisted.

5 / 5

Completeness

It explicitly answers 'what' ('Comprehensive guide to Spark Structured Streaming for production workloads') and 'when' via an explicit 'Use when' clause with many concrete trigger conditions, exactly the pattern of the score-5 anchor example. The 'what' is concrete (names the technology and production scope), so it does not fall to the score-4 anchor where 'when' could be more explicit — the when-clause here is unusually specific.

5 / 5

Trigger Term Quality

It covers the natural phrases a Databricks/Spark user would actually say — 'streaming pipelines', 'Kafka ingestion', 'watermarks', 'checkpoints', 'stream-stream joins', 'multiple sinks', 'cost and performance' — plus synonyms and API-level specifics: the abbreviation pair 'Real-Time Mode (RTM)', both trigger names 'processingTime, availableNow', and the legacy API names 'applyInPandasWithState and flatMapGroupsWithState'. This matches the score-5 anchor (comprehensive natural terms including synonyms), not the score-4 anchor which expects missing natural terms.

5 / 5

Distinctiveness Conflict Risk

Every trigger is scoped to Spark Structured Streaming — RTM, transformWithState, stream-stream joins, watermarks, streaming checkpoints — giving it a clear niche with distinct triggers and minimal conflict risk, matching the score-5 anchor. It stays above the score-4 anchor ('minor overlap risk') because even potentially generic terms like 'optimizing checkpoints' and 'multiple sinks' are framed inside an explicitly streaming-scoped sentence, so it would not fire for a Delta Lake or SQL skill.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

databricks/databricks-agent-skills

Path

plugins/databricks/claude/skills/databricks-spark-structured-streaming/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.