CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

databricks-spark-structured-streaming

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-executed hub-style SKILL.md: tight navigation tables, an executable Quick Start, a production checklist, and a clean one-level-deep reference structure with all links resolving to real files. The main improvements are small: define the placeholder "schema" in the Quick Start, drop the meta intro sentence, and fold the single-row Best Practices table into the checklist.

Suggestions

Define the "schema" variable in the Quick Start snippet (e.g., a minimal StructType example or an explicit note that it must be replaced) so the code is fully copy-paste ready.

Remove the meta-sentence "This skill provides navigation to detailed patterns and best practices" and merge the one-row "Best Practices" table into the "Production Checklist" section to eliminate redundancy.

Add a short "which approach when" pointer (e.g., latency requirement -> RTM vs availableNow, linking to trigger-and-cost-optimization.md) so readers can sequence their choices without opening multiple references.

DimensionReasoningScore

Conciseness

The body is lean — tables with one-line descriptions, a compact Quick Start, and a checklist — with no explanation of concepts Claude already knows. It falls just short of the 5 anchor ("every token earns its place") because of the meta-sentence "This skill provides navigation to detailed patterns and best practices" and the redundant single-row "Best Practices" table that duplicates the separate "Production Checklist" section.

4 / 5

Actionability

The Quick Start is mostly executable, concrete code ("df.writeStream ... .option('checkpointLocation', ...) .trigger(processingTime='30 seconds') .start()"), and the tables give concrete per-topic pointers. It is not a 5 because the snippet references an undefined "schema" variable — a placeholder gap that breaks copy-paste readiness; it is above a 3 because everything else is concrete rather than pseudocode.

4 / 5

Workflow Clarity

As a navigation hub there is no multi-step procedure, but the "Production Checklist" supplies explicit validation checkpoints ("Exactly-once verified (txnVersion/txnAppId)", "Monitoring configured") so the destructive/batch cap does not apply, and the pattern-to-reference mapping is unambiguous. It is not a 5 because the body offers no decision guidance for sequencing (e.g., which trigger mode or reference to pick first for a given goal) and error-recovery feedback loops live only in the references.

4 / 5

Progressive Disclosure

The body is a clear overview hub with 11 well-signaled one-level-deep references, each presented in a descriptive table row (e.g., "See [references/kafka-streaming.md](references/kafka-streaming.md)"); all referenced files were verified to exist, and links inside reference files point only to siblings or other skills, not to a deeper nesting level. This matches the 5 anchor's structure of a concise getting-started section plus clearly labeled reference pointers.

5 / 5

Total

17

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An excellent description: third-person, concrete, and dense with domain-accurate trigger terms, with an explicit 'Use when' clause covering ingestion, stateful operations, triggers, joins, sinks, and cost tuning. The only nit is the mild puffery of the opening word "Comprehensive", which does not materially reduce quality.

DimensionReasoningScore

Specificity

The description lists many concrete actions — "working with Kafka ingestion", "implementing Real-Time Mode (RTM)", "configuring triggers (processingTime, availableNow)", "handling stateful operations with watermarks", "performing stream-stream or stream-static joins", "writing to multiple sinks" — which matches the anchor for multiple specific concrete actions with comprehensive coverage. It is not a 4 because coverage spans ingestion, state, triggers, joins, sinks, and cost tuning with essentially no gaps; the only fluff is the single word "Comprehensive", which is too minor to drop a level.

5 / 5

Completeness

It explicitly answers "what" ("Comprehensive guide to Spark Structured Streaming for production workloads") and "when" via an explicit "Use when" clause with concrete trigger conditions. This mirrors the 5-anchor example structure exactly; a 4 would require the 'when' to be less explicit, but here the trigger list is detailed and concrete.

5 / 5

Trigger Term Quality

Phrases like "Kafka ingestion", "watermarks", "checkpoints", "Real-Time Mode (RTM)", "processingTime, availableNow", "stream-stream or stream-static joins", and "streaming cost and performance" are exactly what a user needing this skill would naturally say. This matches the comprehensive anchor including synonyms and specific API parameter names; it is above the 4 anchor because no obvious common variation (pipeline, checkpoint, sink, trigger) is missing.

5 / 5

Distinctiveness Conflict Risk

"Spark Structured Streaming" with triggers like RTM, watermarks, processingTime/availableNow, and stream-stream joins carves out a clear niche with minimal conflict risk against sibling databricks skills. It is not a 4: the trigger terms are streaming-specific enough that the closely related databricks-core parent would not be wrongly selected.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

databricks/databricks-agent-skills

Path

plugins/databricks/claude/skills/databricks-spark-structured-streaming/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.