CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

databricks-spark-structured-streaming

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

74

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

86%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured overview skill: lean token usage, strong progressive disclosure with a real, complete reference bundle, and a useful production checklist. The only weaknesses are minor: the Quick Start depends on an undefined `schema` variable, and there is no error-recovery guidance for when a stream fails production checks.

Suggestions

Define or placeholder `schema` explicitly in the Quick Start (e.g., a minimal `StructType` example or a comment like `schema = your topic's schema`) so the code is copy-paste executable.

Add a short failure/recovery note (e.g., what to inspect when a stream stalls — checkpoint state, input lag, watermark delay — or how to validate a new stream before promoting it) to close the workflow validation gap.

Fold the single-row 'Best Practices' table into the 'Configuration' table (or the Production Checklist section) to remove the redundant pointer to streaming-best-practices.md.

DimensionReasoningScore

Conciseness

The ~70-line body is pure function: a two-sentence orientation, one Quick Start code block, compact navigation tables, and a seven-item checklist. There is no explanation of concepts Claude already knows (no Spark/streaming primer) and no padded prose; every section earns its tokens. This matches the 'lean and efficient, every token earns its place' anchor rather than anchor 4, which requires over-explanation to trim.

5 / 5

Actionability

The Quick Start is concrete, real API code and the Production Checklist gives specific, executable items ("Checkpoint location is persistent (UC volumes, not DBFS)", "Exactly-once verified (txnVersion/txnAppId)"). However, the code is not copy-paste ready: `schema` in `from_json(col("value").cast("string"), schema)` is undefined, a minor gap that places it at anchor 4 rather than the fully-executable anchor 5.

4 / 5

Workflow Clarity

There is a coherent arc — Quick Start (get running) → Core Patterns/Configuration navigation (find your pattern) → Production Checklist (verify readiness) — and the checklist serves as a validation checkpoint for production streams, including exactly-once and monitoring verification. Anchor 5 requires explicit feedback loops for error recovery, which are absent here (no guidance on what to do when a stream fails or validation finds a problem), so it sits at 'clear sequence with most checkpoints present; minor validation gaps'.

4 / 5

Progressive Disclosure

Exemplary hub-and-spoke structure: a lean overview with a getting-started snippet, and all eleven referenced files (kafka-streaming.md, real-time-mode.md, lakebase-sink-python.md, stream-stream-joins.md, stream-static-joins.md, multi-sink-writes.md, merge-operations.md, checkpoint-best-practices.md, stateful-operations.md, trigger-and-cost-optimization.md, streaming-best-practices.md) exist on disk, are one level deep, and are clearly signaled in labeled tables with per-topic descriptions. This is a direct match for the anchor-5 pattern of well-signaled one-level-deep references.

5 / 5

Total

18

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A well-crafted description: third-person, explicit 'Use when' trigger clause with nine concrete scenarios, domain-accurate terminology including abbreviation synonyms (RTM, processingTime, availableNow), and a clearly bounded niche. No fluff beyond a single mild "Comprehensive guide" opener, which does not dilute the concrete action list.

DimensionReasoningScore

Specificity

Lists nine concrete actions — "building streaming pipelines", "working with Kafka ingestion", "implementing Real-Time Mode (RTM)", "configuring triggers (processingTime, availableNow)", "handling stateful operations with watermarks", "optimizing checkpoints", "performing stream-stream or stream-static joins", "writing to multiple sinks", "tuning streaming cost and performance" — which matches the comprehensive multi-action anchor. Score 4 would require minor coverage gaps, but every major streaming capability area is named explicitly.

5 / 5

Completeness

Explicitly answers both: what — "Comprehensive guide to Spark Structured Streaming for production workloads", and when — a "Use when..." clause enumerating concrete trigger scenarios. This mirrors the anchor-5 exemplar shape (capability list + explicit 'Use when' with concrete triggers); anchor 4 would require the 'when' to be less specific, which it is not.

5 / 5

Trigger Term Quality

Natural user phrasing is covered comprehensively with synonyms and abbreviation pairs: "Kafka ingestion", "Real-Time Mode (RTM)", "processingTime, availableNow", "watermarks", "checkpoints", "stream-stream or stream-static joins", "multiple sinks". These are exactly the terms a Spark/Databricks user would say; anchor 4 would apply only if natural terms were missing, and none of the core vocabulary is absent.

5 / 5

Distinctiveness Conflict Risk

Clear niche (Spark Structured Streaming on Databricks) with distinct triggers like watermarks, stream-stream joins, trigger configuration, and RTM. These terms are streaming-specific and unlikely to fire for batch Spark or other databricks skills; anchor 5 (clear niche, minimal conflict risk) is the best fit.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

databricks/databricks-agent-skills

Path

plugins/databricks/claude/skills/databricks-spark-structured-streaming/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.