Staging layer exists

Intermediate layer exists

Marts layer exists

Staging materialized as view

Intermediate materialized as ephemeral

Incremental merge strategy

on_schema_change set

cluster_by configured

Watermark filter in incremental block

Column tests: unique and not_null

accepted_range test on amount

Recency test present

Source freshness config

Surrogate key macro used

Reusable macro defined

Kafka partitions=12

Kafka replication-factor=3

Kafka retention config

Kafka cleanup.policy=delete

Checkpoint location set

shuffle.partitions=12

failOnDataLoss=false

Watermark applied

Delta Lake append output

processingTime trigger

foreachBatch pattern used

DLQ write with error columns

Prometheus Counter metric

Prometheus Gauge metric

Uses pipeline_orchestrator.py

Orchestrator source/destination flags

Orchestrator incremental mode

Uses data_quality_validator.py

Validator checks specified

Airflow retries=2

Airflow retry_delay=5min

Airflow email_on_failure=True

Airflow schedule 0 5 * * *

Airflow catchup=False

Airflow tags present

Data contract schema section

Data contract SLA section

Data contract consumers section

KryoSerializer configured

Adaptive query execution enabled

Executor memory 8g

Executor cores 4

Driver memory 4g

Shuffle partitions 200

Broadcast join for small table

MEMORY_AND_DISK persistence

Unpersist called

Bronze: mergeSchema=true

Bronze: metadata columns

Bronze: append mode

Silver: DeltaTable merge upsert

Gold: partitioned by date

All layers use Delta format

Surrogate key via generate_surrogate_key

incremental materialization

strategy='check' with check_cols

effective_start_date column

effective_end_date = 9999-12-31

is_current boolean

row_hash for change detection

is_incremental() filter block

Natural key retained

Staging materialized as view

Tests: unique and not_null on surrogate key

Tests: recency on updated_at or effective_start_date

Macro file created

acks='all'

enable_idempotence=True

max_in_flight=5

retries=max int

transactional_id set

init_transactions called

begin/commit/abort pattern

send_offsets_to_transaction

catchup=True

max_active_runs set

Processes by execution_date not today

Idempotent load pattern

Loads to date partition