Efficient dbt Cloud Job Orchestration

Build an Airflow DAG that orchestrates a long-running dbt Cloud job while efficiently managing worker resources.

Requirements

Your DAG must:

1. Trigger a dbt Cloud job by ID and wait for completion
2. Free up worker slots while waiting for the job to complete
3. Check job status every 60 seconds
4. Have a maximum wait time of 1 hour (3600 seconds)
5. Return the job run ID when complete

Configuration

• DAG name: efficient_dbt_orchestration
• Schedule: Daily at midnight UTC
• dbt Cloud connection ID: dbt_cloud_default
• dbt Cloud job ID: 12345

@generates

API

# DAG configuration
dag_id: str = "efficient_dbt_orchestration"
schedule: str = "@daily"
start_date: datetime

# Task configuration
dbt_cloud_connection_id: str = "dbt_cloud_default"
job_id: int = 12345
polling_interval: int = 60  # seconds
timeout: int = 3600  # seconds

Test Cases

• The task is configured with deferrable mode enabled @test
• The polling interval is set to 60 seconds @test
• The timeout is set to 3600 seconds @test

Dependencies { .dependencies }

apache-airflow { .dependency }

Provides workflow orchestration framework.

apache-airflow-providers-dbt-cloud { .dependency }

Provides dbt Cloud integration for Airflow.