Data Pipeline Orchestration

Build an Airflow DAG that orchestrates a data transformation workflow by triggering dbt Cloud jobs.

Requirements

You need to create an Airflow DAG that:

1. Triggers a dbt Cloud job named "daily_transform" in the "analytics" project and "production" environment
2. Waits synchronously for the job to complete with a timeout of 600 seconds
3. Uses a polling interval of 30 seconds to check job status
4. Stores the job run ID for downstream tasks using XCom
5. Sets a descriptive triggering reason: "Airflow scheduled run"

The DAG should:

• Be scheduled to run daily at 2 AM UTC
• Have a DAG ID of "dbt_daily_pipeline"
• Use the dbt Cloud connection ID "dbt_cloud_default"
• Include appropriate default arguments (owner, retries, etc.)

Test Cases

• The DAG successfully triggers the dbt Cloud job by name (project/environment/job combination) @test
• The operator waits synchronously for job completion @test
• The operator returns a job run ID that can be accessed by downstream tasks @test
• The timeout and polling interval are correctly configured @test

Implementation

@generates

Dependencies { .dependencies }

apache-airflow-providers-dbt-cloud { .dependency }

Provides integration between Apache Airflow and dbt Cloud for orchestrating data transformation workflows.

apache-airflow { .dependency }

Provides the workflow orchestration platform and DAG framework.