FTP File Monitor with Graceful Degradation

Create an Apache Airflow DAG that monitors an FTP server for data files with flexible failure handling to ensure downstream tasks can proceed even when expected files are missing.

Requirements

You need to implement a DAG that monitors two types of data files on an FTP server:

1. Critical data file: Must be present for the workflow to continue

   • Path: /data/critical/daily_report.csv
   • If not found within the timeout period, the DAG should fail

2. Optional supplementary file: Nice to have but not required

   • Path: /data/optional/enrichment_data.csv
   • If not found within the timeout period, skip this check and allow the DAG to continue

Both monitoring tasks should:

• Check for file existence on the FTP server
• Poll every 30 seconds
• Have a maximum wait time of 5 minutes
• Use poke mode for monitoring

The DAG should be named ftp_file_monitor and scheduled to run daily.

Test Cases

• The critical file sensor fails the DAG when the file is not found within timeout @test
• The optional file sensor allows the DAG to continue (skips) when the file is not found within timeout @test
• Both sensors successfully detect files when they exist @test

Implementation

@generates

Dependencies { .dependencies }

apache-airflow-providers-ftp { .dependency }

Provides FTP file monitoring capabilities with timeout and soft-fail support.