0
# Apache Airflow Papermill Provider
1
2
Apache Airflow provider for executing Jupyter notebooks with Papermill. This provider enables data teams to integrate notebook-based analytics and machine learning workflows into their Airflow DAGs by executing parameterized Jupyter notebooks through the Papermill library.
3
4
## Package Information
5
6
- **Package Name**: apache-airflow-providers-papermill
7
- **Language**: Python
8
- **Installation**: `pip install apache-airflow-providers-papermill`
9
10
## Core Imports
11
12
```python
13
from airflow.providers.papermill.operators.papermill import PapermillOperator
14
```
15
16
For lineage tracking:
17
18
```python
19
from airflow.providers.papermill.operators.papermill import NoteBook
20
```
21
22
## Basic Usage
23
24
```python
25
from airflow import DAG
26
from airflow.providers.papermill.operators.papermill import PapermillOperator
27
from airflow.utils.dates import days_ago
28
from datetime import timedelta
29
30
# Define DAG
31
dag = DAG(
32
'example_papermill',
33
default_args={'owner': 'airflow'},
34
schedule_interval='0 0 * * *',
35
start_date=days_ago(2),
36
dagrun_timeout=timedelta(minutes=60),
37
)
38
39
# Execute a notebook with parameters
40
run_notebook = PapermillOperator(
41
task_id="run_analysis_notebook",
42
input_nb="/path/to/analysis.ipynb",
43
output_nb="/path/to/output-{{ execution_date }}.ipynb",
44
parameters={"date": "{{ execution_date }}", "source": "airflow"},
45
dag=dag
46
)
47
```
48
49
## Capabilities
50
51
### Notebook Execution
52
53
Execute Jupyter notebooks through Papermill with parameter injection and lineage tracking support.
54
55
```python { .api }
56
class PapermillOperator(BaseOperator):
57
"""
58
Executes a jupyter notebook through papermill that is annotated with parameters
59
60
:param input_nb: input notebook (can also be a NoteBook or a File inlet)
61
:type input_nb: str
62
:param output_nb: output notebook (can also be a NoteBook or File outlet)
63
:type output_nb: str
64
:param parameters: the notebook parameters to set
65
:type parameters: dict
66
"""
67
68
supports_lineage = True
69
70
@apply_defaults
71
def __init__(
72
self,
73
*,
74
input_nb: Optional[str] = None,
75
output_nb: Optional[str] = None,
76
parameters: Optional[Dict] = None,
77
**kwargs,
78
) -> None: ...
79
80
def execute(self, context): ...
81
```
82
83
### Lineage Entity
84
85
Represents Jupyter notebooks for Airflow lineage tracking.
86
87
```python { .api }
88
@attr.s(auto_attribs=True)
89
class NoteBook(File):
90
"""Jupyter notebook"""
91
92
type_hint: Optional[str] = "jupyter_notebook"
93
parameters: Optional[Dict] = {}
94
meta_schema: str = __name__ + '.NoteBook'
95
```
96
97
## Types
98
99
```python { .api }
100
from typing import Dict, Optional
101
import attr
102
import papermill as pm
103
from airflow.lineage.entities import File
104
from airflow.models import BaseOperator
105
from airflow.utils.decorators import apply_defaults
106
```
107
108
## Advanced Usage Examples
109
110
### Template Variables
111
112
Use Airflow's templating system for dynamic notebook paths and parameters:
113
114
```python
115
run_notebook = PapermillOperator(
116
task_id="daily_report",
117
input_nb="/notebooks/daily_report_template.ipynb",
118
output_nb="/reports/daily_report_{{ ds }}.ipynb",
119
parameters={
120
"report_date": "{{ ds }}",
121
"execution_time": "{{ execution_date }}",
122
"run_id": "{{ run_id }}"
123
}
124
)
125
```
126
127
### Lineage Tracking
128
129
The operator automatically creates lineage entities that can be used by downstream tasks:
130
131
```python
132
from airflow.lineage import AUTO
133
from airflow.operators.python import PythonOperator
134
135
def process_notebook_output(inlets, **context):
136
# Access the output notebook through lineage
137
notebook_path = inlets[0].url
138
# Process the executed notebook...
139
140
process_task = PythonOperator(
141
task_id='process_output',
142
python_callable=process_notebook_output,
143
inlets=AUTO # Automatically detects upstream notebook outputs
144
)
145
146
run_notebook >> process_task
147
```
148
149
### Multiple Notebook Execution
150
151
Execute multiple notebooks in sequence by setting up multiple inlets and outlets:
152
153
```python
154
# Note: This pattern requires careful setup of inlets/outlets
155
# The operator will execute notebooks in pairs (inlet[i] -> outlet[i])
156
multi_notebook = PapermillOperator(
157
task_id="run_multiple_notebooks",
158
# Set up inlets and outlets manually for multiple notebooks
159
dag=dag
160
)
161
# Additional configuration needed for multiple notebook execution
162
```
163
164
## Error Handling
165
166
The operator performs validation during execution:
167
168
- Raises `ValueError` if inlets or outlets are not properly configured (i.e., "Input notebook or output notebook is not specified")
169
- Papermill execution errors are propagated as task failures
170
171
## Integration Notes
172
173
### Papermill Configuration
174
175
The operator calls `papermill.execute_notebook()` with these settings:
176
- `progress_bar=False` - Disables progress display for cleaner logs
177
- `report_mode=True` - Enables report generation mode
178
- Parameters are passed through for notebook injection
179
180
### Airflow Features
181
182
- **Templating**: All string parameters support Airflow's Jinja templating
183
- **Lineage**: Automatic lineage tracking through `NoteBook` entities
184
- **XCom**: Can be used with XCom for passing data between tasks
185
- **Retries**: Supports standard Airflow retry mechanisms
186
- **Connections**: Can use Airflow connections for remote notebook storage
187
188
## Migration Notes
189
190
The legacy import path is deprecated and issues a warning:
191
192
```python
193
# DEPRECATED - issues warning
194
from airflow.operators.papermill_operator import PapermillOperator
195
```
196
197
Use the current import path:
198
199
```python
200
# Current import path
201
from airflow.providers.papermill.operators.papermill import PapermillOperator
202
```