0
# Exception Handling
1
2
Comprehensive exception hierarchy for handling execution errors, missing parameters, storage issues, and optional dependency problems. Papermill provides detailed error information to help diagnose and resolve issues during notebook execution.
3
4
## Capabilities
5
6
### Core Exception Classes
7
8
Base exception classes for papermill operations.
9
10
```python { .api }
11
class PapermillException(Exception):
12
"""
13
Base exception for all papermill-related errors.
14
15
Raised when an exception is encountered when operating on a notebook.
16
"""
17
18
class PapermillExecutionError(PapermillException):
19
"""
20
Raised when an exception is encountered in a notebook during execution.
21
22
Attributes:
23
- cell_index: Index of the cell where error occurred
24
- exec_count: Execution count of the failed cell
25
- source: Source code of the failed cell
26
- ename: Exception name
27
- evalue: Exception value/message
28
- traceback: Full traceback of the exception
29
"""
30
31
def __init__(
32
self,
33
cell_index: int,
34
exec_count: int,
35
source: str,
36
ename: str,
37
evalue: str,
38
traceback: list[str]
39
): ...
40
41
class PapermillMissingParameterException(PapermillException):
42
"""
43
Raised when a parameter without a value is required to operate on a notebook.
44
"""
45
46
class PapermillRateLimitException(PapermillException):
47
"""
48
Raised when an I/O request has been rate limited.
49
"""
50
51
class PapermillOptionalDependencyException(PapermillException):
52
"""
53
Raised when an exception is encountered when an optional plugin is missing.
54
"""
55
```
56
57
### Warning Classes
58
59
Warning classes for non-fatal issues.
60
61
```python { .api }
62
class PapermillWarning(Warning):
63
"""
64
Base warning for papermill operations.
65
"""
66
67
class PapermillParameterOverwriteWarning(PapermillWarning):
68
"""
69
Warning when callee overwrites caller argument to pass down the stream.
70
"""
71
```
72
73
### AWS-Related Exceptions
74
75
Specific exceptions for AWS operations.
76
77
```python { .api }
78
class AwsError(Exception):
79
"""
80
Raised when an AWS Exception is encountered.
81
"""
82
83
class FileExistsError(AwsError):
84
"""
85
Raised when a File already exists on S3.
86
"""
87
```
88
89
## Exception Factory Functions
90
91
Utility functions for generating context-specific exceptions.
92
93
```python { .api }
94
def missing_dependency_generator(package: str, dep: str) -> callable:
95
"""
96
Generates a function that raises PapermillOptionalDependencyException
97
for missing dependencies.
98
99
Parameters:
100
- package: Name of the package that requires the dependency
101
- dep: Name of the dependency that is missing
102
103
Returns:
104
callable: Function that raises the appropriate exception when called
105
"""
106
107
def missing_environment_variable_generator(package: str, env_key: str) -> callable:
108
"""
109
Generates a function that raises PapermillOptionalDependencyException
110
for missing environment variables.
111
112
Parameters:
113
- package: Name of the package that requires the environment variable
114
- env_key: Name of the missing environment variable
115
116
Returns:
117
callable: Function that raises the appropriate exception when called
118
"""
119
```
120
121
## Usage Examples
122
123
### Basic Exception Handling
124
125
```python
126
import papermill as pm
127
from papermill import PapermillExecutionError, PapermillMissingParameterException
128
129
try:
130
pm.execute_notebook(
131
'analysis.ipynb',
132
'output.ipynb',
133
parameters={'threshold': 0.8}
134
)
135
except PapermillExecutionError as e:
136
print(f"Execution failed in cell {e.cell_index}")
137
print(f"Error: {e.ename}: {e.evalue}")
138
print(f"Cell source:\n{e.source}")
139
140
except PapermillMissingParameterException as e:
141
print(f"Missing required parameter: {e}")
142
143
except Exception as e:
144
print(f"Unexpected error: {e}")
145
```
146
147
### Detailed Error Analysis
148
149
```python
150
import papermill as pm
151
from papermill import PapermillExecutionError
152
153
def analyze_execution_error(notebook_path, output_path, parameters):
154
try:
155
return pm.execute_notebook(notebook_path, output_path, parameters=parameters)
156
157
except PapermillExecutionError as e:
158
error_info = {
159
'notebook': notebook_path,
160
'failed_cell_index': e.cell_index,
161
'execution_count': e.exec_count,
162
'error_type': e.ename,
163
'error_message': e.evalue,
164
'cell_source': e.source,
165
'full_traceback': e.traceback
166
}
167
168
# Log detailed error information
169
print(f"EXECUTION FAILED: {notebook_path}")
170
print(f"Cell {e.cell_index} (execution count {e.exec_count})")
171
print(f"Error: {e.ename}: {e.evalue}")
172
print("=" * 75)
173
print("Cell Source:")
174
print(e.source)
175
print("=" * 75)
176
print("Traceback:")
177
for line in e.traceback:
178
print(line)
179
180
return error_info
181
182
# Usage
183
result = analyze_execution_error(
184
'problematic_notebook.ipynb',
185
'output.ipynb',
186
{'param1': 'value1'}
187
)
188
```
189
190
### Parameter Validation with Custom Exceptions
191
192
```python
193
import papermill as pm
194
from papermill import PapermillMissingParameterException
195
196
class ParameterValidationError(Exception):
197
"""Custom exception for parameter validation errors."""
198
pass
199
200
def validate_and_execute(notebook_path, output_path, parameters):
201
# Get expected parameters
202
expected_params = pm.inspect_notebook(notebook_path)
203
204
# Check for required parameters
205
required_params = [
206
name for name, info in expected_params.items()
207
if info['default'] is None
208
]
209
210
missing_params = set(required_params) - set(parameters.keys())
211
if missing_params:
212
raise PapermillMissingParameterException(
213
f"Missing required parameters: {missing_params}"
214
)
215
216
# Validate parameter types (example)
217
for param_name, param_value in parameters.items():
218
if param_name in expected_params:
219
expected_type = expected_params[param_name]['inferred_type_name']
220
if expected_type == 'int' and not isinstance(param_value, int):
221
raise ParameterValidationError(
222
f"Parameter '{param_name}' should be int, got {type(param_value)}"
223
)
224
225
try:
226
return pm.execute_notebook(notebook_path, output_path, parameters=parameters)
227
except Exception as e:
228
print(f"Execution failed: {e}")
229
raise
230
231
# Usage
232
try:
233
result = validate_and_execute(
234
'notebook.ipynb',
235
'output.ipynb',
236
{'required_param': 'value', 'numeric_param': 42}
237
)
238
except (PapermillMissingParameterException, ParameterValidationError) as e:
239
print(f"Parameter error: {e}")
240
```
241
242
### Storage Backend Error Handling
243
244
```python
245
import papermill as pm
246
from papermill import PapermillException, AwsError
247
248
def robust_notebook_execution(input_path, output_path, parameters):
249
max_retries = 3
250
retry_count = 0
251
252
while retry_count < max_retries:
253
try:
254
return pm.execute_notebook(input_path, output_path, parameters=parameters)
255
256
except AwsError as e:
257
print(f"AWS error (attempt {retry_count + 1}): {e}")
258
if "rate limit" in str(e).lower():
259
import time
260
time.sleep(2 ** retry_count) # Exponential backoff
261
retry_count += 1
262
continue
263
else:
264
raise
265
266
except FileNotFoundError as e:
267
print(f"File not found: {e}")
268
raise
269
270
except PapermillException as e:
271
print(f"Papermill error: {e}")
272
raise
273
274
except Exception as e:
275
print(f"Unexpected error: {e}")
276
if retry_count < max_retries - 1:
277
retry_count += 1
278
continue
279
else:
280
raise
281
282
raise Exception(f"Failed after {max_retries} attempts")
283
284
# Usage
285
try:
286
result = robust_notebook_execution(
287
's3://bucket/input.ipynb',
288
's3://bucket/output.ipynb',
289
{'param': 'value'}
290
)
291
except Exception as e:
292
print(f"Final failure: {e}")
293
```
294
295
### Optional Dependency Handling
296
297
```python
298
import papermill as pm
299
from papermill import PapermillOptionalDependencyException
300
301
def execute_with_optional_features(notebook_path, output_path, use_s3=False):
302
try:
303
if use_s3:
304
# This might raise PapermillOptionalDependencyException if boto3 is missing
305
return pm.execute_notebook(
306
f's3://bucket/{notebook_path}',
307
f's3://bucket/{output_path}'
308
)
309
else:
310
return pm.execute_notebook(notebook_path, output_path)
311
312
except PapermillOptionalDependencyException as e:
313
print(f"Optional dependency missing: {e}")
314
print("Falling back to local execution...")
315
316
# Fallback to local execution
317
return pm.execute_notebook(notebook_path, output_path)
318
319
# Usage
320
result = execute_with_optional_features(
321
'notebook.ipynb',
322
'output.ipynb',
323
use_s3=True
324
)
325
```
326
327
### Exception Logging and Monitoring
328
329
```python
330
import logging
331
import papermill as pm
332
from papermill import PapermillExecutionError, PapermillException
333
334
# Configure logging
335
logging.basicConfig(
336
level=logging.INFO,
337
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
338
)
339
logger = logging.getLogger(__name__)
340
341
def monitored_execution(notebook_path, output_path, parameters):
342
try:
343
logger.info(f"Starting execution: {notebook_path}")
344
result = pm.execute_notebook(notebook_path, output_path, parameters=parameters)
345
logger.info(f"Execution completed successfully: {output_path}")
346
return result
347
348
except PapermillExecutionError as e:
349
logger.error(f"Execution failed in cell {e.cell_index}: {e.ename}: {e.evalue}")
350
logger.error(f"Failed cell source: {e.source}")
351
352
# Send alert/notification
353
send_execution_alert(notebook_path, e)
354
raise
355
356
except Exception as e:
357
logger.error(f"Unexpected error executing {notebook_path}: {str(e)}")
358
raise
359
360
def send_execution_alert(notebook_path, error):
361
# Implement your alerting logic here
362
alert_data = {
363
'notebook': notebook_path,
364
'error_type': error.ename,
365
'error_message': error.evalue,
366
'cell_index': error.cell_index,
367
'timestamp': str(datetime.now())
368
}
369
# Send to monitoring system, email, Slack, etc.
370
pass
371
372
# Usage
373
try:
374
result = monitored_execution(
375
'critical_analysis.ipynb',
376
'results.ipynb',
377
{'dataset': 'production_data.csv'}
378
)
379
except Exception:
380
# Error already logged and alert sent
381
pass
382
```