0
# NBVal
1
2
A py.test plugin to validate Jupyter notebooks by executing notebook cells and comparing their outputs against stored results. NBVal enables automated testing of notebook content to ensure consistency and functionality, supporting features like output sanitization through regex patterns, parallel execution compatibility, coverage integration, and flexible kernel selection.
3
4
## Package Information
5
6
- **Package Name**: nbval
7
- **Package Type**: pytest plugin
8
- **Language**: Python
9
- **Installation**: `pip install nbval`
10
- **Requirements**: Python >=3.7, <4
11
12
## Core Imports
13
14
```python
15
import nbval
16
```
17
18
For accessing plugin functionality directly:
19
20
```python
21
from nbval.plugin import IPyNbFile, IPyNbCell, NbCellError
22
from nbval.kernel import RunningKernel, start_new_kernel
23
```
24
25
## Basic Usage
26
27
NBVal works as a pytest plugin, activated through command-line flags:
28
29
```bash
30
# Run notebooks validating all outputs
31
pytest --nbval
32
33
# Run notebooks only validating marked cells
34
pytest --nbval-lax
35
36
# Test a specific notebook
37
pytest --nbval my_notebook.ipynb
38
39
# Use output sanitization
40
pytest --nbval my_notebook.ipynb --nbval-sanitize-with sanitize_config.cfg
41
42
# Use current environment kernel
43
pytest --nbval --nbval-current-env
44
45
# Use specific kernel
46
pytest --nbval --nbval-kernel-name python3
47
```
48
49
### Cell Markers
50
51
Control cell behavior using comment markers or metadata tags:
52
53
```python
54
# NBVAL_IGNORE_OUTPUT - Skip output validation
55
print("This output won't be checked")
56
57
# NBVAL_CHECK_OUTPUT - Force output validation (useful with --nbval-lax)
58
print("This output will be validated")
59
60
# NBVAL_RAISES_EXCEPTION - Expect cell to raise exception
61
raise ValueError("Expected error")
62
63
# NBVAL_SKIP - Skip cell execution entirely
64
# This cell won't run
65
```
66
67
### Output Sanitization Configuration
68
69
Create a sanitization file to clean outputs before comparison:
70
71
```ini
72
[timestamps]
73
regex: \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
74
replace: TIMESTAMP
75
76
[memory_addresses]
77
regex: 0x[0-9a-fA-F]+
78
replace: 0xADDRESS
79
```
80
81
## Capabilities
82
83
### Package Version Access
84
85
Access to package version information.
86
87
```python { .api }
88
__version__: str
89
```
90
91
### Pytest Plugin Hooks
92
93
Core pytest integration hooks for notebook collection and configuration.
94
95
```python { .api }
96
def pytest_addoption(parser):
97
"""
98
Add nbval command-line options to pytest.
99
100
Args:
101
parser: pytest argument parser
102
"""
103
104
def pytest_configure(config):
105
"""
106
Configure nbval plugin based on options.
107
108
Args:
109
config: pytest configuration object
110
"""
111
112
def pytest_collect_file(file_path, parent):
113
"""
114
Collect .ipynb files for testing.
115
116
Args:
117
file_path: Path to potential test file
118
parent: Parent collector
119
120
Returns:
121
IPyNbFile instance for .ipynb files, None otherwise
122
"""
123
```
124
125
### Notebook File Collection
126
127
Pytest collector for handling notebook files as test suites.
128
129
```python { .api }
130
class IPyNbFile(pytest.File):
131
"""
132
Pytest collector for notebook files.
133
"""
134
135
def setup(self):
136
"""Set up notebook execution environment."""
137
138
def collect(self):
139
"""
140
Collect notebook cells as test items.
141
142
Returns:
143
List of IPyNbCell instances
144
"""
145
146
def teardown(self):
147
"""Clean up notebook execution environment."""
148
149
def get_kernel_message(self, timeout=None):
150
"""
151
Get message from running kernel.
152
153
Args:
154
timeout: Message timeout in seconds
155
156
Returns:
157
Kernel message dictionary
158
"""
159
160
def setup_sanitize_files(self):
161
"""Set up output sanitization configuration."""
162
163
def get_sanitize_files(self):
164
"""
165
Get sanitization configuration.
166
167
Returns:
168
List of sanitization patterns
169
"""
170
```
171
172
### Notebook Cell Testing
173
174
Pytest item representing individual notebook cells for execution and validation.
175
176
```python { .api }
177
class IPyNbCell(pytest.Item):
178
"""
179
Pytest item representing a notebook cell.
180
"""
181
182
def runtest(self):
183
"""Execute cell and compare outputs."""
184
185
def repr_failure(self, excinfo):
186
"""
187
Represent test failure for reporting.
188
189
Args:
190
excinfo: Exception information
191
192
Returns:
193
Formatted failure representation
194
"""
195
196
def reportinfo(self):
197
"""
198
Get test location information.
199
200
Returns:
201
Tuple of (file_path, line_number, test_name)
202
"""
203
204
def compare_outputs(self, test, ref, skip_compare=None):
205
"""
206
Compare cell outputs against reference.
207
208
Args:
209
test: Actual execution output
210
ref: Reference output from notebook
211
skip_compare: Output types to skip comparison
212
213
Returns:
214
True if outputs match, False otherwise
215
"""
216
217
def setup(self):
218
"""Set up cell execution environment."""
219
220
def raise_cell_error(self, **kwargs):
221
"""
222
Raise formatted cell execution error.
223
224
Args:
225
**kwargs: Error details
226
227
Raises:
228
NbCellError: Formatted cell error
229
"""
230
231
def sanitize(self, s):
232
"""
233
Apply output sanitization patterns.
234
235
Args:
236
s: String to sanitize
237
238
Returns:
239
Sanitized string
240
"""
241
242
def sanitize_outputs(self, outputs):
243
"""
244
Apply sanitization to output list.
245
246
Args:
247
outputs: List of output dictionaries
248
249
Returns:
250
List of sanitized outputs
251
"""
252
```
253
254
### Error Handling
255
256
Custom exception for notebook cell execution errors.
257
258
```python { .api }
259
class NbCellError(Exception):
260
"""
261
Custom exception for cell execution errors.
262
263
Attributes:
264
cell_num: Cell number that failed
265
source: Cell source code
266
inner_traceback: Original exception traceback
267
"""
268
269
def __init__(self, cell_num, msg, source, traceback=None, *args, **kwargs):
270
"""
271
Initialize cell error.
272
273
Args:
274
cell_num: Failed cell number
275
msg: Error message
276
source: Cell source code
277
traceback: Original traceback
278
*args: Additional arguments
279
**kwargs: Additional keyword arguments
280
"""
281
```
282
283
### Kernel Management
284
285
Jupyter kernel execution and lifecycle management.
286
287
```python { .api }
288
class RunningKernel:
289
"""
290
Manages Jupyter kernel execution.
291
"""
292
293
def __init__(self, kernel_name, cwd=None, startup_timeout=60):
294
"""
295
Initialize kernel manager.
296
297
Args:
298
kernel_name: Name of kernel to start
299
cwd: Working directory for kernel
300
startup_timeout: Kernel startup timeout in seconds
301
"""
302
303
def execute_cell_input(self, cell_input, allow_stdin=True):
304
"""
305
Execute cell input in kernel.
306
307
Args:
308
cell_input: Cell source code to execute
309
allow_stdin: Whether to allow stdin
310
311
Returns:
312
Execution message ID
313
"""
314
315
def get_message(self, timeout=None):
316
"""
317
Get message from kernel.
318
319
Args:
320
timeout: Message timeout in seconds
321
322
Returns:
323
Kernel message dictionary
324
"""
325
326
def await_reply(self, msg_id, timeout=None):
327
"""
328
Wait for execution reply.
329
330
Args:
331
msg_id: Message ID to wait for
332
timeout: Reply timeout in seconds
333
334
Returns:
335
Reply message dictionary
336
"""
337
338
def await_idle(self, msg_id, timeout=None):
339
"""
340
Wait for kernel to become idle.
341
342
Args:
343
msg_id: Message ID to wait for
344
timeout: Idle timeout in seconds
345
"""
346
347
def is_alive(self):
348
"""
349
Check if kernel is alive.
350
351
Returns:
352
True if kernel is running, False otherwise
353
"""
354
355
def restart(self):
356
"""Restart the kernel."""
357
358
def interrupt(self):
359
"""Interrupt kernel execution."""
360
361
def stop(self):
362
"""Stop and cleanup kernel."""
363
364
@property
365
def language(self):
366
"""
367
Get kernel language.
368
369
Returns:
370
Kernel language name
371
"""
372
373
def start_new_kernel(startup_timeout=60, kernel_name=None, **kwargs):
374
"""
375
Start new Jupyter kernel.
376
377
Args:
378
startup_timeout: Kernel startup timeout in seconds
379
kernel_name: Name of kernel to start
380
**kwargs: Additional kernel arguments
381
382
Returns:
383
Tuple of (kernel_manager, kernel_client)
384
"""
385
386
class NbvalKernelspecManager(KernelSpecManager):
387
"""Custom kernel spec manager for nbval."""
388
```
389
390
### Utility Functions
391
392
Helper functions for notebook processing and output handling.
393
394
```python { .api }
395
def find_comment_markers(cellsource):
396
"""
397
Find special comment markers in cell source.
398
399
Args:
400
cellsource: Cell source code
401
402
Returns:
403
Dictionary of found markers
404
"""
405
406
def find_metadata_tags(cell_metadata):
407
"""
408
Find nbval tags in cell metadata.
409
410
Args:
411
cell_metadata: Cell metadata dictionary
412
413
Returns:
414
Dictionary of found tags
415
"""
416
417
def coalesce_streams(outputs):
418
"""
419
Merge stream outputs for consistent results.
420
421
Args:
422
outputs: List of output dictionaries
423
424
Returns:
425
List of coalesced outputs
426
"""
427
428
def transform_streams_for_comparison(outputs):
429
"""
430
Transform stream outputs for comparison.
431
432
Args:
433
outputs: List of output dictionaries
434
435
Returns:
436
List of transformed outputs
437
"""
438
439
def get_sanitize_patterns(string):
440
"""
441
Parse regex patterns from sanitize config.
442
443
Args:
444
string: Configuration string
445
446
Returns:
447
List of (regex, replacement) tuples
448
"""
449
450
def hash_string(s):
451
"""
452
Create MD5 hash of string.
453
454
Args:
455
s: String to hash
456
457
Returns:
458
MD5 hash hexdigest
459
"""
460
461
def _trim_base64(s):
462
"""
463
Trim and hash base64 strings for cleaner output display.
464
465
Args:
466
s: String to trim if it's base64
467
468
Returns:
469
Trimmed string with hash if base64, original string otherwise
470
"""
471
472
def _indent(s, indent=' '):
473
"""
474
Indent each line with specified indentation.
475
476
Args:
477
s: String to indent
478
indent: Indentation string (default: ' ')
479
480
Returns:
481
Indented string
482
"""
483
```
484
485
### Coverage Integration
486
487
Coverage reporting setup and teardown for notebook testing.
488
489
```python { .api }
490
def setup_coverage(config, kernel, floc, output_loc):
491
"""
492
Set up coverage reporting.
493
494
Args:
495
config: pytest configuration
496
kernel: Running kernel instance
497
floc: File location
498
output_loc: Output location
499
"""
500
501
def teardown_coverage(config, kernel):
502
"""
503
Tear down coverage reporting.
504
505
Args:
506
config: pytest configuration
507
kernel: Running kernel instance
508
"""
509
```
510
511
### NBDime Integration
512
513
Reporter for visual diff display of notebook failures.
514
515
```python { .api }
516
class NbdimeReporter:
517
"""Pytest reporter for nbdime integration."""
518
519
def __init__(self, config, file=None):
520
"""
521
Initialize nbdime reporter.
522
523
Args:
524
config: pytest configuration
525
file: Output file handle
526
"""
527
528
def pytest_runtest_logreport(self, report):
529
"""
530
Handle test log reports.
531
532
Args:
533
report: pytest test report
534
"""
535
536
def pytest_collectreport(self, report):
537
"""
538
Handle collection reports.
539
540
Args:
541
report: pytest collection report
542
"""
543
544
def pytest_sessionfinish(self, exitstatus):
545
"""
546
Handle session finish.
547
548
Args:
549
exitstatus: pytest exit status
550
"""
551
552
def make_report(self, outcome):
553
"""
554
Create nbdime report.
555
556
Args:
557
outcome: Test outcome
558
559
Returns:
560
Report data
561
"""
562
```
563
564
## Command Line Options
565
566
NBVal adds the following options to pytest:
567
568
- `--nbval`: Run notebooks validating all outputs
569
- `--nbval-lax`: Run notebooks only validating marked cells
570
- `--nbval-sanitize-with FILE`: File with regex patterns to sanitize outputs
571
- `--nbval-current-env`: Use current environment kernel
572
- `--nbval-kernel-name KERNEL`: Force specific kernel name
573
- `--nbval-cell-timeout SECONDS`: Cell execution timeout
574
- `--nbval-kernel-startup-timeout SECONDS`: Kernel startup timeout
575
- `--nbdime`: View failed cells with nbdime
576
577
## Cell Control Markers
578
579
### Comment Markers
580
581
Use in cell source code:
582
583
- `# NBVAL_IGNORE_OUTPUT`: Skip output validation
584
- `# NBVAL_CHECK_OUTPUT`: Force output validation
585
- `# NBVAL_RAISES_EXCEPTION`: Expect cell to raise exception
586
- `# NBVAL_SKIP`: Skip cell execution
587
- `# PYTEST_VALIDATE_IGNORE_OUTPUT`: Legacy marker
588
589
### Metadata Tags
590
591
Use in cell metadata.tags:
592
593
- `nbval-ignore-output`, `nbval-check-output`, `nbval-raises-exception`, `nbval-skip`, `raises-exception`
594
595
## Constants
596
597
```python { .api }
598
CURRENT_ENV_KERNEL_NAME: str
599
comment_markers: dict
600
metadata_tags: dict
601
602
class bcolors:
603
"""Color codes for terminal output."""
604
605
class nocolors:
606
"""No-color version of bcolors."""
607
```
608
609
## Types
610
611
```python { .api }
612
version_info: tuple
613
```
614
615
## Dependencies
616
617
- pytest >= 7
618
- jupyter_client
619
- nbformat
620
- ipykernel
621
- coverage
622
- Optional: nbdime (for visual diff reporting)