0
# NBVal
1

2
A py.test plugin to validate Jupyter notebooks by executing notebook cells and comparing their outputs against stored results. NBVal enables automated testing of notebook content to ensure consistency and functionality, supporting features like output sanitization through regex patterns, parallel execution compatibility, coverage integration, and flexible kernel selection.
3

4
## Package Information
5

6
- **Package Name**: nbval
7
- **Package Type**: pytest plugin
8
- **Language**: Python
9
- **Installation**: `pip install nbval`
10
- **Requirements**: Python >=3.7, <4
11

12
## Core Imports
13

14
```python
15
import nbval
16
```
17

18
For accessing plugin functionality directly:
19

20
```python
21
from nbval.plugin import IPyNbFile, IPyNbCell, NbCellError
22
from nbval.kernel import RunningKernel, start_new_kernel
23
```
24

25
## Basic Usage
26

27
NBVal works as a pytest plugin, activated through command-line flags:
28

29
```bash
30
# Run notebooks validating all outputs
31
pytest --nbval
32

33
# Run notebooks only validating marked cells
34
pytest --nbval-lax
35

36
# Test a specific notebook
37
pytest --nbval my_notebook.ipynb
38

39
# Use output sanitization
40
pytest --nbval my_notebook.ipynb --nbval-sanitize-with sanitize_config.cfg
41

42
# Use current environment kernel
43
pytest --nbval --nbval-current-env
44

45
# Use specific kernel
46
pytest --nbval --nbval-kernel-name python3
47
```
48

49
### Cell Markers
50

51
Control cell behavior using comment markers or metadata tags:
52

53
```python
54
# NBVAL_IGNORE_OUTPUT - Skip output validation
55
print("This output won't be checked")
56

57
# NBVAL_CHECK_OUTPUT - Force output validation (useful with --nbval-lax)
58
print("This output will be validated")
59

60
# NBVAL_RAISES_EXCEPTION - Expect cell to raise exception
61
raise ValueError("Expected error")
62

63
# NBVAL_SKIP - Skip cell execution entirely
64
# This cell won't run
65
```
66

67
### Output Sanitization Configuration
68

69
Create a sanitization file to clean outputs before comparison:
70

71
```ini
72
[timestamps]
73
regex: \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
74
replace: TIMESTAMP
75

76
[memory_addresses]
77
regex: 0x[0-9a-fA-F]+
78
replace: 0xADDRESS
79
```
80

81
## Capabilities
82

83
### Package Version Access
84

85
Access to package version information.
86

87
```python { .api }
88
__version__: str
89
```
90

91
### Pytest Plugin Hooks
92

93
Core pytest integration hooks for notebook collection and configuration.
94

95
```python { .api }
96
def pytest_addoption(parser):
97
    """
98
    Add nbval command-line options to pytest.
99
    
100
    Args:
101
        parser: pytest argument parser
102
    """
103

104
def pytest_configure(config):
105
    """
106
    Configure nbval plugin based on options.
107
    
108
    Args:
109
        config: pytest configuration object
110
    """
111

112
def pytest_collect_file(file_path, parent):
113
    """
114
    Collect .ipynb files for testing.
115
    
116
    Args:
117
        file_path: Path to potential test file
118
        parent: Parent collector
119
        
120
    Returns:
121
        IPyNbFile instance for .ipynb files, None otherwise
122
    """
123
```
124

125
### Notebook File Collection
126

127
Pytest collector for handling notebook files as test suites.
128

129
```python { .api }
130
class IPyNbFile(pytest.File):
131
    """
132
    Pytest collector for notebook files.
133
    """
134
    
135
    def setup(self):
136
        """Set up notebook execution environment."""
137
    
138
    def collect(self):
139
        """
140
        Collect notebook cells as test items.
141
        
142
        Returns:
143
            List of IPyNbCell instances
144
        """
145
    
146
    def teardown(self):
147
        """Clean up notebook execution environment."""
148
    
149
    def get_kernel_message(self, timeout=None):
150
        """
151
        Get message from running kernel.
152
        
153
        Args:
154
            timeout: Message timeout in seconds
155
            
156
        Returns:
157
            Kernel message dictionary
158
        """
159
    
160
    def setup_sanitize_files(self):
161
        """Set up output sanitization configuration."""
162
    
163
    def get_sanitize_files(self):
164
        """
165
        Get sanitization configuration.
166
        
167
        Returns:
168
            List of sanitization patterns
169
        """
170
```
171

172
### Notebook Cell Testing
173

174
Pytest item representing individual notebook cells for execution and validation.
175

176
```python { .api }
177
class IPyNbCell(pytest.Item):
178
    """
179
    Pytest item representing a notebook cell.
180
    """
181
    
182
    def runtest(self):
183
        """Execute cell and compare outputs."""
184
    
185
    def repr_failure(self, excinfo):
186
        """
187
        Represent test failure for reporting.
188
        
189
        Args:
190
            excinfo: Exception information
191
            
192
        Returns:
193
            Formatted failure representation
194
        """
195
    
196
    def reportinfo(self):
197
        """
198
        Get test location information.
199
        
200
        Returns:
201
            Tuple of (file_path, line_number, test_name)
202
        """
203
    
204
    def compare_outputs(self, test, ref, skip_compare=None):
205
        """
206
        Compare cell outputs against reference.
207
        
208
        Args:
209
            test: Actual execution output
210
            ref: Reference output from notebook
211
            skip_compare: Output types to skip comparison
212
            
213
        Returns:
214
            True if outputs match, False otherwise
215
        """
216
    
217
    def setup(self):
218
        """Set up cell execution environment."""
219
    
220
    def raise_cell_error(self, **kwargs):
221
        """
222
        Raise formatted cell execution error.
223
        
224
        Args:
225
            **kwargs: Error details
226
            
227
        Raises:
228
            NbCellError: Formatted cell error
229
        """
230
    
231
    def sanitize(self, s):
232
        """
233
        Apply output sanitization patterns.
234
        
235
        Args:
236
            s: String to sanitize
237
            
238
        Returns:
239
            Sanitized string
240
        """
241
    
242
    def sanitize_outputs(self, outputs):
243
        """
244
        Apply sanitization to output list.
245
        
246
        Args:
247
            outputs: List of output dictionaries
248
            
249
        Returns:
250
            List of sanitized outputs
251
        """
252
```
253

254
### Error Handling
255

256
Custom exception for notebook cell execution errors.
257

258
```python { .api }
259
class NbCellError(Exception):
260
    """
261
    Custom exception for cell execution errors.
262
    
263
    Attributes:
264
        cell_num: Cell number that failed
265
        source: Cell source code
266
        inner_traceback: Original exception traceback
267
    """
268
    
269
    def __init__(self, cell_num, msg, source, traceback=None, *args, **kwargs):
270
        """
271
        Initialize cell error.
272
        
273
        Args:
274
            cell_num: Failed cell number
275
            msg: Error message
276
            source: Cell source code
277
            traceback: Original traceback
278
            *args: Additional arguments
279
            **kwargs: Additional keyword arguments
280
        """
281
```
282

283
### Kernel Management
284

285
Jupyter kernel execution and lifecycle management.
286

287
```python { .api }
288
class RunningKernel:
289
    """
290
    Manages Jupyter kernel execution.
291
    """
292
    
293
    def __init__(self, kernel_name, cwd=None, startup_timeout=60):
294
        """
295
        Initialize kernel manager.
296
        
297
        Args:
298
            kernel_name: Name of kernel to start
299
            cwd: Working directory for kernel
300
            startup_timeout: Kernel startup timeout in seconds
301
        """
302
    
303
    def execute_cell_input(self, cell_input, allow_stdin=True):
304
        """
305
        Execute cell input in kernel.
306
        
307
        Args:
308
            cell_input: Cell source code to execute
309
            allow_stdin: Whether to allow stdin
310
            
311
        Returns:
312
            Execution message ID
313
        """
314
    
315
    def get_message(self, timeout=None):
316
        """
317
        Get message from kernel.
318
        
319
        Args:
320
            timeout: Message timeout in seconds
321
            
322
        Returns:
323
            Kernel message dictionary
324
        """
325
    
326
    def await_reply(self, msg_id, timeout=None):
327
        """
328
        Wait for execution reply.
329
        
330
        Args:
331
            msg_id: Message ID to wait for
332
            timeout: Reply timeout in seconds
333
            
334
        Returns:
335
            Reply message dictionary
336
        """
337
    
338
    def await_idle(self, msg_id, timeout=None):
339
        """
340
        Wait for kernel to become idle.
341
        
342
        Args:
343
            msg_id: Message ID to wait for
344
            timeout: Idle timeout in seconds
345
        """
346
    
347
    def is_alive(self):
348
        """
349
        Check if kernel is alive.
350
        
351
        Returns:
352
            True if kernel is running, False otherwise
353
        """
354
    
355
    def restart(self):
356
        """Restart the kernel."""
357
    
358
    def interrupt(self):
359
        """Interrupt kernel execution."""
360
    
361
    def stop(self):
362
        """Stop and cleanup kernel."""
363
    
364
    @property
365
    def language(self):
366
        """
367
        Get kernel language.
368
        
369
        Returns:
370
            Kernel language name
371
        """
372

373
def start_new_kernel(startup_timeout=60, kernel_name=None, **kwargs):
374
    """
375
    Start new Jupyter kernel.
376
    
377
    Args:
378
        startup_timeout: Kernel startup timeout in seconds
379
        kernel_name: Name of kernel to start
380
        **kwargs: Additional kernel arguments
381
        
382
    Returns:
383
        Tuple of (kernel_manager, kernel_client)
384
    """
385

386
class NbvalKernelspecManager(KernelSpecManager):
387
    """Custom kernel spec manager for nbval."""
388
```
389

390
### Utility Functions
391

392
Helper functions for notebook processing and output handling.
393

394
```python { .api }
395
def find_comment_markers(cellsource):
396
    """
397
    Find special comment markers in cell source.
398
    
399
    Args:
400
        cellsource: Cell source code
401
        
402
    Returns:
403
        Dictionary of found markers
404
    """
405

406
def find_metadata_tags(cell_metadata):
407
    """
408
    Find nbval tags in cell metadata.
409
    
410
    Args:
411
        cell_metadata: Cell metadata dictionary
412
        
413
    Returns:
414
        Dictionary of found tags
415
    """
416

417
def coalesce_streams(outputs):
418
    """
419
    Merge stream outputs for consistent results.
420
    
421
    Args:
422
        outputs: List of output dictionaries
423
        
424
    Returns:
425
        List of coalesced outputs
426
    """
427

428
def transform_streams_for_comparison(outputs):
429
    """
430
    Transform stream outputs for comparison.
431
    
432
    Args:
433
        outputs: List of output dictionaries
434
        
435
    Returns:
436
        List of transformed outputs
437
    """
438

439
def get_sanitize_patterns(string):
440
    """
441
    Parse regex patterns from sanitize config.
442
    
443
    Args:
444
        string: Configuration string
445
        
446
    Returns:
447
        List of (regex, replacement) tuples
448
    """
449

450
def hash_string(s):
451
    """
452
    Create MD5 hash of string.
453
    
454
    Args:
455
        s: String to hash
456
        
457
    Returns:
458
        MD5 hash hexdigest
459
    """
460

461
def _trim_base64(s):
462
    """
463
    Trim and hash base64 strings for cleaner output display.
464
    
465
    Args:
466
        s: String to trim if it's base64
467
        
468
    Returns:
469
        Trimmed string with hash if base64, original string otherwise
470
    """
471

472
def _indent(s, indent='  '):
473
    """
474
    Indent each line with specified indentation.
475
    
476
    Args:
477
        s: String to indent
478
        indent: Indentation string (default: '  ')
479
        
480
    Returns:
481
        Indented string
482
    """
483
```
484

485
### Coverage Integration
486

487
Coverage reporting setup and teardown for notebook testing.
488

489
```python { .api }
490
def setup_coverage(config, kernel, floc, output_loc):
491
    """
492
    Set up coverage reporting.
493
    
494
    Args:
495
        config: pytest configuration
496
        kernel: Running kernel instance
497
        floc: File location
498
        output_loc: Output location
499
    """
500

501
def teardown_coverage(config, kernel):
502
    """
503
    Tear down coverage reporting.
504
    
505
    Args:
506
        config: pytest configuration
507
        kernel: Running kernel instance
508
    """
509
```
510

511
### NBDime Integration
512

513
Reporter for visual diff display of notebook failures.
514

515
```python { .api }
516
class NbdimeReporter:
517
    """Pytest reporter for nbdime integration."""
518
    
519
    def __init__(self, config, file=None):
520
        """
521
        Initialize nbdime reporter.
522
        
523
        Args:
524
            config: pytest configuration
525
            file: Output file handle
526
        """
527
    
528
    def pytest_runtest_logreport(self, report):
529
        """
530
        Handle test log reports.
531
        
532
        Args:
533
            report: pytest test report
534
        """
535
    
536
    def pytest_collectreport(self, report):
537
        """
538
        Handle collection reports.
539
        
540
        Args:
541
            report: pytest collection report
542
        """
543
    
544
    def pytest_sessionfinish(self, exitstatus):
545
        """
546
        Handle session finish.
547
        
548
        Args:
549
            exitstatus: pytest exit status
550
        """
551
    
552
    def make_report(self, outcome):
553
        """
554
        Create nbdime report.
555
        
556
        Args:
557
            outcome: Test outcome
558
            
559
        Returns:
560
            Report data
561
        """
562
```
563

564
## Command Line Options
565

566
NBVal adds the following options to pytest:
567

568
- `--nbval`: Run notebooks validating all outputs
569
- `--nbval-lax`: Run notebooks only validating marked cells
570
- `--nbval-sanitize-with FILE`: File with regex patterns to sanitize outputs  
571
- `--nbval-current-env`: Use current environment kernel
572
- `--nbval-kernel-name KERNEL`: Force specific kernel name
573
- `--nbval-cell-timeout SECONDS`: Cell execution timeout
574
- `--nbval-kernel-startup-timeout SECONDS`: Kernel startup timeout
575
- `--nbdime`: View failed cells with nbdime
576

577
## Cell Control Markers
578

579
### Comment Markers
580

581
Use in cell source code:
582

583
- `# NBVAL_IGNORE_OUTPUT`: Skip output validation
584
- `# NBVAL_CHECK_OUTPUT`: Force output validation
585
- `# NBVAL_RAISES_EXCEPTION`: Expect cell to raise exception  
586
- `# NBVAL_SKIP`: Skip cell execution
587
- `# PYTEST_VALIDATE_IGNORE_OUTPUT`: Legacy marker
588

589
### Metadata Tags
590

591
Use in cell metadata.tags:
592

593
- `nbval-ignore-output`, `nbval-check-output`, `nbval-raises-exception`, `nbval-skip`, `raises-exception`
594

595
## Constants
596

597
```python { .api }
598
CURRENT_ENV_KERNEL_NAME: str
599
comment_markers: dict
600
metadata_tags: dict
601

602
class bcolors:
603
    """Color codes for terminal output."""
604
    
605
class nocolors:
606
    """No-color version of bcolors."""
607
```
608

609
## Types
610

611
```python { .api }
612
version_info: tuple
613
```
614

615
## Dependencies
616

617
- pytest >= 7
618
- jupyter_client  
619
- nbformat
620
- ipykernel
621
- coverage
622
- Optional: nbdime (for visual diff reporting)