0
# Code Block Processing
1
2
Code block analysis and transformation system that parses Python code, identifies name usage patterns, and prepares them for link generation. This system handles various Python code formats including standard code blocks, IPython console sessions, and doctests.
3
4
## Capabilities
5
6
### Core Parsing Functions
7
8
Main functions for extracting name usage from Python source code using AST analysis.
9
10
```python { .api }
11
def parse_names(source: str, doctree_node) -> list[Name]:
12
"""
13
Parse names from Python source code.
14
15
Analyzes the abstract syntax tree of the provided source code to identify
16
all name accesses, import statements, and their relationships. Tracks
17
variable assignments and scope to build accurate name chains.
18
19
Parameters:
20
- source: Python source code string to analyze
21
- doctree_node: Docutils node for error reporting context
22
23
Returns:
24
- list[Name]: List of Name objects representing all identified name accesses
25
"""
26
27
def linenos(node: ast.AST) -> tuple[int, int]:
28
"""
29
Return lineno and end_lineno safely from AST nodes.
30
31
Provides safe access to line number information from AST nodes,
32
handling cases where end_lineno might not be available.
33
34
Parameters:
35
- node: AST node to extract line numbers from
36
37
Returns:
38
- tuple[int, int]: Start line number and end line number
39
"""
40
```
41
42
### Code Block Transformers
43
44
Functions that clean and normalize different Python code formats for analysis.
45
46
```python { .api }
47
def clean_pycon(source: str) -> tuple[str, str]:
48
"""
49
Clean up Python console syntax to pure Python.
50
51
Converts Python console session format (with >>> and ... prompts)
52
to clean Python code suitable for AST analysis.
53
54
Parameters:
55
- source: Raw console session text with prompts
56
57
Returns:
58
- tuple[str, str]: (original_source, cleaned_source)
59
60
Example:
61
Input: ">>> import os\\n>>> os.path.join('a', 'b')"
62
Output: (">>> import os...", "import os\\nos.path.join('a', 'b')")
63
"""
64
65
def clean_ipython(source: str) -> tuple[str, str]:
66
"""
67
Clean up IPython magics and console syntax to pure Python.
68
69
Processes IPython console session format, removing magic commands
70
and console prompts while preserving executable Python code.
71
Requires IPython to be installed.
72
73
Parameters:
74
- source: Raw IPython session text
75
76
Returns:
77
- tuple[str, str]: (original_source, cleaned_source)
78
79
Example:
80
Input: "In [1]: %matplotlib inline\\nIn [2]: import numpy"
81
Output: ("In [1]: %matplotlib...", "import numpy")
82
"""
83
```
84
85
### Block Analysis System
86
87
Core classes for analyzing code blocks and managing transformations.
88
89
```python { .api }
90
class SourceTransform:
91
"""
92
Represents transforms on source code.
93
94
Contains the analysis results for a single code block, including
95
the source code, extracted names, and metadata for link generation.
96
"""
97
source: str # Original source code
98
names: list[Name] # Parsed name accesses
99
example: CodeExample # Metadata about the example
100
doc_lineno: int # Line number in source document
101
102
class CodeBlockAnalyser(nodes.SparseNodeVisitor):
103
"""
104
Analyzes and transforms Python code blocks with reference links.
105
106
Visits code block nodes in the document tree, extracts Python code,
107
applies transformers, parses names, and prepares for link injection.
108
"""
109
110
def __init__(self, doctree, source_dir: str, global_preface: list[str],
111
custom_blocks: dict, concat_default: bool,
112
default_highlight_lang: str, warn_default_parse_fail: bool): ...
113
```
114
115
### HTML Link Application
116
117
Functions for injecting links into generated HTML output.
118
119
```python { .api }
120
def link_html(document: str, out_dir: str, transforms: list[SourceTransform],
121
inventory: dict, custom_blocks: dict, search_css_classes: list,
122
builder_name: str = "html") -> None:
123
"""
124
Inject links into HTML code blocks on disk.
125
126
Post-processes generated HTML files to add clickable links to code
127
elements based on resolved name mappings. Modifies HTML files in place.
128
129
Parameters:
130
- document: Document name/path
131
- out_dir: Output directory containing HTML files
132
- transforms: List of source transformations with resolved names
133
- inventory: Mapping from names to documentation URLs
134
- custom_blocks: Custom block type processors
135
- search_css_classes: Additional CSS classes to search for code blocks
136
- builder_name: Sphinx builder name (default: "html")
137
"""
138
```
139
140
### Built-in Block Types
141
142
Constants defining supported code block language identifiers and their transformers.
143
144
```python { .api }
145
BUILTIN_BLOCKS: dict = {
146
# Standard Python blocks (no transformation needed)
147
"default": None,
148
"python": None,
149
"Python": None,
150
"python3": None,
151
"py": None,
152
"py3": None,
153
"pyi": None,
154
"sage": None,
155
"bazel": None,
156
"starlark": None,
157
158
# Special blocks with transformers
159
"pycon": clean_pycon, # Python console sessions
160
"ipython": clean_ipython, # IPython sessions (if available)
161
"ipython3": clean_ipython
162
}
163
```
164
165
### AST Visitor Implementation
166
167
Core visitor class that performs detailed AST analysis to track imports and name usage.
168
169
```python { .api }
170
class ImportTrackerVisitor(ast.NodeVisitor):
171
"""
172
Track imports and their use through source code.
173
174
Visits AST nodes to build a complete picture of how names are
175
imported, assigned, and accessed throughout the code. Handles
176
complex scoping rules including nested functions, classes, and
177
comprehensions.
178
"""
179
180
def __init__(self, doctree_node): ...
181
182
# Key visitor methods for different AST node types
183
def visit_Import(self, node: ast.Import): ...
184
def visit_ImportFrom(self, node: ast.ImportFrom): ...
185
def visit_Name(self, node: ast.Name) -> PendingAccess: ...
186
def visit_Attribute(self, node: ast.Attribute) -> PendingAccess: ...
187
def visit_Call(self, node: ast.Call) -> PendingAccess: ...
188
def visit_Assign(self, node: ast.Assign) -> Assignment: ...
189
def visit_FunctionDef(self, node: ast.FunctionDef): ...
190
def visit_ClassDef(self, node: ast.ClassDef): ...
191
# ... and many more AST node handlers
192
```
193
194
## Usage Examples
195
196
### Custom Block Transformer
197
198
Define a custom transformer for a specialized code format:
199
200
```python
201
# conf.py
202
def process_sage_code(source: str) -> tuple[str, str]:
203
"""Transform SageMath code for analysis."""
204
# Remove sage-specific syntax
205
cleaned = source.replace('sage:', 'python:')
206
# Apply other transformations
207
cleaned = cleaned.replace('%time', '# %time')
208
return source, cleaned
209
210
codeautolink_custom_blocks = {
211
'sage': process_sage_code,
212
'custom-python': lambda s: (s, s.replace('>>>', ''))
213
}
214
```
215
216
### Global Preface Configuration
217
218
Set up common imports for all code blocks:
219
220
```python
221
# conf.py
222
codeautolink_global_preface = """
223
import numpy as np
224
import matplotlib.pyplot as plt
225
import pandas as pd
226
227
# Common setup
228
plt.style.use('seaborn')
229
np.random.seed(42)
230
"""
231
```
232
233
### Advanced HTML Processing
234
235
The extension searches for code blocks in HTML using these patterns:
236
237
```python
238
# Default CSS classes searched
239
search_patterns = [
240
'highlight-python', # Standard Sphinx
241
'highlight-py', # Short form
242
'highlight-default', # Default language
243
'code-block', # Generic blocks
244
]
245
246
# Additional classes via configuration
247
codeautolink_search_css_classes = [
248
'custom-code',
249
'my-python-blocks'
250
]
251
```
252
253
### Error Handling in Processing
254
255
The system provides detailed error reporting for parsing failures:
256
257
```python
258
# Enable detailed warnings in conf.py
259
codeautolink_warn_on_default_parse_fail = True
260
261
# Example error output:
262
# WARNING: Could not parse code block on line 45 in document.rst:
263
# SyntaxError: invalid syntax (<string>, line 3)
264
# Source: 'def invalid_syntax('
265
```
266
267
### Supported Code Formats
268
269
The extension recognizes and processes these code block formats:
270
271
```rst
272
.. code-block:: python
273
274
import os # Standard Python
275
276
.. code-block:: pycon
277
278
>>> import sys # Python console
279
>>> sys.version
280
281
.. code-block:: ipython
282
283
In [1]: %matplotlib inline # IPython session
284
In [2]: import numpy as np
285
286
.. code-block:: py3
287
288
async def main(): # Python 3 specific
289
await some_function()
290
```