0
# AST Processing and Source Analysis
1

2
Abstract syntax tree processing for extracting Python code structure, type annotations, and source code metadata. Provides deep analysis of Python source code to generate accurate documentation with proper type information and cross-references.
3

4
## Capabilities
5

6
### AST Parsing and Analysis
7

8
Parse Python objects and source code into abstract syntax trees for analysis.
9

10
```python { .api }
11
def parse(obj: Any) -> ast.AST | None:
12
    """
13
    Parse Python object into abstract syntax tree.
14
    
15
    Parameters:
16
    - obj: Any - Python object to parse (module, class, function, etc.)
17
    
18
    Returns:
19
    - ast.AST | None: Parsed AST node, None if parsing fails
20
    
21
    Features:
22
    - Handles various Python object types
23
    - Preserves source location information
24
    - Robust error handling for unparseable objects
25
    """
26

27
def unparse(tree: ast.AST) -> str:
28
    """
29
    Convert AST back to Python source code.
30
    
31
    Parameters:
32
    - tree: ast.AST - AST node to convert
33
    
34
    Returns:
35
    - str: Python source code representation
36
    
37
    Features:
38
    - Preserves original formatting where possible
39
    - Handles complex expressions and statements
40
    - Compatible with various Python versions
41
    """
42
```
43

44
### Source Code Extraction
45

46
Extract source code and metadata from Python objects.
47

48
```python { .api }
49
def get_source(obj: Any) -> str | None:
50
    """
51
    Get source code for Python object.
52
    
53
    Parameters:
54
    - obj: Any - Python object to get source for
55
    
56
    Returns:
57
    - str | None: Source code string, None if unavailable
58
    
59
    Features:
60
    - Works with modules, classes, functions, methods
61
    - Handles dynamically created objects gracefully
62
    - Preserves indentation and formatting
63
    """
64
```
65

66
### AST Tree Traversal
67

68
Traverse and analyze AST structures recursively.
69

70
```python { .api }
71
def walk_tree(tree: ast.AST) -> Iterator[ast.AST]:
72
    """
73
    Walk AST tree recursively, yielding all nodes.
74
    
75
    Parameters:
76
    - tree: ast.AST - Root AST node to walk
77
    
78
    Yields:
79
    - ast.AST: Each AST node in depth-first order
80
    
81
    Features:
82
    - Depth-first traversal of entire tree
83
    - Handles all AST node types
84
    - Preserves parent-child relationships
85
    """
86
```
87

88
### Source-Based Sorting
89

90
Sort documentation objects by their source code position.
91

92
```python { .api }
93
def sort_by_source(items: list[Doc]) -> list[Doc]:
94
    """
95
    Sort documentation objects by source file position.
96
    
97
    Parameters:
98
    - items: list[Doc] - Documentation objects to sort
99
    
100
    Returns:
101
    - list[Doc]: Sorted list in source file order
102
    
103
    Features:
104
    - Maintains natural source code ordering
105
    - Handles objects from multiple files
106
    - Preserves relative positions within files
107
    """
108
```
109

110
### Type Checking Analysis
111

112
Analyze TYPE_CHECKING blocks and conditional imports.
113

114
```python { .api }
115
def type_checking_sections(tree: ast.AST) -> list[tuple[int, int]]:
116
    """
117
    Find TYPE_CHECKING conditional blocks in AST.
118
    
119
    Parameters:
120
    - tree: ast.AST - Module AST to analyze
121
    
122
    Returns:
123
    - list[tuple[int, int]]: List of (start_line, end_line) for TYPE_CHECKING blocks
124
    
125
    Features:
126
    - Identifies typing.TYPE_CHECKING conditions
127
    - Handles various conditional patterns
128
    - Supports forward reference resolution
129
    """
130
```
131

132
## Data Structures
133

134
### AST Information Container
135

136
Container for AST-related metadata and analysis results.
137

138
```python { .api }
139
class AstInfo:
140
    """
141
    Container for AST information and metadata.
142
    
143
    Stores parsed AST data along with source location,
144
    type information, and analysis results.
145
    """
146
    pass
147
```
148

149
## Usage Examples
150

151
### Basic AST Analysis
152

153
```python
154
from pdoc.doc_ast import parse, get_source, unparse
155
import ast
156

157
def analyze_function(func):
158
    """Analyze a function's AST structure"""
159
    
160
    # Get source code
161
    source = get_source(func)
162
    if source:
163
        print(f"Source code:\n{source}")
164
    
165
    # Parse into AST
166
    tree = parse(func)
167
    if tree:
168
        print(f"AST type: {type(tree).__name__}")
169
        
170
        # Convert back to source
171
        reconstructed = unparse(tree)
172
        print(f"Reconstructed:\n{reconstructed}")
173

174
# Example usage
175
def example_function(x: int, y: str = "default") -> bool:
176
    """Example function for AST analysis."""
177
    return len(y) > x
178

179
analyze_function(example_function)
180
```
181

182
### AST Tree Walking
183

184
```python
185
from pdoc.doc_ast import parse, walk_tree
186
import ast
187

188
def analyze_ast_structure(obj):
189
    """Walk through AST and analyze structure"""
190
    
191
    tree = parse(obj)
192
    if not tree:
193
        return
194
        
195
    node_counts = {}
196
    for node in walk_tree(tree):
197
        node_type = type(node).__name__
198
        node_counts[node_type] = node_counts.get(node_type, 0) + 1
199
        
200
        # Print function definitions
201
        if isinstance(node, ast.FunctionDef):
202
            print(f"Function: {node.name}")
203
            print(f"  Args: {len(node.args.args)}")
204
            print(f"  Line: {node.lineno}")
205
        
206
        # Print class definitions
207
        elif isinstance(node, ast.ClassDef):
208
            print(f"Class: {node.name}")
209
            print(f"  Bases: {len(node.bases)}")
210
            print(f"  Line: {node.lineno}")
211
    
212
    print(f"\nAST node counts: {node_counts}")
213

214
# Example usage
215
class ExampleClass:
216
    def method1(self, x: int) -> str:
217
        return str(x)
218
    
219
    def method2(self, y: list) -> int:
220
        return len(y)
221

222
analyze_ast_structure(ExampleClass)
223
```
224

225
### Source Code Ordering
226

227
```python
228
from pdoc.doc_ast import sort_by_source
229
from pdoc.doc import Module
230

231
def display_source_order(module_name: str):
232
    """Display module members in source file order"""
233
    
234
    module = Module.from_name(module_name)
235
    
236
    # Get all members as list
237
    members = list(module.members.values())
238
    
239
    # Sort by source position
240
    sorted_members = sort_by_source(members)
241
    
242
    print(f"Members of {module_name} in source order:")
243
    for member in sorted_members:
244
        print(f"  {member.name} ({type(member).__name__})")
245

246
# Example usage
247
display_source_order("math")
248
```
249

250
### Type Checking Analysis
251

252
```python
253
from pdoc.doc_ast import type_checking_sections, parse
254
import ast
255

256
def analyze_type_checking(module_source: str):
257
    """Analyze TYPE_CHECKING blocks in module source"""
258
    
259
    # Parse module source
260
    tree = ast.parse(module_source)
261
    
262
    # Find TYPE_CHECKING sections
263
    sections = type_checking_sections(tree)
264
    
265
    if sections:
266
        print("TYPE_CHECKING blocks found:")
267
        for start, end in sections:
268
            print(f"  Lines {start}-{end}")
269
            
270
            # Extract the conditional block
271
            lines = module_source.split('\n')
272
            block_lines = lines[start-1:end]
273
            print("  Content:")
274
            for line in block_lines:
275
                print(f"    {line}")
276
    else:
277
        print("No TYPE_CHECKING blocks found")
278

279
# Example module source with TYPE_CHECKING
280
example_source = '''
281
from typing import TYPE_CHECKING
282

283
if TYPE_CHECKING:
284
    from collections.abc import Sequence
285
    from typing import Optional
286

287
def process_data(items):
288
    return len(items)
289
'''
290

291
analyze_type_checking(example_source)
292
```
293

294
### Advanced AST Processing
295

296
```python
297
from pdoc.doc_ast import parse, walk_tree, get_source
298
import ast
299

300
class DocstringExtractor(ast.NodeVisitor):
301
    """Custom AST visitor to extract docstrings"""
302
    
303
    def __init__(self):
304
        self.docstrings = {}
305
    
306
    def visit_FunctionDef(self, node):
307
        """Visit function definitions and extract docstrings"""
308
        docstring = ast.get_docstring(node)
309
        if docstring:
310
            self.docstrings[node.name] = docstring
311
        self.generic_visit(node)
312
    
313
    def visit_ClassDef(self, node):
314
        """Visit class definitions and extract docstrings"""
315
        docstring = ast.get_docstring(node)
316
        if docstring:
317
            self.docstrings[node.name] = docstring
318
        self.generic_visit(node)
319

320
def extract_all_docstrings(obj):
321
    """Extract all docstrings from Python object"""
322
    
323
    source = get_source(obj)
324
    if not source:
325
        return {}
326
        
327
    tree = ast.parse(source)
328
    extractor = DocstringExtractor()
329
    extractor.visit(tree)
330
    
331
    return extractor.docstrings
332

333
# Example usage
334
class Example:
335
    """Class docstring"""
336
    
337
    def method(self):
338
        """Method docstring"""
339
        pass
340

341
docstrings = extract_all_docstrings(Example)
342
for name, doc in docstrings.items():
343
    print(f"{name}: {doc}")
344
```
345

346
### Integration with Documentation Generation
347

348
```python
349
from pdoc.doc_ast import parse, sort_by_source, get_source
350
from pdoc.doc import Module, Function, Class
351

352
def enhanced_module_analysis(module_name: str):
353
    """Enhanced module analysis using AST processing"""
354
    
355
    module = Module.from_name(module_name)
356
    
357
    # Sort members by source order
358
    sorted_members = sort_by_source(list(module.members.values()))
359
    
360
    print(f"Analysis of {module_name}:")
361
    
362
    for member in sorted_members:
363
        print(f"\n{member.name} ({type(member).__name__}):")
364
        
365
        # Get source code
366
        source = get_source(member.obj)
367
        if source:
368
            lines = len(source.split('\n'))
369
            print(f"  Source lines: {lines}")
370
        
371
        # Parse AST for additional analysis
372
        tree = parse(member.obj)
373
        if tree:
374
            if isinstance(member, Function) and isinstance(tree, ast.FunctionDef):
375
                print(f"  Parameters: {len(tree.args.args)}")
376
                print(f"  Has return annotation: {tree.returns is not None}")
377
            elif isinstance(member, Class) and isinstance(tree, ast.ClassDef):
378
                print(f"  Base classes: {len(tree.bases)}")
379
                methods = [n for n in tree.body if isinstance(n, ast.FunctionDef)]
380
                print(f"  Methods: {len(methods)}")
381

382
# Example usage
383
enhanced_module_analysis("json")
384
```
385

386
## Error Handling
387

388
The AST processing system handles various error conditions gracefully:
389

390
- **Unparseable source**: Returns None for objects without accessible source
391
- **Syntax errors**: Catches and logs parsing errors without crashing
392
- **Missing files**: Handles cases where source files are not available
393
- **Dynamic objects**: Gracefully handles runtime-created objects
394
- **Encoding issues**: Robust handling of various source file encodings