0
# PYI Parsing
1

2
Parser for Python type stub files (.pyi) that converts textual type declarations into PyTD AST representations for analysis and manipulation. The PYI parser enables PyType to work with existing type stub files and generate canonical type declarations.
3

4
## Capabilities
5

6
### String Parsing
7

8
Parse .pyi content from strings into PyTD AST format, providing the core parsing functionality for type stub processing.
9

10
```python { .api }
11
def parse_string(src, filename=None, options=None):
12
    """
13
    Parse .pyi string content to PyTD AST.
14
    
15
    Parameters:
16
    - src (str): .pyi file content as string
17
    - filename (str, optional): Filename for error reporting
18
    - options (PyiOptions, optional): Parser configuration options
19
    
20
    Returns:
21
    pytd.TypeDeclUnit: PyTD AST representation of the type declarations
22
    
23
    Raises:
24
    ParseError: If the .pyi content contains syntax errors
25
    """
26
```
27

28
Example usage:
29

30
```python
31
from pytype.pyi import parser
32

33
pyi_content = '''
34
from typing import List, Optional
35

36
class Calculator:
37
    def __init__(self) -> None: ...
38
    def add(self, x: int, y: int) -> int: ...
39
    def divide(self, x: int, y: int) -> Optional[float]: ...
40
    
41
def create_calculator() -> Calculator: ...
42

43
PI: float
44
'''
45

46
try:
47
    ast = parser.parse_string(pyi_content, filename="calculator.pyi")
48
    print(f"Parsed {len(ast.classes)} classes and {len(ast.functions)} functions")
49
except parser.ParseError as e:
50
    print(f"Parse error: {e}")
51
```
52

53
### File Parsing
54

55
Parse .pyi files directly from the filesystem with comprehensive error handling and options support.
56

57
```python { .api }
58
def parse_pyi(src, filename, options):
59
    """
60
    Parse .pyi file with full option support.
61
    
62
    Parameters:
63
    - src (str): .pyi file content
64
    - filename (str): Path to .pyi file for error reporting
65
    - options (PyiOptions): Complete parser configuration
66
    
67
    Returns:
68
    pytd.TypeDeclUnit: Parsed PyTD AST
69
    
70
    Raises:
71
    ParseError: If parsing fails due to syntax errors or invalid declarations
72
    """
73
```
74

75
Example usage:
76

77
```python
78
from pytype.pyi import parser
79

80
# Configure parser options
81
options = parser.PyiOptions()
82
options.python_version = (3, 11)
83

84
# Read and parse file
85
with open("mymodule.pyi", "r") as f:
86
    content = f.read()
87

88
ast = parser.parse_pyi(content, "mymodule.pyi", options)
89
```
90

91
### Canonical PYI Generation
92

93
Generate canonical .pyi representations from PyTD ASTs, ensuring consistent formatting and optimization of type declarations.
94

95
```python { .api }
96
def canonical_pyi(pyi, multiline_args=False, options=None):
97
    """
98
    Generate canonical .pyi string from PyTD AST.
99
    
100
    Parameters:
101
    - pyi (pytd.TypeDeclUnit): PyTD AST to convert
102
    - multiline_args (bool): Whether to format function arguments across multiple lines
103
    - options (PyiOptions, optional): Formatting options
104
    
105
    Returns:
106
    str: Canonical .pyi file content with consistent formatting
107
    """
108
```
109

110
Example usage:
111

112
```python
113
from pytype.pyi import parser
114
from pytype import io
115

116
# Generate AST from source code
117
source = '''
118
class DataProcessor:
119
    def __init__(self, config=None):
120
        self.config = config or {}
121
    
122
    def process(self, data):
123
        return [item.strip().upper() for item in data if item]
124
'''
125

126
ast = io.generate_pyi_ast(source)
127

128
# Convert to canonical .pyi format
129
canonical = parser.canonical_pyi(ast, multiline_args=True)
130
print(canonical)
131
```
132

133
### Parser Configuration
134

135
Configure parser behavior through options class for different Python versions and parsing modes.
136

137
```python { .api }
138
class PyiOptions:
139
    """
140
    Configuration options for .pyi parsing.
141
    
142
    Controls parser behavior including Python version compatibility,
143
    error handling, and output formatting preferences.
144
    """
145
    
146
    def __init__(self):
147
        """Initialize parser options with defaults."""
148
        
149
    python_version: tuple       # Target Python version (e.g., (3, 11))
150
    strict_import: bool         # Strict import checking
151
    preserve_union_order: bool  # Maintain union type order
152
    verify: bool               # Verify parsed AST correctness
153
```
154

155
Example configuration:
156

157
```python
158
from pytype.pyi import parser
159

160
# Configure for Python 3.11 with strict checking
161
options = parser.PyiOptions()
162
options.python_version = (3, 11)
163
options.strict_import = True
164
options.verify = True
165

166
# Use in parsing
167
ast = parser.parse_string(pyi_content, options=options)
168
```
169

170
### Error Handling
171

172
Comprehensive error handling for malformed .pyi files and parsing failures.
173

174
```python { .api }
175
class ParseError(Exception):
176
    """
177
    Exception raised when .pyi parsing fails.
178
    
179
    Provides detailed information about syntax errors, invalid type
180
    declarations, and other parsing failures.
181
    """
182
    
183
    def __init__(self, message, filename=None, lineno=None):
184
        """
185
        Initialize parse error.
186
        
187
        Parameters:
188
        - message (str): Error description
189
        - filename (str, optional): File where error occurred
190
        - lineno (int, optional): Line number of error
191
        """
192
```
193

194
Example error handling:
195

196
```python
197
from pytype.pyi import parser
198

199
malformed_pyi = '''
200
class InvalidClass:
201
    def bad_method(self, x: InvalidType) -> None: ...  # Unknown type
202
    def missing_colon(self, x int) -> None: ...        # Syntax error
203
'''
204

205
try:
206
    ast = parser.parse_string(malformed_pyi, filename="test.pyi")
207
except parser.ParseError as e:
208
    print(f"Parse error in {e.filename}:{e.lineno}: {e.message}")
209
```
210

211
### Advanced Parsing Features
212

213
Support for complex type declarations and modern Python typing features.
214

215
```python
216
from pytype.pyi import parser
217

218
# Parse complex type declarations
219
complex_pyi = '''
220
from typing import Generic, TypeVar, Protocol, Literal, Union
221
from collections.abc import Callable, Iterable
222

223
T = TypeVar('T')
224
U = TypeVar('U', bound=str)
225

226
class Container(Generic[T]):
227
    def __init__(self, items: list[T]) -> None: ...
228
    def get(self, index: int) -> T: ...
229
    def map(self, func: Callable[[T], U]) -> Container[U]: ...
230

231
class Processor(Protocol):
232
    def process(self, data: str) -> str: ...
233

234
Status = Literal['pending', 'running', 'completed', 'failed']
235

236
def handle_result(
237
    result: Union[str, int, None],
238
    processor: Processor,
239
    status: Status = 'pending'
240
) -> dict[str, str]: ...
241
'''
242

243
ast = parser.parse_string(complex_pyi)
244
print(f"Parsed {len(ast.type_params)} type parameters")
245
print(f"Found {len([c for c in ast.classes if c.template])} generic classes")
246
```
247

248
### Integration with PyType Workflow
249

250
The PYI parser integrates seamlessly with PyType's analysis workflow:
251

252
```python
253
from pytype.pyi import parser
254
from pytype.pytd import pytd_utils
255
from pytype import io, config
256

257
# 1. Parse existing stub file
258
with open("library.pyi", "r") as f:
259
    existing_stub = f.read()
260

261
existing_ast = parser.parse_string(existing_stub, filename="library.pyi")
262

263
# 2. Generate new stub from source
264
source_code = open("library.py", "r").read()
265
options = config.Options.create()
266
new_ast = io.generate_pyi_ast(source_code, options=options)
267

268
# 3. Combine ASTs
269
combined_ast = pytd_utils.Concat(existing_ast, new_ast)
270

271
# 4. Generate canonical output
272
canonical_stub = parser.canonical_pyi(combined_ast, multiline_args=True)
273

274
# 5. Write updated stub
275
with open("library_updated.pyi", "w") as f:
276
    f.write(canonical_stub)
277
```
278

279
### Performance Considerations
280

281
For large .pyi files or batch processing:
282

283
```python
284
from pytype.pyi import parser
285
import os
286

287
def parse_stub_directory(stub_dir):
288
    """Parse all .pyi files in a directory efficiently."""
289
    
290
    # Reuse options for better performance
291
    options = parser.PyiOptions()
292
    options.python_version = (3, 11)
293
    
294
    parsed_asts = {}
295
    
296
    for filename in os.listdir(stub_dir):
297
        if filename.endswith('.pyi'):
298
            filepath = os.path.join(stub_dir, filename)
299
            with open(filepath, 'r') as f:
300
                content = f.read()
301
            
302
            try:
303
                ast = parser.parse_pyi(content, filename, options)
304
                parsed_asts[filename] = ast
305
            except parser.ParseError as e:
306
                print(f"Failed to parse {filename}: {e}")
307
    
308
    return parsed_asts
309
```