0
# Module Loading
1

2
Infrastructure for loading and resolving Python modules, PyTD files, and type information with support for custom import mappings and pickled type data. The module loading system enables PyType to analyze complex projects with intricate dependency structures.
3

4
## Capabilities
5

6
### Loader Creation
7

8
Create configured loaders for different analysis scenarios, providing the foundation for module resolution and type information loading.
9

10
```python { .api }
11
def create_loader(options, missing_modules=()):
12
    """
13
    Create PyTD loader with specified configuration.
14
    
15
    Parameters:
16
    - options (Options): Configuration options including Python paths,
17
                        import mappings, and module resolution settings
18
    - missing_modules (tuple): Modules to treat as missing/unavailable
19
    
20
    Returns:
21
    Loader: Configured loader instance for module resolution
22
    
23
    Raises:
24
    ImportError: If loader configuration fails
25
    """
26
```
27

28
Example usage:
29

30
```python
31
from pytype import config, load_pytd
32

33
# Create options with custom Python path
34
options = config.Options.create()
35
options.pythonpath = ["/my/project/src", "/my/project/lib"]
36
options.imports_map = "/my/project/.pytype/imports.txt"
37

38
# Create loader with known missing modules
39
loader = load_pytd.create_loader(
40
    options, 
41
    missing_modules=("unavailable_dep", "optional_package")
42
)
43
```
44

45
### Loader Class
46

47
Main loader class providing comprehensive module resolution and type information loading capabilities.
48

49
```python { .api }
50
class Loader:
51
    """
52
    Main loader for PyTD modules and type information.
53
    
54
    Handles module resolution, import mapping, caching, and provides
55
    the core infrastructure for PyType's analysis process.
56
    """
57
    
58
    def __init__(self, python_version, python_exe, module_utils):
59
        """
60
        Initialize loader with environment configuration.
61
        
62
        Parameters:
63
        - python_version (tuple): Target Python version
64
        - python_exe (str): Path to Python executable
65
        - module_utils: Module resolution utilities
66
        """
67
    
68
    def load_file(self, module_name, filename):
69
        """
70
        Load module from specific file path.
71
        
72
        Parameters:
73
        - module_name (str): Module name for loading
74
        - filename (str): Path to module file
75
        
76
        Returns:
77
        Module: Loaded module with type information
78
        """
79
    
80
    def import_name(self, module_name):
81
        """
82
        Import module by name with full resolution.
83
        
84
        Parameters:
85
        - module_name (str): Fully qualified module name
86
        
87
        Returns:
88
        Module: Loaded module with resolved dependencies
89
        
90
        Raises:
91
        ImportError: If module cannot be resolved or loaded
92
        """
93
    
94
    def lookup_builtin(self, name):
95
        """
96
        Look up built-in type or function.
97
        
98
        Parameters:
99
        - name (str): Built-in name to resolve
100
        
101
        Returns:
102
        Type information for built-in, or None if not found
103
        """
104
    
105
    def has_module(self, module_name):
106
        """
107
        Check if module is available for loading.
108
        
109
        Parameters:
110
        - module_name (str): Module name to check
111
        
112
        Returns:
113
        bool: True if module can be loaded
114
        """
115
```
116

117
Example usage:
118

119
```python
120
from pytype import load_pytd, config
121

122
# Create and use loader
123
options = config.Options.create()
124
loader = load_pytd.create_loader(options)
125

126
# Load specific modules
127
try:
128
    os_module = loader.import_name("os")
129
    json_module = loader.import_name("json")
130
    
131
    # Check module availability
132
    if loader.has_module("requests"):
133
        requests_module = loader.import_name("requests")
134
        
135
except ImportError as e:
136
    print(f"Failed to load module: {e}")
137
```
138

139
### Module Representation
140

141
Classes for representing loaded modules and their metadata.
142

143
```python { .api }
144
class Module:
145
    """
146
    Represents a parsed and loaded Python module.
147
    
148
    Contains the PyTD AST representation along with metadata
149
    about the module's source and dependencies.
150
    """
151
    
152
    def __init__(self, ast, filename=None, module_name=None):
153
        """
154
        Initialize module representation.
155
        
156
        Parameters:
157
        - ast (TypeDeclUnit): PyTD AST for the module
158
        - filename (str, optional): Source file path
159
        - module_name (str, optional): Fully qualified module name
160
        """
161

162
class ResolvedModule:
163
    """
164
    Dataclass containing resolved module information.
165
    
166
    Provides complete information about a successfully resolved
167
    module including its location, type, and metadata.
168
    """
169
    
170
    def __init__(self, module_name, filename, ast):
171
        """
172
        Initialize resolved module information.
173
        
174
        Parameters:
175
        - module_name (str): Fully qualified module name
176
        - filename (str): Path to module source
177
        - ast (TypeDeclUnit): Parsed PyTD AST
178
        """
179
```
180

181
### Pickled PyI Loader
182

183
Specialized loader for working with pickled .pyi files, enabling faster loading of pre-processed type information.
184

185
```python { .api }
186
class PickledPyiLoader:
187
    """
188
    Loader for pickled .pyi files.
189
    
190
    Provides optimized loading of pre-processed type stub files,
191
    significantly improving analysis performance for large projects.
192
    """
193
    
194
    def __init__(self, python_version, module_utils):
195
        """
196
        Initialize pickled .pyi loader.
197
        
198
        Parameters:
199
        - python_version (tuple): Target Python version
200
        - module_utils: Module resolution utilities
201
        """
202
    
203
    def load_from_pickle(self, filename):
204
        """
205
        Load PyTD AST from pickled file.
206
        
207
        Parameters:
208
        - filename (str): Path to pickled .pyi file
209
        
210
        Returns:
211
        TypeDeclUnit: Loaded PyTD AST
212
        
213
        Raises:
214
        IOError: If file cannot be read or unpickled
215
        """
216
    
217
    def save_to_pickle(self, ast, filename):
218
        """
219
        Save PyTD AST to pickled file.
220
        
221
        Parameters:
222
        - ast (TypeDeclUnit): PyTD AST to pickle
223
        - filename (str): Output pickle file path
224
        """
225
```
226

227
Example usage:
228

229
```python
230
from pytype import load_pytd, config
231
from pytype.pytd import pytd_utils
232

233
# Create pickled loader
234
options = config.Options.create()
235
loader = load_pytd.PickledPyiLoader(options.python_version, None)
236

237
# Save AST to pickle for faster loading
238
ast = # ... some PyTD AST
239
loader.save_to_pickle(ast, "mymodule.pkl")
240

241
# Load from pickle
242
cached_ast = loader.load_from_pickle("mymodule.pkl")
243
```
244

245
### Import Resolution
246

247
Advanced import resolution with support for custom mappings and complex project structures.
248

249
```python
250
from pytype import load_pytd, config
251

252
# Configure custom import mappings
253
options = config.Options.create()
254
options.imports_map = "imports.txt"  # Custom import mappings
255

256
# imports.txt content:
257
# internal.utils myproject.internal.utils
258
# external_lib /path/to/external/stubs
259

260
loader = load_pytd.create_loader(options)
261

262
# Import resolution uses mappings automatically
263
try:
264
    # This will resolve using the mapping
265
    utils_module = loader.import_name("internal.utils")
266
    
267
    # This will use the external stub path
268
    external_module = loader.import_name("external_lib")
269
    
270
except ImportError as e:
271
    print(f"Import resolution failed: {e}")
272
```
273

274
### Caching and Performance
275

276
The loader system includes comprehensive caching for improved performance:
277

278
```python
279
from pytype import load_pytd, config
280

281
options = config.Options.create()
282
loader = load_pytd.create_loader(options)
283

284
# First load - from source
285
module1 = loader.import_name("collections")  # Loads from source
286

287
# Second load - from cache
288
module2 = loader.import_name("collections")  # Uses cached version
289

290
assert module1 is module2  # Same object from cache
291
```
292

293
### Error Classes
294

295
```python { .api }
296
class BadDependencyError(Exception):
297
    """Error loading module dependencies."""
298

299
class ImportError(Exception):  
300
    """Module import resolution failure."""
301
```
302

303
### Error Handling
304

305
Comprehensive error handling for module loading failures:
306

307
```python
308
from pytype import load_pytd, config
309

310
options = config.Options.create()
311
loader = load_pytd.create_loader(options)
312

313
try:
314
    # Attempt to load potentially missing module
315
    module = loader.import_name("nonexistent.module")
316
    
317
except ImportError as e:
318
    print(f"Module not found: {e}")
319
    
320
    # Check if module is available before loading
321
    if loader.has_module("alternative.module"):
322
        alt_module = loader.import_name("alternative.module")
323
    else:
324
        print("No alternative module available")
325

326
except Exception as e:
327
    print(f"Unexpected loading error: {e}")
328
```
329

330
### Integration with Analysis
331

332
The module loading system integrates seamlessly with PyType's analysis workflow:
333

334
```python
335
from pytype import io, config, load_pytd
336

337
# Create configured loader
338
options = config.Options.create()
339
options.pythonpath = ["/my/project"]
340
loader = load_pytd.create_loader(options)
341

342
# Use loader in analysis
343
source_code = '''
344
import json
345
import myproject.utils
346

347
def process_data(data):
348
    config = json.loads(data)
349
    return myproject.utils.transform(config)
350
'''
351

352
# Analysis uses loader for import resolution
353
result = io.check_py(source_code, options=options, loader=loader)
354
```
355

356
### Custom Module Resolution
357

358
For complex project structures, implement custom module resolution:
359

360
```python
361
from pytype import load_pytd, config
362

363
class CustomLoader(load_pytd.Loader):
364
    """Custom loader with project-specific resolution logic."""
365
    
366
    def import_name(self, module_name):
367
        # Custom resolution logic
368
        if module_name.startswith("myproject."):
369
            # Handle internal modules specially
370
            internal_path = self._resolve_internal_module(module_name)
371
            return self.load_file(module_name, internal_path)
372
        
373
        # Fall back to standard resolution
374
        return super().import_name(module_name)
375
    
376
    def _resolve_internal_module(self, module_name):
377
        # Project-specific module resolution
378
        parts = module_name.split(".")
379
        return f"/project/src/{'/'.join(parts[1:])}.py"
380

381
# Use custom loader
382
options = config.Options.create()
383
loader = CustomLoader(options.python_version, options.python_exe, None)
384
```
385

386
### Bulk Module Loading
387

388
For analyzing projects with many dependencies:
389

390
```python
391
from pytype import load_pytd, config
392

393
def preload_common_modules(loader, modules):
394
    """Preload commonly used modules for better performance."""
395
    
396
    loaded = {}
397
    for module_name in modules:
398
        try:
399
            loaded[module_name] = loader.import_name(module_name)
400
            print(f"Preloaded {module_name}")
401
        except ImportError:
402
            print(f"Skipped {module_name} (not available)")
403
    
404
    return loaded
405

406
# Preload standard library modules
407
options = config.Options.create()
408
loader = load_pytd.create_loader(options)
409

410
common_modules = [
411
    "os", "sys", "json", "typing", "collections",
412
    "itertools", "functools", "datetime"
413
]
414

415
preloaded = preload_common_modules(loader, common_modules)
416
```