0
# ImportMagic
1
2
A Python library for automated import management and symbol resolution. ImportMagic builds comprehensive symbol indexes from installed packages, analyzes Python source code to identify unresolved symbol references, and automatically generates appropriate import statements to resolve them.
3
4
## Package Information
5
6
- **Package Name**: importmagic
7
- **Language**: Python
8
- **Installation**: `pip install importmagic`
9
10
## Core Imports
11
12
```python
13
import importmagic
14
```
15
16
Specific functionality imports:
17
18
```python
19
from importmagic import SymbolIndex, Scope, Imports, get_update, update_imports
20
```
21
22
## Basic Usage
23
24
```python
25
import importmagic
26
import sys
27
28
# Build an index of available symbols
29
index = importmagic.SymbolIndex()
30
index.build_index(sys.path)
31
32
# Analyze source code for symbol usage
33
python_source = '''
34
import os
35
def example():
36
basename = path.basename("/tmp/file.txt") # unresolved: path
37
return basename
38
'''
39
40
scope = importmagic.Scope.from_source(python_source)
41
unresolved, unreferenced = scope.find_unresolved_and_unreferenced_symbols()
42
43
# Get updated source with resolved imports
44
new_source = importmagic.update_imports(python_source, index, unresolved, unreferenced)
45
print(new_source)
46
# Output includes: from os import path
47
```
48
49
## Architecture
50
51
ImportMagic follows a three-phase approach to automated import management:
52
53
### 1. Symbol Indexing (`SymbolIndex`)
54
- **Purpose**: Build comprehensive catalogs of available symbols from Python packages
55
- **Process**: Recursively scans Python paths, parsing modules to extract exportable symbols
56
- **Scoring**: Assigns relevance scores based on module location (local > third-party > system > future)
57
- **Storage**: Hierarchical tree structure enabling fast symbol lookup and JSON serialization
58
59
### 2. Source Analysis (`Scope`)
60
- **Purpose**: Analyze Python source code to identify symbol usage patterns
61
- **Process**: Uses AST (Abstract Syntax Tree) visitor pattern to traverse code structure
62
- **Tracking**: Maintains separate sets of defined symbols and referenced symbols across nested scopes
63
- **Resolution**: Identifies unresolved references (missing imports) and unreferenced imports (unused)
64
65
### 3. Import Management (`Imports`)
66
- **Purpose**: Generate and format import statements according to PEP8 conventions
67
- **Process**: Tokenizes existing imports, applies additions/removals, regenerates import block
68
- **Formatting**: Supports multiple import styles (parentheses, backslash) with configurable line lengths
69
- **Integration**: Seamlessly updates source code while preserving non-import content
70
71
### Component Relationships
72
73
```
74
SymbolIndex ──┐
75
├─→ Imports ──→ Updated Source Code
76
Source Code ──┘ ↗
77
│ │
78
↓ │
79
Scope ──→ Analysis Results ─┘
80
```
81
82
The workflow typically involves:
83
1. Building a `SymbolIndex` from available Python paths
84
2. Creating a `Scope` from source code to identify symbol usage
85
3. Using `Imports` to resolve unresolved symbols and clean up unreferenced imports
86
4. Generating updated source code with properly formatted import statements
87
88
This architecture enables both high-level convenience functions (`update_imports`) and fine-grained control over each phase for advanced use cases.
89
90
## Capabilities
91
92
### Symbol Indexing
93
94
Build searchable indexes of Python symbols from modules and packages for import resolution.
95
96
```python { .api }
97
class SymbolIndex:
98
def __init__(self, name=None, parent=None, score=1.0, location='L', blacklist_re=None, locations=None):
99
"""
100
Create a new symbol index.
101
102
Parameters:
103
- name: Optional name for this index node
104
- parent: Parent SymbolIndex node
105
- score: Score multiplier for symbols in this index
106
- location: Location classification ('F', '3', 'S', 'L')
107
- blacklist_re: Regex pattern for modules to exclude
108
- locations: Library location mappings
109
"""
110
111
def build_index(self, paths):
112
"""
113
Build comprehensive symbol index from list of paths.
114
115
Parameters:
116
- paths: List of directory paths to index (typically sys.path)
117
"""
118
119
def symbol_scores(self, symbol):
120
"""
121
Find scored matches for a symbol name.
122
123
Parameters:
124
- symbol: Dotted symbol name to search for
125
126
Returns:
127
List of tuples (score, module, variable) ordered by score
128
"""
129
130
def index_path(self, root):
131
"""
132
Index a single path (file or directory).
133
134
Parameters:
135
- root: File path or package directory to index
136
"""
137
138
def serialize(self, fd=None):
139
"""
140
Serialize index to JSON format.
141
142
Parameters:
143
- fd: Optional file descriptor to write to
144
145
Returns:
146
JSON string if fd is None, otherwise writes to fd
147
"""
148
149
@classmethod
150
def deserialize(cls, file):
151
"""
152
Load index from JSON file.
153
154
Parameters:
155
- file: File object to read from
156
157
Returns:
158
SymbolIndex instance
159
"""
160
161
def index_source(self, filename, source):
162
"""
163
Index symbols from Python source code.
164
165
Parameters:
166
- filename: Name or path of the source file
167
- source: Python source code string
168
"""
169
170
def index_file(self, module, filename):
171
"""
172
Index symbols from a Python file.
173
174
Parameters:
175
- module: Module name for the file
176
- filename: Path to the Python file
177
"""
178
179
def index_builtin(self, name, location):
180
"""
181
Index symbols from a built-in module.
182
183
Parameters:
184
- name: Built-in module name
185
- location: Location classification for the module
186
"""
187
188
def find(self, path):
189
"""
190
Find index node for a dotted path.
191
192
Parameters:
193
- path: Dotted symbol path to find
194
195
Returns:
196
SymbolIndex node or None if not found
197
"""
198
199
def location_for(self, path):
200
"""
201
Get location classification for a symbol path.
202
203
Parameters:
204
- path: Dotted symbol path
205
206
Returns:
207
Location code string ('F', '3', 'S', 'L')
208
"""
209
210
def add(self, name, score):
211
"""
212
Add a symbol with score to current index node.
213
214
Parameters:
215
- name: Symbol name to add
216
- score: Score value for the symbol
217
"""
218
219
def add_explicit_export(self, name, score):
220
"""
221
Add an explicitly exported symbol with score.
222
223
Parameters:
224
- name: Symbol name to export
225
- score: Score value for the exported symbol
226
"""
227
228
def enter(self, name, location='L', score=1.0):
229
"""
230
Enter a sub-namespace context manager.
231
232
Parameters:
233
- name: Namespace name
234
- location: Location classification (default 'L')
235
- score: Score multiplier (default 1.0)
236
237
Returns:
238
Context manager yielding child SymbolIndex
239
"""
240
241
def depth(self):
242
"""
243
Get the depth of this index node in the tree.
244
245
Returns:
246
Integer depth from root (0 for root)
247
"""
248
249
def path(self):
250
"""
251
Get dotted path from root to this index node.
252
253
Returns:
254
Dotted string path
255
"""
256
257
def boost(self):
258
"""
259
Get boost multiplier for this location type.
260
261
Returns:
262
Float boost multiplier
263
"""
264
```
265
266
### Source Code Analysis
267
268
Analyze Python source code to identify symbol definitions, references, and scope relationships.
269
270
```python { .api }
271
class Scope:
272
def __init__(self, parent=None, define_builtins=True, is_class=False):
273
"""
274
Create a new scope for symbol tracking.
275
276
Parameters:
277
- parent: Parent scope (None for root scope)
278
- define_builtins: Whether to include Python builtins
279
- is_class: Whether this scope represents a class
280
"""
281
282
@classmethod
283
def from_source(cls, src, trace=False, define_builtins=True):
284
"""
285
Create scope by analyzing Python source code.
286
287
Parameters:
288
- src: Python source code string or AST node
289
- trace: Enable tracing for debugging
290
- define_builtins: Whether to include Python builtins
291
292
Returns:
293
Scope instance with analyzed symbol information
294
"""
295
296
def find_unresolved_and_unreferenced_symbols(self):
297
"""
298
Find symbols that need import resolution or removal.
299
300
Returns:
301
Tuple of (unresolved_set, unreferenced_set)
302
"""
303
304
def define(self, name):
305
"""
306
Mark a symbol as defined in current scope.
307
308
Parameters:
309
- name: Symbol name to define
310
"""
311
312
def reference(self, name):
313
"""
314
Mark a symbol as referenced.
315
316
Parameters:
317
- name: Symbol name that was referenced
318
"""
319
320
def enter(self, is_class=False):
321
"""
322
Enter a child scope context manager.
323
324
Parameters:
325
- is_class: Whether the child scope represents a class
326
327
Returns:
328
Context manager yielding child Scope
329
"""
330
331
def start_symbol(self):
332
"""
333
Start building a compound symbol context manager.
334
335
Returns:
336
Context manager for compound symbol construction
337
"""
338
339
def start_definition(self):
340
"""
341
Start a symbol definition context manager.
342
343
Returns:
344
Context manager for symbol definition
345
"""
346
347
def start_reference(self):
348
"""
349
Start a symbol reference context manager.
350
351
Returns:
352
Context manager for symbol reference
353
"""
354
355
def extend_symbol(self, segment, extend_only=False):
356
"""
357
Extend current compound symbol with a segment.
358
359
Parameters:
360
- segment: Symbol segment to add
361
- extend_only: If True, only extend without creating reference
362
"""
363
364
def end_symbol(self):
365
"""
366
End current compound symbol and mark as referenced.
367
"""
368
369
def flush_symbol(self):
370
"""
371
Flush any pending symbol operations.
372
"""
373
```
374
375
### Import Management
376
377
Manage import statements with parsing, modification, and PEP8-compliant formatting.
378
379
```python { .api }
380
class Imports:
381
def __init__(self, index, source):
382
"""
383
Create import manager for source code.
384
385
Parameters:
386
- index: SymbolIndex for symbol resolution
387
- source: Python source code string
388
"""
389
390
def add_import(self, name, alias=None):
391
"""
392
Add a direct module import.
393
394
Parameters:
395
- name: Module name to import
396
- alias: Optional import alias
397
"""
398
399
def add_import_from(self, module, name, alias=None):
400
"""
401
Add a from-module import.
402
403
Parameters:
404
- module: Module to import from
405
- name: Symbol name to import
406
- alias: Optional import alias
407
"""
408
409
def remove(self, references):
410
"""
411
Remove imports by referenced names.
412
413
Parameters:
414
- references: Set of symbol names to remove
415
"""
416
417
def get_update(self):
418
"""
419
Get import block update information.
420
421
Returns:
422
Tuple of (start_line, end_line, import_text)
423
"""
424
425
def update_source(self):
426
"""
427
Return complete updated source code.
428
429
Returns:
430
Updated Python source code string
431
"""
432
433
@classmethod
434
def set_style(cls, **kwargs):
435
"""
436
Configure import formatting style.
437
438
Parameters:
439
- multiline: Multiline style ('parentheses' or 'backslash')
440
- max_columns: Maximum line length for imports
441
"""
442
443
class Import:
444
def __init__(self, location, name, alias):
445
"""
446
Represent a single import statement.
447
448
Parameters:
449
- location: Import location classification
450
- name: Module or symbol name
451
- alias: Import alias or None
452
"""
453
454
def __hash__(self):
455
"""
456
Hash function for Import objects.
457
458
Returns:
459
Hash value based on location, name, and alias
460
"""
461
462
def __eq__(self, other):
463
"""
464
Equality comparison for Import objects.
465
466
Parameters:
467
- other: Other Import object to compare
468
469
Returns:
470
True if imports are equal
471
"""
472
473
def __lt__(self, other):
474
"""
475
Less-than comparison for sorting Import objects.
476
477
Parameters:
478
- other: Other Import object to compare
479
480
Returns:
481
True if this import should sort before other
482
"""
483
```
484
485
### High-Level Import Functions
486
487
Convenient functions for automated import management without manual class instantiation.
488
489
```python { .api }
490
def get_update(src, index, unresolved, unreferenced):
491
"""
492
Generate import block update for source code.
493
494
Parameters:
495
- src: Python source code string
496
- index: SymbolIndex instance for symbol resolution
497
- unresolved: Set of unresolved symbol names
498
- unreferenced: Set of unreferenced import names
499
500
Returns:
501
Tuple of (start_line, end_line, import_block_text)
502
"""
503
504
def update_imports(src, index, unresolved, unreferenced):
505
"""
506
Update source code with resolved imports.
507
508
Parameters:
509
- src: Python source code string
510
- index: SymbolIndex instance for symbol resolution
511
- unresolved: Set of unresolved symbol names
512
- unreferenced: Set of unreferenced import names
513
514
Returns:
515
Updated Python source code string
516
"""
517
```
518
519
## Types
520
521
```python { .api }
522
# Location Classifications
523
LOCATION_ORDER = 'FS3L' # Future, System, Third-party, Local
524
525
# Symbol Index Locations
526
LOCATIONS = {
527
'F': 'Future', # __future__ imports
528
'3': 'Third party', # Third-party packages
529
'S': 'System', # Standard library
530
'L': 'Local' # Local modules
531
}
532
533
# Location Score Boosts
534
LOCATION_BOOSTS = {
535
'3': 1.2, # Third-party packages
536
'L': 1.5, # Local modules
537
}
538
539
# Package Aliases with Score Boosts
540
PACKAGE_ALIASES = {
541
'os.path': ('posixpath', 1.2), # Prefer os.path over posixpath/ntpath
542
'os': ('os', 1.2), # Boost os due to heavy aliasing
543
}
544
545
# Built-in Modules (treated specially during indexing)
546
BUILTIN_MODULES = sys.builtin_module_names + ('os',)
547
548
# Default module blacklist pattern
549
DEFAULT_BLACKLIST_RE = re.compile(r'\btest[s]?|test[s]?\b', re.I)
550
551
# Builtin Symbol Sets
552
GLOBALS = ['__name__', '__file__', '__loader__', '__package__', '__path__']
553
PYTHON3_BUILTINS = ['PermissionError']
554
ALL_BUILTINS = set(dir(__builtin__)) | set(GLOBALS) | set(PYTHON3_BUILTINS)
555
556
# Iterator class for token processing
557
class Iterator:
558
def __init__(self, tokens, start=None, end=None):
559
"""
560
Iterator for processing token sequences.
561
562
Parameters:
563
- tokens: List of tokens to iterate over
564
- start: Starting index (default 0)
565
- end: Ending index (default len(tokens))
566
"""
567
568
def next(self):
569
"""
570
Get next token and advance cursor.
571
572
Returns:
573
Tuple of (index, token) or (None, None) if exhausted
574
"""
575
576
def peek(self):
577
"""
578
Get current token without advancing cursor.
579
580
Returns:
581
Current token or None if exhausted
582
"""
583
584
def until(self, type):
585
"""
586
Collect tokens until specified type is found.
587
588
Parameters:
589
- type: Token type to search for
590
591
Returns:
592
List of (index, token) tuples
593
"""
594
595
def rewind(self):
596
"""
597
Move cursor back one position.
598
"""
599
```
600
601
## Advanced Usage Examples
602
603
### Fine-Grained Import Control
604
605
```python
606
import importmagic
607
import sys
608
609
# Build index
610
index = importmagic.SymbolIndex()
611
index.build_index(sys.path)
612
613
# Analyze source
614
python_source = '''
615
def example():
616
result = basename("/tmp/file.txt")
617
data = json.loads('{"key": "value"}')
618
return result, data
619
'''
620
621
scope = importmagic.Scope.from_source(python_source)
622
unresolved, unreferenced = scope.find_unresolved_and_unreferenced_symbols()
623
624
# Manual import management
625
imports = importmagic.Imports(index, python_source)
626
imports.remove(unreferenced)
627
628
# Add specific imports with control over selection
629
for symbol in unresolved:
630
scores = index.symbol_scores(symbol)
631
if scores:
632
score, module, variable = scores[0] # Take highest scored match
633
if variable is None:
634
imports.add_import(module) # Direct module import
635
else:
636
imports.add_import_from(module, variable) # From-import
637
638
updated_source = imports.update_source()
639
```
640
641
### Custom Import Formatting
642
643
```python
644
# Configure import style
645
importmagic.Imports.set_style(
646
multiline='parentheses', # Use parentheses for multiline imports
647
max_columns=80 # Maximum line length
648
)
649
650
# Process source with custom formatting
651
imports = importmagic.Imports(index, source)
652
# ... add/remove imports
653
formatted_source = imports.update_source()
654
```
655
656
### Index Persistence
657
658
```python
659
# Save index to file
660
index = importmagic.SymbolIndex()
661
index.build_index(sys.path)
662
with open('symbol_index.json', 'w') as f:
663
index.serialize(f)
664
665
# Load index from file
666
with open('symbol_index.json', 'r') as f:
667
index = importmagic.SymbolIndex.deserialize(f)
668
```
669
670
## Error Handling
671
672
ImportMagic handles various error conditions gracefully:
673
674
- **Malformed source code**: AST parsing errors are caught and logged
675
- **Missing modules**: Unavailable imports are skipped during indexing
676
- **Circular imports**: Detected and avoided during symbol resolution
677
- **Invalid symbols**: Malformed symbol names are filtered out
678
679
The library is designed to be robust for use in development tools and IDEs where source code may be incomplete or contain syntax errors.