0
# Memory Managers
1

2
Core memory management classes that control resource allocation, handle system limits, and provide configurable memory mapping strategies. These managers automatically track mapped files, manage resource cleanup using LRU algorithms, and prevent system resource exhaustion.
3

4
## Capabilities
5

6
### SlidingWindowMapManager
7

8
Memory manager that maps portions of files using a sliding window approach, ideal for accessing large files on systems with limited address space or when working with multiple large files simultaneously.
9

10
```python { .api }
11
class SlidingWindowMapManager:
12
    def __init__(self, window_size=-1, max_memory_size=0, max_open_handles=sys.maxsize):
13
        """
14
        Initialize sliding window memory manager.
15
        
16
        Parameters:
17
        - window_size (int): Maximum size of each memory window in bytes.
18
          -1 = auto-sized based on architecture (64MB on 32-bit, 1024MB on 64-bit)
19
          0 = unlimited (maps entire file)
20
        - max_memory_size (int): Maximum total memory to map simultaneously in bytes.
21
          0 = auto-sized based on architecture (1GB on 32-bit, 8GB on 64-bit)
22
        - max_open_handles (int): Maximum number of open file handles.
23
          Default is sys.maxsize (system limit)
24
        """
25
    
26
    def make_cursor(self, path_or_fd):
27
        """
28
        Create a cursor for accessing a file through memory mapping.
29
        
30
        Parameters:
31
        - path_or_fd (str | int): File path or open file descriptor
32
        
33
        Returns:
34
        WindowCursor: Cursor object for accessing the file
35
        
36
        Note: File descriptors must remain valid for the duration of use.
37
        File paths are preferred for automatic resource management.
38
        """
39
    
40
    def collect(self):
41
        """
42
        Force collection of unused mapped regions to free resources.
43
        
44
        Returns:
45
        int: Number of freed handles
46
        """
47
    
48
    def num_file_handles(self):
49
        """
50
        Get current number of open file handles.
51
        
52
        Returns:
53
        int: Count of currently open file handles
54
        """
55
    
56
    def num_open_files(self):
57
        """
58
        Get number of files currently opened by the manager.
59
        
60
        Returns:
61
        int: Count of opened files
62
        """
63
    
64
    def window_size(self):
65
        """
66
        Get configured window size for new mappings.
67
        
68
        Returns:
69
        int: Window size in bytes
70
        """
71
    
72
    def mapped_memory_size(self):
73
        """
74
        Get total amount of currently mapped memory.
75
        
76
        Returns:
77
        int: Total mapped memory in bytes
78
        """
79
    
80
    def max_file_handles(self):
81
        """
82
        Get maximum allowed file handles.
83
        
84
        Returns:
85
        int: Maximum handle limit
86
        """
87
    
88
    def max_mapped_memory_size(self):
89
        """
90
        Get maximum allowed mapped memory.
91
        
92
        Returns:
93
        int: Maximum memory limit in bytes
94
        """
95
```
96

97
#### Usage Example
98

99
```python
100
import smmap
101

102
# Create manager optimized for large file access
103
manager = smmap.SlidingWindowMapManager(
104
    window_size=64*1024*1024,  # 64MB windows
105
    max_memory_size=1024*1024*1024,  # 1GB total limit
106
    max_open_handles=100  # Limit open handles
107
)
108

109
# Access multiple large files efficiently
110
cursor1 = manager.make_cursor('/path/to/large_file1.dat')
111
cursor2 = manager.make_cursor('/path/to/large_file2.dat') 
112

113
# Windows are automatically managed as you access different regions
114
cursor1.use_region(offset=0, size=1024*1024)
115
cursor2.use_region(offset=1000000, size=512*1024)
116

117
print(f"Memory used: {manager.mapped_memory_size()} bytes")
118
print(f"Handles open: {manager.num_file_handles()}")
119

120
# Manually free unused regions if needed
121
freed = manager.collect()
122
print(f"Freed {freed} handles")
123
```
124

125
### StaticWindowMapManager
126

127
Memory manager that maps entire files at once, optimized for 64-bit systems and scenarios where you need access to complete files with maximum performance.
128

129
```python { .api }
130
class StaticWindowMapManager:
131
    def __init__(self, window_size=0, max_memory_size=0, max_open_handles=sys.maxsize):
132
        """
133
        Initialize static window memory manager.
134
        
135
        Parameters:
136
        - window_size (int): Window size (0 = map entire file, ignored for static manager)
137
        - max_memory_size (int): Maximum total memory to map simultaneously.
138
          0 = auto-sized based on architecture
139
        - max_open_handles (int): Maximum number of open file handles
140
        """
141
    
142
    def force_map_handle_removal_win(self, base_path):
143
        """
144
        Windows-specific: Force removal of memory mappings for files under base_path.
145
        
146
        Parameters:
147
        - base_path (str): Base path for files to unmap
148
        
149
        Returns:
150
        int: Number of closed handles
151
        
152
        Note: Only available on Windows. Required for file deletion when files
153
        are memory-mapped. Use with caution - ensures no cursors access the
154
        unmapped regions after calling this method.
155
        """
156
```
157

158
All other methods are inherited from SlidingWindowMapManager.
159

160
#### Usage Example
161

162
```python
163
import smmap
164

165
# Create manager for mapping entire files
166
manager = smmap.StaticWindowMapManager(
167
    max_memory_size=4*1024*1024*1024,  # 4GB limit
168
    max_open_handles=50
169
)
170

171
# Each cursor maps the entire file
172
cursor = manager.make_cursor('/path/to/file.dat')
173
cursor.use_region()  # Maps entire file
174

175
# Direct access to any part of the file
176
buffer = cursor.buffer()
177
data_at_end = buffer[-1000:]  # Last 1000 bytes
178
random_access = buffer[1000000:1001000]  # 1000 bytes at offset 1MB
179

180
# Windows-specific: Force close mappings to allow file deletion
181
if sys.platform == 'win32':
182
    closed = manager.force_map_handle_removal_win('/temp/processing/')
183
    print(f"Closed {closed} handles for cleanup")
184
```
185

186
## Resource Management
187

188
Both managers implement automatic resource management:
189

190
- **LRU Collection**: Least recently used regions are automatically unmapped when limits are exceeded
191
- **Reference Counting**: Regions are kept mapped as long as cursors reference them
192
- **Handle Limits**: File handle limits prevent system resource exhaustion
193
- **Memory Limits**: Configurable memory limits prevent excessive memory usage
194
- **Cleanup**: Automatic cleanup on cursor destruction and manual collection methods
195

196
## Choosing Between Managers
197

198
**Use SlidingWindowMapManager when:**
199
- Working with files larger than available memory
200
- Accessing multiple large files simultaneously
201
- Running on 32-bit systems
202
- Memory usage needs to be strictly controlled
203

204
**Use StaticWindowMapManager when:**
205
- Working on 64-bit systems with sufficient memory
206
- Files fit comfortably in available address space
207
- Maximum performance is required (no windowing overhead)
208
- Random access patterns across entire files are common