tessl/pypi-smmap
A pure Python implementation of a sliding window memory map manager
—
Pending
memory-managers.mddocs/
Memory Managers
Core memory management classes that control resource allocation, handle system limits, and provide configurable memory mapping strategies. These managers automatically track mapped files, manage resource cleanup using LRU algorithms, and prevent system resource exhaustion.
Capabilities
SlidingWindowMapManager
Memory manager that maps portions of files using a sliding window approach, ideal for accessing large files on systems with limited address space or when working with multiple large files simultaneously.
class SlidingWindowMapManager:
def __init__(self, window_size=-1, max_memory_size=0, max_open_handles=sys.maxsize):
"""
Initialize sliding window memory manager.
Parameters:
- window_size (int): Maximum size of each memory window in bytes.
-1 = auto-sized based on architecture (64MB on 32-bit, 1024MB on 64-bit)
0 = unlimited (maps entire file)
- max_memory_size (int): Maximum total memory to map simultaneously in bytes.
0 = auto-sized based on architecture (1GB on 32-bit, 8GB on 64-bit)
- max_open_handles (int): Maximum number of open file handles.
Default is sys.maxsize (system limit)
"""
def make_cursor(self, path_or_fd):
"""
Create a cursor for accessing a file through memory mapping.
Parameters:
- path_or_fd (str | int): File path or open file descriptor
Returns:
WindowCursor: Cursor object for accessing the file
Note: File descriptors must remain valid for the duration of use.
File paths are preferred for automatic resource management.
"""
def collect(self):
"""
Force collection of unused mapped regions to free resources.
Returns:
int: Number of freed handles
"""
def num_file_handles(self):
"""
Get current number of open file handles.
Returns:
int: Count of currently open file handles
"""
def num_open_files(self):
"""
Get number of files currently opened by the manager.
Returns:
int: Count of opened files
"""
def window_size(self):
"""
Get configured window size for new mappings.
Returns:
int: Window size in bytes
"""
def mapped_memory_size(self):
"""
Get total amount of currently mapped memory.
Returns:
int: Total mapped memory in bytes
"""
def max_file_handles(self):
"""
Get maximum allowed file handles.
Returns:
int: Maximum handle limit
"""
def max_mapped_memory_size(self):
"""
Get maximum allowed mapped memory.
Returns:
int: Maximum memory limit in bytes
"""Usage Example
import smmap
# Create manager optimized for large file access
manager = smmap.SlidingWindowMapManager(
window_size=64*1024*1024, # 64MB windows
max_memory_size=1024*1024*1024, # 1GB total limit
max_open_handles=100 # Limit open handles
)
# Access multiple large files efficiently
cursor1 = manager.make_cursor('/path/to/large_file1.dat')
cursor2 = manager.make_cursor('/path/to/large_file2.dat')
# Windows are automatically managed as you access different regions
cursor1.use_region(offset=0, size=1024*1024)
cursor2.use_region(offset=1000000, size=512*1024)
print(f"Memory used: {manager.mapped_memory_size()} bytes")
print(f"Handles open: {manager.num_file_handles()}")
# Manually free unused regions if needed
freed = manager.collect()
print(f"Freed {freed} handles")StaticWindowMapManager
Memory manager that maps entire files at once, optimized for 64-bit systems and scenarios where you need access to complete files with maximum performance.
class StaticWindowMapManager:
def __init__(self, window_size=0, max_memory_size=0, max_open_handles=sys.maxsize):
"""
Initialize static window memory manager.
Parameters:
- window_size (int): Window size (0 = map entire file, ignored for static manager)
- max_memory_size (int): Maximum total memory to map simultaneously.
0 = auto-sized based on architecture
- max_open_handles (int): Maximum number of open file handles
"""
def force_map_handle_removal_win(self, base_path):
"""
Windows-specific: Force removal of memory mappings for files under base_path.
Parameters:
- base_path (str): Base path for files to unmap
Returns:
int: Number of closed handles
Note: Only available on Windows. Required for file deletion when files
are memory-mapped. Use with caution - ensures no cursors access the
unmapped regions after calling this method.
"""All other methods are inherited from SlidingWindowMapManager.
Usage Example
import smmap
# Create manager for mapping entire files
manager = smmap.StaticWindowMapManager(
max_memory_size=4*1024*1024*1024, # 4GB limit
max_open_handles=50
)
# Each cursor maps the entire file
cursor = manager.make_cursor('/path/to/file.dat')
cursor.use_region() # Maps entire file
# Direct access to any part of the file
buffer = cursor.buffer()
data_at_end = buffer[-1000:] # Last 1000 bytes
random_access = buffer[1000000:1001000] # 1000 bytes at offset 1MB
# Windows-specific: Force close mappings to allow file deletion
if sys.platform == 'win32':
closed = manager.force_map_handle_removal_win('/temp/processing/')
print(f"Closed {closed} handles for cleanup")Resource Management
Both managers implement automatic resource management:
- LRU Collection: Least recently used regions are automatically unmapped when limits are exceeded
- Reference Counting: Regions are kept mapped as long as cursors reference them
- Handle Limits: File handle limits prevent system resource exhaustion
- Memory Limits: Configurable memory limits prevent excessive memory usage
- Cleanup: Automatic cleanup on cursor destruction and manual collection methods
Choosing Between Managers
Use SlidingWindowMapManager when:
- Working with files larger than available memory
- Accessing multiple large files simultaneously
- Running on 32-bit systems
- Memory usage needs to be strictly controlled
Use StaticWindowMapManager when:
- Working on 64-bit systems with sufficient memory
- Files fit comfortably in available address space
- Maximum performance is required (no windowing overhead)
- Random access patterns across entire files are common
Install with Tessl CLI
npx tessl i tessl/pypi-smmap