or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.mdmemory-managers.mdutilities.mdwindow-access.md
tile.json

tessl/pypi-smmap

A pure Python implementation of a sliding window memory map manager

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/smmap@6.0.x

To install, run

npx @tessl/cli install tessl/pypi-smmap@6.0.0

index.mddocs/

smmap

A pure Python implementation of a sliding window memory map manager that efficiently handles memory-mapped file access for large files. It automatically manages system resources by tracking mapped files and client usage, unloading unused maps when resources become scarce or memory limits are reached.

Package Information

  • Package Name: smmap
  • Language: Python
  • Installation: pip install smmap
  • Python Requirements: >=3.8
  • License: BSD

Core Imports

import smmap

Import specific components:

from smmap import SlidingWindowMapManager, StaticWindowMapManager
from smmap import WindowCursor, SlidingWindowMapBuffer
from smmap import align_to_mmap, is_64_bit, ALLOCATIONGRANULARITY

Basic Usage

import smmap

# Create a memory manager (recommended for most uses)
manager = smmap.SlidingWindowMapManager()

# Create a cursor to access a file
cursor = manager.make_cursor('/path/to/large/file.dat')

# Map a region of the file into memory
cursor.use_region(offset=0, size=1024*1024)  # Map 1MB starting at offset 0

if cursor.is_valid():
    # Access the mapped memory as a buffer
    buffer = cursor.buffer()
    first_byte = buffer[0]
    first_chunk = buffer[:100]
    
    # Clean up when done
    cursor.unuse_region()

# Alternative: Use the buffer interface for easier access
with smmap.SlidingWindowMapBuffer(manager.make_cursor('/path/to/file.dat')) as buf:
    # Direct byte-level access with automatic windowing
    first_byte = buf[0]
    last_10_bytes = buf[-10:]
    chunk = buf[1000:2000]

Architecture

smmap uses a sliding window approach to efficiently manage memory-mapped file access:

  • Memory Managers: Control memory allocation, handle limits, and manage resource cleanup using LRU algorithms
  • Window Cursors: Provide low-level access to mapped memory regions with precise control over mapping and unmapping
  • Buffers: Offer a high-level string-like interface that automatically handles windowing behind the scenes
  • Utility Classes: Support memory alignment, region tracking, and cross-platform compatibility

The library automatically handles resource management, preventing file descriptor leaks and managing memory usage within configurable limits. On 32-bit systems, it maps only portions of large files as needed, while 64-bit systems can optionally map entire files for maximum performance.

Capabilities

Memory Managers

Core memory management classes that control resource allocation, handle system limits, and provide configurable memory mapping strategies for different system architectures and use cases.

class SlidingWindowMapManager:
    def __init__(self, window_size=-1, max_memory_size=0, max_open_handles=sys.maxsize): ...
    def make_cursor(self, path_or_fd): ...
    def collect(self): ...

class StaticWindowMapManager:
    def __init__(self, window_size=0, max_memory_size=0, max_open_handles=sys.maxsize): ...
    def make_cursor(self, path_or_fd): ...
    def collect(self): ...

Memory Managers

Window Access

Low-level cursor interface for precise memory region control and high-level buffer interface providing automatic windowing with string-like access patterns.

class WindowCursor:
    def use_region(self, offset=0, size=0, flags=0): ...
    def unuse_region(self): ...
    def buffer(self): ...
    def is_valid(self): ...

class SlidingWindowMapBuffer:
    def __init__(self, cursor=None, offset=0, size=sys.maxsize, flags=0): ...
    def begin_access(self, cursor=None, offset=0, size=sys.maxsize, flags=0): ...
    def end_access(self): ...

Window Access

Utilities

Memory alignment functions, system architecture detection, and utility classes for window management and memory region tracking.

def align_to_mmap(num, round_up): ...
def is_64_bit(): ...

class MapWindow:
    def __init__(self, offset, size): ...
    def align(self): ...

class MapRegion:
    def __init__(self, path_or_fd, ofs, size, flags=0): ...
    def buffer(self): ...

Utilities

Constants

ALLOCATIONGRANULARITY: int  # System memory allocation granularity (page size)

Package Metadata

__version__: str = "6.0.0"
version_info: tuple = (6, 0, 0)
__author__: str = "Sebastian Thiel"
__contact__: str = "byronimo@gmail.com"
__homepage__: str = "https://github.com/gitpython-developers/smmap"