or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

communication.mdcontext-config.mdindex.mdpools.mdprocess-management.mdshared-objects.mdsynchronization.md
tile.json

tessl/pypi-multiprocess

A fork of Python's multiprocessing module that extends multiprocessing to provide enhanced serialization using dill

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/multiprocess@0.70.x

To install, run

npx @tessl/cli install tessl/pypi-multiprocess@0.70.0

index.mddocs/

Multiprocess

A fork of Python's multiprocessing module that extends multiprocessing to provide enhanced serialization using dill. Multiprocess leverages multiprocessing to support the spawning of processes using the API of the Python standard library's threading module, with better object serialization capabilities for complex Python objects.

Package Information

  • Package Name: multiprocess
  • Language: Python
  • Installation: pip install multiprocess
  • Python Support: 3.9-3.14, PyPy 3.9-3.11
  • Dependencies: dill>=0.4.0, setuptools>=42

Core Imports

import multiprocess

Common imports for specific functionality:

from multiprocess import Process, Queue, Pool
from multiprocess import Lock, Semaphore, Event, Condition
from multiprocess import Manager, Pipe

For shared memory functionality:

from multiprocess.shared_memory import SharedMemory, ShareableList
from multiprocess.sharedctypes import copy, synchronized

Basic Usage

from multiprocess import Process, Queue

def worker(q):
    q.put('hello world')

if __name__ == '__main__':
    q = Queue()
    p = Process(target=worker, args=[q])
    p.start()
    print(q.get())  # Output: hello world
    p.join()

Architecture

Multiprocess maintains full API compatibility with Python's built-in multiprocessing module while adding enhanced serialization capabilities:

  • Process Management: Create and control worker processes using Process class
  • Synchronization: Thread-like synchronization primitives (Lock, Semaphore, Event, etc.)
  • Communication: Inter-process communication via Queues, Pipes, and shared objects
  • Pool Processing: Parallel task execution using worker process pools
  • Enhanced Serialization: Extended object serialization using dill for complex objects
  • Context Management: Support for different process start methods (fork, spawn, forkserver)

The enhanced serialization using dill allows multiprocess to handle complex Python objects that standard multiprocessing cannot serialize, including lambda functions, nested functions, and complex class instances.

Capabilities

Process Management

Core functionality for creating, managing, and controlling processes. Provides the Process class with threading-like API, along with utilities for process inspection and lifecycle management.

class Process:
    def __init__(self, target=None, name=None, args=(), kwargs={}, daemon=None): ...
    def start(self): ...
    def join(self, timeout=None): ...
    def terminate(self): ...
    def kill(self): ...
    def is_alive(self): ...

def current_process(): ...
def active_children(): ...
def parent_process(): ...
def cpu_count(): ...

Process Management

Synchronization Primitives

Thread-like synchronization objects for coordinating processes. Includes locks, semaphores, events, conditions, and barriers for process synchronization and mutual exclusion.

def Lock(): ...
def RLock(): ...
def Semaphore(value=1): ...
def BoundedSemaphore(value=1): ...
def Event(): ...
def Condition(lock=None): ...
def Barrier(parties, action=None, timeout=None): ...

Synchronization Primitives

Inter-Process Communication

Communication mechanisms for data exchange between processes. Provides queues for message passing and pipes for bidirectional communication with various queue types and connection objects.

class Queue:
    def __init__(self, maxsize=0): ...
    def put(self, item, block=True, timeout=None): ...
    def get(self, block=True, timeout=None): ...
    def empty(self): ...
    def full(self): ...

class JoinableQueue(Queue):
    def task_done(self): ...
    def join(self): ...

class SimpleQueue:
    def put(self, item): ...
    def get(self): ...
    def empty(self): ...

def Pipe(duplex=True): ...

Inter-Process Communication

Process Pools

Parallel task execution using worker process pools. Provides Pool class for distributing tasks across multiple processes with various execution patterns and async result handling.

class Pool:
    def __init__(self, processes=None, initializer=None, initargs=(), 
                 maxtasksperchild=None): ...
    def map(self, func, iterable, chunksize=None): ...
    def map_async(self, func, iterable, chunksize=None, callback=None, 
                  error_callback=None): ...
    def imap(self, func, iterable, chunksize=1): ...
    def imap_unordered(self, func, iterable, chunksize=1): ...
    def starmap(self, func, iterable, chunksize=None): ...
    def starmap_async(self, func, iterable, chunksize=None, callback=None, 
                      error_callback=None): ...
    def apply(self, func, args=(), kwds={}): ...
    def apply_async(self, func, args=(), kwds={}, callback=None, 
                    error_callback=None): ...
    def close(self): ...
    def terminate(self): ...
    def join(self): ...

Process Pools

Shared Objects and Memory

Objects and memory that can be shared between processes. Includes both high-level managed objects and low-level shared memory constructs for different sharing patterns and performance requirements.

def Manager(): ...

class SharedMemory:
    def __init__(self, name=None, create=False, size=0): ...
    def close(self): ...
    def unlink(self): ...

class ShareableList:
    def __init__(self, sequence=None, name=None): ...
    def copy(self): ...
    def count(self, value): ...
    def index(self, value): ...

def Value(typecode_or_type, *args, lock=True): ...
def Array(typecode_or_type, size_or_initializer, lock=True): ...
def RawValue(typecode_or_type, *args): ...
def RawArray(typecode_or_type, size_or_initializer): ...

Shared Objects and Memory

Context and Configuration

Context management and configuration options for different process start methods and serialization behavior. Includes logging utilities, debugging support, and advanced configuration for process creation and object serialization.

def get_context(method=None): ...
def set_start_method(method, force=False): ...
def get_start_method(allow_none=False): ...
def get_all_start_methods(): ...
def set_executable(executable): ...
def set_forkserver_preload(module_names): ...
def get_logger(): ...
def log_to_stderr(level=None): ...
def freeze_support(): ...
def allow_connection_pickling(): ...

Context and Configuration

Exception Types

class ProcessError(Exception): ...
class BufferTooShort(ProcessError): ...
class TimeoutError(ProcessError): ...
class AuthenticationError(ProcessError): ...

Constants

SUBDEBUG: int = 5
SUBWARNING: int = 25