tessl/pypi-multitasking

Non-blocking Python methods using decorators

—

Pending

Overview

Eval results

Files

Configuration

Name: tessl/pypi-multitasking
Author: tessl

Global configuration functions that affect all pools and tasks created after the configuration change. These settings provide high-level control over multitasking behavior.

Capabilities

Thread Limit Configuration

Set the global maximum number of concurrent threads or processes.

def set_max_threads(threads: Optional[int] = None) -> None:
    """
    Configure the maximum number of concurrent threads/processes.
    
    This function allows users to override the default CPU-based thread
    count. Setting this affects new pools but not existing ones.
    
    Args:
        threads: Maximum concurrent tasks. If None, uses CPU count.
                 Must be positive integer or None.
    """

Usage Example:

import multitasking

# Check current configuration
print(f"CPU cores detected: {multitasking.config['CPU_CORES']}")
print(f"Current max threads: {multitasking.config['MAX_THREADS']}")

# Set custom thread limit
multitasking.set_max_threads(10)
print(f"New max threads: {multitasking.config['MAX_THREADS']}")

# Reset to CPU count
multitasking.set_max_threads()
print(f"Reset to: {multitasking.config['MAX_THREADS']}")

# Scale based on CPU cores for I/O-bound tasks
multitasking.set_max_threads(multitasking.config["CPU_CORES"] * 5)
print(f"I/O optimized: {multitasking.config['MAX_THREADS']} threads")

# Unlimited threading (use carefully!)
multitasking.set_max_threads(0)

Engine Selection

Configure whether new pools use threading or multiprocessing by default.

def set_engine(kind: str = "") -> None:
    """
    Configure the execution engine for new pools.
    
    This determines whether new tasks run in threads or separate
    processes. Threads share memory but processes are more isolated.
    Only affects pools created after this call.
    
    Args:
        kind: Engine type. Contains "process" for multiprocessing,
              anything else defaults to threading.
    
    Note:
        Threading: Faster startup, shared memory, GIL limitations
        Processing: Slower startup, isolated memory, true parallelism
    """

Usage Example:

import multitasking

# Default is threading
print(f"Default engine: {multitasking.config['ENGINE']}")

# For I/O-bound tasks (default, recommended for most cases)
multitasking.set_engine("thread")
multitasking.createPool("io_pool", threads=20)

@multitasking.task
def fetch_data(url):
    # I/O-bound task - threading is efficient
    pass

# For CPU-bound tasks (avoids GIL limitations)
multitasking.set_engine("process")
multitasking.createPool("cpu_pool", threads=4)

@multitasking.task
def compute_heavy(data):
    # CPU-bound task - multiprocessing avoids GIL
    return sum(x*x for x in data)

# Check current configuration
print(f"Current engine: {multitasking.config['ENGINE']}")

Configuration Guidelines

Choosing Thread Counts

Different workload types benefit from different thread configurations:

import multitasking

# I/O-bound tasks: High concurrency (network requests, file operations)
multitasking.set_max_threads(multitasking.config["CPU_CORES"] * 5)
multitasking.set_engine("thread")

# CPU-bound tasks: Match CPU cores (mathematical computations)
multitasking.set_max_threads(multitasking.config["CPU_CORES"])
multitasking.set_engine("process")

# Mixed workloads: Conservative approach
multitasking.set_max_threads(multitasking.config["CPU_CORES"] * 2)
multitasking.set_engine("thread")

Engine Selection Guidelines

Choose the appropriate execution engine based on task characteristics:

import multitasking

# Threading: Best for I/O-bound tasks
multitasking.set_engine("thread")
# - File operations
# - Network requests  
# - Database queries
# - Web scraping
# - API calls

# Multiprocessing: Best for CPU-intensive tasks  
multitasking.set_engine("process")
# - Mathematical computations
# - Image processing
# - Data analysis
# - Cryptographic operations
# - Scientific computing

Global Configuration Access

The global configuration dictionary provides access to all settings:

import multitasking

# Access configuration values
config = multitasking.config
print(f"CPU cores: {config['CPU_CORES']}")
print(f"Max threads: {config['MAX_THREADS']}")
print(f"Current engine: {config['ENGINE']}")
print(f"Active pool: {config['POOL_NAME']}")
print(f"Shutdown mode: {config['KILL_RECEIVED']}")

# View all pools
print(f"Available pools: {list(config['POOLS'].keys())}")

# View all tasks
print(f"Total tasks created: {len(config['TASKS'])}")

Install with Tessl CLI