or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.md
tile.json

tessl/pypi-msgpack-numpy

Numpy data serialization using msgpack format with type preservation

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/msgpack-numpy@0.4.x

To install, run

npx @tessl/cli install tessl/pypi-msgpack-numpy@0.4.0

index.mddocs/

msgpack-numpy

A library that enables serialization and deserialization of numpy arrays and scalar data types using the highly efficient msgpack format. The package provides both automatic monkey-patching functionality and manual encoder/decoder functions, ensuring preservation of numerical data types and precision during serialization/deserialization operations.

Package Information

  • Package Name: msgpack-numpy
  • Package Type: pypi
  • Language: Python
  • Installation: pip install msgpack-numpy

Core Imports

import msgpack_numpy as m

For manual encoding/decoding:

import msgpack_numpy as m
import msgpack

Basic Usage

Monkey Patching Approach (Automatic)

The easiest way to use msgpack-numpy is to call its monkey patching function:

import msgpack
import msgpack_numpy as m
import numpy as np

# Enable automatic numpy support
m.patch()

# Now all msgpack operations are numpy-aware
x = np.random.rand(5)
x_packed = msgpack.packb(x)
x_unpacked = msgpack.unpackb(x_packed)

Manual Encoding/Decoding

For explicit control over the serialization process:

import msgpack
import msgpack_numpy as m
import numpy as np

x = np.random.rand(5)
x_packed = msgpack.packb(x, default=m.encode)
x_unpacked = msgpack.unpackb(x_packed, object_hook=m.decode)

Using Direct Pack/Unpack Functions

import msgpack_numpy as m
import numpy as np

x = np.array([1, 2, 3], dtype=np.int32)
x_packed = m.packb(x)
x_unpacked = m.unpackb(x_packed)

Capabilities

Core Encoder/Decoder Functions

Functions for explicit serialization and deserialization of numpy data types.

def encode(obj, chain=None):
    """
    Data encoder for serializing numpy data types.
    
    Parameters:
    - obj: Object to encode (numpy arrays, scalars, complex numbers, or other objects)
    - chain: Optional chained encoder function for non-numpy objects
    
    Returns:
    Dictionary with serialization metadata or original object if not numpy type
    """

def decode(obj, chain=None):
    """
    Decoder for deserializing numpy data types.
    
    Parameters:
    - obj: Object to decode (serialized dictionary or other objects)
    - chain: Optional chained decoder function for non-numpy objects
    
    Returns:
    Reconstructed numpy array/scalar or original object if not serialized numpy type
    """

Packer and Unpacker Classes

Msgpack classes with automatic numpy support built-in.

class Packer:
    """
    Msgpack packer with automatic numpy encoding support.
    Inherits from msgpack.Packer with numpy-aware default encoder.
    """
    def __init__(self, default=None, **kwargs):
        """
        Initialize packer with numpy support.
        
        Parameters:
        - default: Optional additional default encoder (chained with numpy encoder)
        - **kwargs: Additional msgpack.Packer parameters (version-dependent)
        """
    
    def pack(self, obj):
        """Pack object with numpy support."""

class Unpacker:
    """
    Msgpack unpacker with automatic numpy decoding support.
    Inherits from msgpack.Unpacker with numpy-aware object hook.
    """
    def __init__(self, file_like=None, object_hook=None, **kwargs):
        """
        Initialize unpacker with numpy support.
        
        Parameters:
        - file_like: Optional file-like object to read from
        - object_hook: Optional additional object hook (chained with numpy decoder)
        - **kwargs: Additional msgpack.Unpacker parameters (version-dependent)
        """

Pack and Unpack Functions

High-level functions for packing and unpacking with numpy support.

def pack(o, stream, **kwargs):
    """
    Pack an object and write it to a stream with numpy support.
    
    Parameters:
    - o: Object to pack
    - stream: Output stream to write to
    - **kwargs: Additional keyword arguments passed to Packer
    """

def packb(o, **kwargs):
    """
    Pack an object and return the packed bytes with numpy support.
    
    Parameters:
    - o: Object to pack
    - **kwargs: Additional keyword arguments passed to Packer
    
    Returns:
    bytes: Packed bytes
    """

def unpack(stream, **kwargs):
    """
    Unpack a packed object from a stream with numpy support.
    
    Parameters:
    - stream: Input stream to read from
    - **kwargs: Additional keyword arguments passed to underlying unpack function
    
    Returns:
    Unpacked object with numpy types preserved
    """

def unpackb(packed, **kwargs):
    """
    Unpack a packed object with numpy support.
    
    Parameters:
    - packed: Packed bytes to unpack
    - **kwargs: Additional keyword arguments passed to underlying unpackb function
    
    Returns:
    Unpacked object with numpy types preserved
    """

Function Aliases

Convenience aliases for common operations.

# Aliases for pack/unpack functions
load = unpack    # Alias for unpack function
loads = unpackb  # Alias for unpackb function  
dump = pack      # Alias for pack function
dumps = packb    # Alias for packb function

Monkey Patching

Function to automatically enable numpy support in the msgpack module.

def patch():
    """
    Monkey patch msgpack module to enable support for serializing numpy types.
    
    This function replaces msgpack's Packer, Unpacker, and pack/unpack functions
    with numpy-aware versions. After calling this function, all msgpack operations
    will automatically handle numpy arrays and scalar types.
    """

Supported Data Types

The library supports serialization/deserialization of:

  • Numpy arrays: All dtypes including structured dtypes and object arrays
  • Numpy scalars: All numpy scalar types (bool_, int8, int16, int32, int64, float32, float64, complex64, complex128, etc.)
  • Python complex numbers: Native Python complex type
  • Regular Python objects: Passed through to chain functions or standard msgpack handling

Type Preservation

msgpack-numpy preserves:

  • Exact numpy data types (dtype information)
  • Array shapes and dimensions
  • Numerical precision
  • Complex number representations

Note: Arrays with dtype='O' are serialized using pickle as a fallback, which may have performance implications.

Notes

  • Deserialized numpy arrays are read-only and must be copied if modification is needed
  • The library maintains compatibility with Python 2.7 and 3.5+
  • Compatible with msgpack versions from <0.4.0 to >=1.0.0
  • Cross-platform support (Windows, macOS, Linux)