CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/pypi-msgpack-numpy

Numpy data serialization using msgpack format with type preservation

Pending
Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Pending

The risk profile of this skill

Overview
Eval results
Files

msgpack-numpy

A library that enables serialization and deserialization of numpy arrays and scalar data types using the highly efficient msgpack format. The package provides both automatic monkey-patching functionality and manual encoder/decoder functions, ensuring preservation of numerical data types and precision during serialization/deserialization operations.

Package Information

  • Package Name: msgpack-numpy
  • Package Type: pypi
  • Language: Python
  • Installation: pip install msgpack-numpy

Core Imports

import msgpack_numpy as m

For manual encoding/decoding:

import msgpack_numpy as m
import msgpack

Basic Usage

Monkey Patching Approach (Automatic)

The easiest way to use msgpack-numpy is to call its monkey patching function:

import msgpack
import msgpack_numpy as m
import numpy as np

# Enable automatic numpy support
m.patch()

# Now all msgpack operations are numpy-aware
x = np.random.rand(5)
x_packed = msgpack.packb(x)
x_unpacked = msgpack.unpackb(x_packed)

Manual Encoding/Decoding

For explicit control over the serialization process:

import msgpack
import msgpack_numpy as m
import numpy as np

x = np.random.rand(5)
x_packed = msgpack.packb(x, default=m.encode)
x_unpacked = msgpack.unpackb(x_packed, object_hook=m.decode)

Using Direct Pack/Unpack Functions

import msgpack_numpy as m
import numpy as np

x = np.array([1, 2, 3], dtype=np.int32)
x_packed = m.packb(x)
x_unpacked = m.unpackb(x_packed)

Capabilities

Core Encoder/Decoder Functions

Functions for explicit serialization and deserialization of numpy data types.

def encode(obj, chain=None):
    """
    Data encoder for serializing numpy data types.
    
    Parameters:
    - obj: Object to encode (numpy arrays, scalars, complex numbers, or other objects)
    - chain: Optional chained encoder function for non-numpy objects
    
    Returns:
    Dictionary with serialization metadata or original object if not numpy type
    """

def decode(obj, chain=None):
    """
    Decoder for deserializing numpy data types.
    
    Parameters:
    - obj: Object to decode (serialized dictionary or other objects)
    - chain: Optional chained decoder function for non-numpy objects
    
    Returns:
    Reconstructed numpy array/scalar or original object if not serialized numpy type
    """

Packer and Unpacker Classes

Msgpack classes with automatic numpy support built-in.

class Packer:
    """
    Msgpack packer with automatic numpy encoding support.
    Inherits from msgpack.Packer with numpy-aware default encoder.
    """
    def __init__(self, default=None, **kwargs):
        """
        Initialize packer with numpy support.
        
        Parameters:
        - default: Optional additional default encoder (chained with numpy encoder)
        - **kwargs: Additional msgpack.Packer parameters (version-dependent)
        """
    
    def pack(self, obj):
        """Pack object with numpy support."""

class Unpacker:
    """
    Msgpack unpacker with automatic numpy decoding support.
    Inherits from msgpack.Unpacker with numpy-aware object hook.
    """
    def __init__(self, file_like=None, object_hook=None, **kwargs):
        """
        Initialize unpacker with numpy support.
        
        Parameters:
        - file_like: Optional file-like object to read from
        - object_hook: Optional additional object hook (chained with numpy decoder)
        - **kwargs: Additional msgpack.Unpacker parameters (version-dependent)
        """

Pack and Unpack Functions

High-level functions for packing and unpacking with numpy support.

def pack(o, stream, **kwargs):
    """
    Pack an object and write it to a stream with numpy support.
    
    Parameters:
    - o: Object to pack
    - stream: Output stream to write to
    - **kwargs: Additional keyword arguments passed to Packer
    """

def packb(o, **kwargs):
    """
    Pack an object and return the packed bytes with numpy support.
    
    Parameters:
    - o: Object to pack
    - **kwargs: Additional keyword arguments passed to Packer
    
    Returns:
    bytes: Packed bytes
    """

def unpack(stream, **kwargs):
    """
    Unpack a packed object from a stream with numpy support.
    
    Parameters:
    - stream: Input stream to read from
    - **kwargs: Additional keyword arguments passed to underlying unpack function
    
    Returns:
    Unpacked object with numpy types preserved
    """

def unpackb(packed, **kwargs):
    """
    Unpack a packed object with numpy support.
    
    Parameters:
    - packed: Packed bytes to unpack
    - **kwargs: Additional keyword arguments passed to underlying unpackb function
    
    Returns:
    Unpacked object with numpy types preserved
    """

Function Aliases

Convenience aliases for common operations.

# Aliases for pack/unpack functions
load = unpack    # Alias for unpack function
loads = unpackb  # Alias for unpackb function  
dump = pack      # Alias for pack function
dumps = packb    # Alias for packb function

Monkey Patching

Function to automatically enable numpy support in the msgpack module.

def patch():
    """
    Monkey patch msgpack module to enable support for serializing numpy types.
    
    This function replaces msgpack's Packer, Unpacker, and pack/unpack functions
    with numpy-aware versions. After calling this function, all msgpack operations
    will automatically handle numpy arrays and scalar types.
    """

Supported Data Types

The library supports serialization/deserialization of:

  • Numpy arrays: All dtypes including structured dtypes and object arrays
  • Numpy scalars: All numpy scalar types (bool_, int8, int16, int32, int64, float32, float64, complex64, complex128, etc.)
  • Python complex numbers: Native Python complex type
  • Regular Python objects: Passed through to chain functions or standard msgpack handling

Type Preservation

msgpack-numpy preserves:

  • Exact numpy data types (dtype information)
  • Array shapes and dimensions
  • Numerical precision
  • Complex number representations

Note: Arrays with dtype='O' are serialized using pickle as a fallback, which may have performance implications.

Notes

  • Deserialized numpy arrays are read-only and must be copied if modification is needed
  • The library maintains compatibility with Python 2.7 and 3.5+
  • Compatible with msgpack versions from <0.4.0 to >=1.0.0
  • Cross-platform support (Windows, macOS, Linux)
Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/msgpack-numpy@0.4.x
Publish Source
CLI
Badge
tessl/pypi-msgpack-numpy badge