Numpy data serialization using msgpack format with type preservation
npx @tessl/cli install tessl/pypi-msgpack-numpy@0.4.0A library that enables serialization and deserialization of numpy arrays and scalar data types using the highly efficient msgpack format. The package provides both automatic monkey-patching functionality and manual encoder/decoder functions, ensuring preservation of numerical data types and precision during serialization/deserialization operations.
pip install msgpack-numpyimport msgpack_numpy as mFor manual encoding/decoding:
import msgpack_numpy as m
import msgpackThe easiest way to use msgpack-numpy is to call its monkey patching function:
import msgpack
import msgpack_numpy as m
import numpy as np
# Enable automatic numpy support
m.patch()
# Now all msgpack operations are numpy-aware
x = np.random.rand(5)
x_packed = msgpack.packb(x)
x_unpacked = msgpack.unpackb(x_packed)For explicit control over the serialization process:
import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_packed = msgpack.packb(x, default=m.encode)
x_unpacked = msgpack.unpackb(x_packed, object_hook=m.decode)import msgpack_numpy as m
import numpy as np
x = np.array([1, 2, 3], dtype=np.int32)
x_packed = m.packb(x)
x_unpacked = m.unpackb(x_packed)Functions for explicit serialization and deserialization of numpy data types.
def encode(obj, chain=None):
"""
Data encoder for serializing numpy data types.
Parameters:
- obj: Object to encode (numpy arrays, scalars, complex numbers, or other objects)
- chain: Optional chained encoder function for non-numpy objects
Returns:
Dictionary with serialization metadata or original object if not numpy type
"""
def decode(obj, chain=None):
"""
Decoder for deserializing numpy data types.
Parameters:
- obj: Object to decode (serialized dictionary or other objects)
- chain: Optional chained decoder function for non-numpy objects
Returns:
Reconstructed numpy array/scalar or original object if not serialized numpy type
"""Msgpack classes with automatic numpy support built-in.
class Packer:
"""
Msgpack packer with automatic numpy encoding support.
Inherits from msgpack.Packer with numpy-aware default encoder.
"""
def __init__(self, default=None, **kwargs):
"""
Initialize packer with numpy support.
Parameters:
- default: Optional additional default encoder (chained with numpy encoder)
- **kwargs: Additional msgpack.Packer parameters (version-dependent)
"""
def pack(self, obj):
"""Pack object with numpy support."""
class Unpacker:
"""
Msgpack unpacker with automatic numpy decoding support.
Inherits from msgpack.Unpacker with numpy-aware object hook.
"""
def __init__(self, file_like=None, object_hook=None, **kwargs):
"""
Initialize unpacker with numpy support.
Parameters:
- file_like: Optional file-like object to read from
- object_hook: Optional additional object hook (chained with numpy decoder)
- **kwargs: Additional msgpack.Unpacker parameters (version-dependent)
"""High-level functions for packing and unpacking with numpy support.
def pack(o, stream, **kwargs):
"""
Pack an object and write it to a stream with numpy support.
Parameters:
- o: Object to pack
- stream: Output stream to write to
- **kwargs: Additional keyword arguments passed to Packer
"""
def packb(o, **kwargs):
"""
Pack an object and return the packed bytes with numpy support.
Parameters:
- o: Object to pack
- **kwargs: Additional keyword arguments passed to Packer
Returns:
bytes: Packed bytes
"""
def unpack(stream, **kwargs):
"""
Unpack a packed object from a stream with numpy support.
Parameters:
- stream: Input stream to read from
- **kwargs: Additional keyword arguments passed to underlying unpack function
Returns:
Unpacked object with numpy types preserved
"""
def unpackb(packed, **kwargs):
"""
Unpack a packed object with numpy support.
Parameters:
- packed: Packed bytes to unpack
- **kwargs: Additional keyword arguments passed to underlying unpackb function
Returns:
Unpacked object with numpy types preserved
"""Convenience aliases for common operations.
# Aliases for pack/unpack functions
load = unpack # Alias for unpack function
loads = unpackb # Alias for unpackb function
dump = pack # Alias for pack function
dumps = packb # Alias for packb functionFunction to automatically enable numpy support in the msgpack module.
def patch():
"""
Monkey patch msgpack module to enable support for serializing numpy types.
This function replaces msgpack's Packer, Unpacker, and pack/unpack functions
with numpy-aware versions. After calling this function, all msgpack operations
will automatically handle numpy arrays and scalar types.
"""The library supports serialization/deserialization of:
msgpack-numpy preserves:
Note: Arrays with dtype='O' are serialized using pickle as a fallback, which may have performance implications.