CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/pypi-hiredis

Python wrapper for hiredis that speeds up parsing of Redis multi bulk replies

Pending
Overview
Eval results
Files

hiredis

Python extension that wraps protocol parsing code in hiredis to significantly speed up parsing of Redis multi bulk replies. The library provides a Reader class for parsing replies from Redis data streams with support for Unicode decoding, customizable error handling, and efficient memory usage.

Package Information

  • Package Name: hiredis
  • Package Type: pypi
  • Language: Python
  • Installation: pip install hiredis
  • Python Requirements: Python 3.8+

Core Imports

import hiredis

Specific imports:

from hiredis import Reader, pack_command, HiredisError, ProtocolError, ReplyError, PushNotification

Basic Usage

import hiredis

# Create a reader for parsing Redis replies
reader = hiredis.Reader()

# Feed data from Redis connection
reader.feed("$5\r\nhello\r\n")

# Extract parsed reply
reply = reader.gets()  # Returns b'hello'

# Pack commands into Redis protocol format  
packed = hiredis.pack_command(("SET", "key", "value"))
# Returns: b'*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$5\r\nvalue\r\n'

Capabilities

Redis Reply Parsing

The Reader class parses Redis protocol replies from data streams, providing efficient protocol parsing with configurable decoding and error handling.

class Reader:
    def __init__(
        self,
        protocolError: Callable[[str], Exception] = None,
        replyError: Callable[[str], Exception] = None,
        encoding: Optional[str] = None,
        errors: Optional[str] = None,
        notEnoughData: Any = False,
    ) -> None:
        """
        Create a new Reader for parsing Redis replies.

        Parameters:
        - protocolError: Custom exception class/callable for protocol errors
        - replyError: Custom exception class/callable for reply errors  
        - encoding: Text encoding for bulk data decoding (e.g., 'utf-8')
        - errors: Error handling mode ('strict', 'replace', 'ignore', 'backslashreplace')
        - notEnoughData: Custom sentinel returned when buffer lacks complete reply
        """

    def feed(self, buf: Union[str, bytes], off: int = 0, len: int = -1) -> None:
        """
        Feed data to the internal buffer for parsing.

        Parameters:
        - buf: Data to append to buffer (string or bytes)
        - off: Starting offset in buffer (default: 0)
        - len: Number of bytes to read (default: -1 for all)
        """

    def gets(self, shouldDecode: bool = True) -> Any:
        """
        Extract a reply from the buffer.

        Parameters:
        - shouldDecode: Whether to decode bulk data using configured encoding

        Returns:
        - Parsed reply data, or False if buffer lacks complete reply
        - Returns notEnoughData sentinel if configured
        """

    def setmaxbuf(self, maxbuf: Optional[int]) -> None:
        """
        Set maximum buffer size to control memory usage.

        Parameters:
        - maxbuf: Maximum buffer size in bytes, None for unlimited
        """

    def getmaxbuf(self) -> int:
        """
        Get current maximum buffer size.

        Returns:
        - Current maximum buffer size in bytes
        """

    def len(self) -> int:
        """
        Get length of data in internal buffer.

        Returns:
        - Number of bytes in buffer
        """

    def has_data(self) -> bool:
        """
        Check if buffer contains any data.

        Returns:
        - True if buffer has data, False otherwise
        """

    def set_encoding(
        self, encoding: Optional[str] = None, errors: Optional[str] = None
    ) -> None:
        """
        Update encoding configuration for bulk data decoding.

        Parameters:
        - encoding: Text encoding (e.g., 'utf-8'), None to disable decoding
        - errors: Error handling mode ('strict', 'replace', 'ignore', 'backslashreplace')
        """

Command Packing

Convert Python command tuples into Redis protocol format for transmission to Redis servers.

def pack_command(cmd: Tuple[Union[str, int, float, bytes, memoryview], ...]) -> bytes:
    """
    Pack command arguments into Redis protocol format.

    Parameters:
    - cmd: Tuple containing command and arguments
           Supports: str, int, float, bytes, memoryview

    Returns:
    - Bytes object containing packed Redis protocol command

    Raises:
    - TypeError: If command contains unsupported argument types
    """

Error Handling

Exception classes for different error conditions during Redis protocol parsing.

class HiredisError(Exception):
    """Base exception class for all hiredis errors."""

class ProtocolError(HiredisError):
    """
    Raised when protocol parsing fails due to corrupt data stream.
    Indicates unrecoverable protocol state requiring reconnection.
    """

class ReplyError(HiredisError):
    """
    Represents Redis error replies (-ERR responses).
    This exception is returned by gets(), not raised.
    """

class PushNotification(list):
    """
    Represents Redis push notifications in RESP3 protocol.
    Extends list to contain notification data.
    """

Version Information

__version__: str
# Current package version string (e.g., "3.2.1")

Usage Examples

Unicode Decoding

import hiredis

# Configure reader for UTF-8 decoding
reader = hiredis.Reader(encoding="utf-8", errors="strict")
reader.feed(b"$3\r\n\xe2\x98\x83\r\n")  # UTF-8 snowman
reply = reader.gets()  # Returns '☃' (string, not bytes)

Custom Error Handling

import hiredis

# Use custom exception classes
class MyProtocolError(Exception):
    pass

class MyReplyError(Exception):
    pass

reader = hiredis.Reader(
    protocolError=MyProtocolError,
    replyError=MyReplyError
)

# Protocol errors will raise MyProtocolError
# Reply errors will return MyReplyError instances

Buffer Management

import hiredis

reader = hiredis.Reader()

# Set buffer size limit
reader.setmaxbuf(1024 * 1024)  # 1MB max

# Monitor buffer usage
print(f"Buffer size: {reader.len()} bytes")
print(f"Has data: {reader.has_data()}")
print(f"Max buffer: {reader.getmaxbuf()} bytes")

Multi-Reply Processing

import hiredis

reader = hiredis.Reader()

# Feed data containing multiple replies
reader.feed("$5\r\nhello\r\n$5\r\nworld\r\n")

# Extract all replies
replies = []
while True:
    reply = reader.gets()
    if reply is False:  # No more complete replies
        break
    replies.append(reply)

print(replies)  # [b'hello', b'world']

Command Packing Examples

import hiredis

# Basic command
cmd = hiredis.pack_command(("GET", "mykey"))
# Returns: b'*2\r\n$3\r\nGET\r\n$5\r\nmykey\r\n'

# Complex command with different data types
cmd = hiredis.pack_command((
    "HSET", "hash", "field1", "value1", 
    b"field2", b"binary_value", 
    "field3", 42,
    "field4", 3.14159
))

# Command with memoryview
data = memoryview(b"binary data")
cmd = hiredis.pack_command(("SET", "key", data))

Performance Notes

  • Provides significant performance improvements for parsing multi bulk replies
  • Throughput improvements increase with reply size (up to 83x faster for large LRANGE operations)
  • Reader instances are not thread-safe - use separate instances per thread
  • Memory usage is controlled through buffer size limits
  • Best used as a performance enhancement for Redis client libraries

Install with Tessl CLI

npx tessl i tessl/pypi-hiredis
Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/hiredis@3.2.x
Publish Source
CLI
Badge
tessl/pypi-hiredis badge