or run

tessl search
Log in

Version

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/chonkie@1.5.x

docs

advanced-features.mdchunkers.mdcore-types.mddata-processing.mdembeddings.mdexport.mdindex.mdlogging.mdpipeline.mdrefineries.mdtokenizers.mdvector-databases.md
tile.json

tessl/pypi-chonkie

tessl install tessl/pypi-chonkie@1.5.0

The lightweight ingestion library for fast, efficient and robust RAG pipelines

logging.mddocs/

Logging

Centralized logging configuration for the Chonkie library. The logger module provides a simple, extensible interface using Python's standard logging, with programmatic control and environment variable configuration.

Capabilities

Logger Configuration

Control logging behavior through environment variables or programmatic API. Logging defaults to WARNING level but can be customized per application needs.

from chonkie.logger import get_logger, configure, disable, enable, is_enabled

def get_logger(module_name: str):
    """
    Get a logger instance for a specific module.

    This function returns a standard Python logger with hierarchical naming,
    automatically configured based on CHONKIE_LOG environment variable.

    Args:
        module_name: The name of the module requesting the logger (typically __name__)

    Returns:
        LoggerAdapter instance for the module

    Example:
        >>> from chonkie.logger import get_logger
        >>> logger = get_logger(__name__)
        >>> logger.info("Processing chunks")
        >>> logger.debug("Detailed information")
    """

def configure(level: str = None, format: str = None) -> None:
    """
    Configure Chonkie's logging system programmatically.

    This function allows you to override the CHONKIE_LOG environment variable.
    Can be called multiple times to reconfigure logging.

    Args:
        level: Log level or control string:
            - "off"/"false"/"0"/"disabled": Disable logging
            - "error"/"1": ERROR level only
            - "warning"/"2": WARNING and above (default)
            - "info"/"3": INFO and above
            - "debug"/"4": DEBUG and above
            - None: Use CHONKIE_LOG env var or WARNING if not set
        format: Optional custom format string. Uses default if None.

    Example:
        >>> import chonkie.logger as logger
        >>> logger.configure("debug")
        >>> logger.configure("off")  # Disable logging
        >>> logger.configure("info")  # Re-enable at INFO level
    """

def disable() -> None:
    """
    Disable all Chonkie logging.

    This is equivalent to configure("off").
    Useful for suppressing logs in production or testing environments.

    Example:
        >>> import chonkie.logger as logger
        >>> logger.disable()
        >>> # No logs will be output
    """

def enable(level: str = "INFO") -> None:
    """
    Re-enable Chonkie logging after it has been disabled.

    Args:
        level: The log level to enable. Defaults to INFO.

    Example:
        >>> import chonkie.logger as logger
        >>> logger.disable()
        >>> # ... do some work without logs ...
        >>> logger.enable("debug")
        >>> # Logs are back at DEBUG level
    """

def is_enabled() -> bool:
    """
    Check if logging is currently enabled.

    Returns:
        True if logging is enabled, False otherwise

    Example:
        >>> from chonkie.logger import is_enabled
        >>> if is_enabled():
        ...     logger.debug("This will be logged")
    """

Imports

Access logging functionality from the logger module:

from chonkie.logger import (
    get_logger,
    configure,
    disable,
    enable,
    is_enabled
)

Environment Variable Configuration

The CHONKIE_LOG environment variable controls default logging behavior:

ValueEffect
Not setWARNING and above (default, shows warnings and errors)
off, false, 0, disabled, noneDisable logging
error, 1ERROR level only
warning, 2WARNING and above (same as default)
info, 3INFO and above
debug, 4DEBUG and above (most verbose)

Usage Examples

Basic Logger Usage

from chonkie.logger import get_logger

# Get logger for your module
logger = get_logger(__name__)

# Log at different levels
logger.debug("Detailed debugging information")
logger.info("Informational message about normal operation")
logger.warning("Warning about potential issue")
logger.error("Error occurred during processing")

Programmatic Configuration

import chonkie.logger as logger

# Enable debug logging
logger.configure("debug")

# Use logger in your code
from chonkie import RecursiveChunker
chunker = RecursiveChunker()
chunks = chunker("Some text")  # Will show debug logs

# Disable logging for production
logger.configure("off")

Conditional Logging

from chonkie.logger import get_logger, is_enabled

logger = get_logger(__name__)

# Only compute expensive log messages if logging is enabled
if is_enabled():
    expensive_data = compute_statistics()
    logger.debug(f"Statistics: {expensive_data}")

Environment-Based Configuration

Set environment variable before running your application:

# Enable debug logging
export CHONKIE_LOG=debug
python your_app.py

# Disable all logging
export CHONKIE_LOG=off
python your_app.py

# Use numeric levels
export CHONKIE_LOG=3  # INFO level
python your_app.py

Testing with Logging Disabled

import chonkie.logger as logger

# Disable logging during tests
logger.disable()

# Run your tests without log output
def test_chunking():
    from chonkie import TokenChunker
    chunker = TokenChunker()
    chunks = chunker("Test text")
    assert len(chunks) > 0

# Re-enable if needed
logger.enable("warning")

Default Configuration

  • Default Level: WARNING (shows warnings and errors)
  • Default Format: %(asctime)s | %(levelname)-8s | %(name)s:%(funcName)s:%(lineno)d - %(message)s
  • Output: stderr
  • Hierarchy: Supports hierarchical loggers (e.g., chonkie.chunker.base inherits from chonkie.chunker and chonkie)

Integration with Application Logging

The Chonkie logger integrates seamlessly with Python's standard logging:

import logging
import chonkie.logger as chonkie_logger

# Configure your application logging
logging.basicConfig(level=logging.INFO)

# Configure Chonkie logging to match or differ
chonkie_logger.configure("info")

# Both will use the same logging infrastructure
app_logger = logging.getLogger("myapp")
chonkie_internal_logger = chonkie_logger.get_logger("chonkie.chunker")

app_logger.info("Application message")
# Chonkie internal logs will also appear at INFO level