Model Output Validator

A utility for validating machine learning model outputs by comparing nested data structures with precise floating-point tolerance handling.

Capabilities

Validates matching tree structures

• When given two dictionaries with identical keys {"loss": 1.0, "accuracy": 0.95} and {"loss": 1.0, "accuracy": 0.95}, validation passes @test
• When given two lists of the same length [1.0, 2.0, 3.0] and [1.0, 2.0, 3.0], validation passes @test
• When given nested structures with the same shape {"metrics": [1.0, 2.0], "stats": {"mean": 1.5}} and {"metrics": [1.0, 2.0], "stats": {"mean": 1.5}}, validation passes @test
• When given structures with different shapes {"a": 1.0} and {"b": 1.0}, validation fails with an informative error @test

Handles floating-point precision correctly

• When comparing values that differ by 1 ULP at 1.0 + 1e-15 and 1.0, with max_ulp=2, validation passes @test
• When comparing values that differ by more than the allowed ULP threshold at 1.0 and 1.0 + 1e-14, with max_ulp=2, validation fails @test
• When comparing identical floating-point arrays with shape (3,) containing [1.0, 2.0, 3.0], validation passes with default tolerance @test

Validates nested model outputs

• When comparing nested dictionaries with small floating-point differences where expected={"layer1": {"weights": [1.0, 2.0]}, "layer2": {"bias": [0.5]}} and actual={"layer1": {"weights": [1.0 + 1e-15, 2.0]}, "layer2": {"bias": [0.5]}}, with max_ulp=10, validation passes @test
• When comparing lists of dictionaries where expected=[{"loss": 0.1}, {"loss": 0.2}] and actual=[{"loss": 0.1 + 1e-15}, {"loss": 0.2}], with max_ulp=5, validation passes @test

Implementation

@generates

API

def validate_outputs(expected, actual, max_ulp=1):
    """
    Validates that two nested data structures are equivalent within floating-point tolerance.

    Args:
        expected: Expected output structure (nested dicts, lists, arrays)
        actual: Actual output structure to validate
        max_ulp: Maximum difference in Units in Last Place (ULP) for floating-point comparison

    Raises:
        ValueError: If structures don't match or values differ beyond tolerance
    """
    pass

Dependencies { .dependencies }

chex { .dependency }

Provides tree comparison utilities with ULP-based floating-point precision handling.

@satisfied-by