or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

calibration.mdclassification.mdindex.mdmetrics.mdregression.mdrisk-control.mdutils.md
tile.json

tessl/pypi-mapie

A scikit-learn-compatible module for estimating prediction intervals using conformal prediction methods.

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/mapie@1.0.x

To install, run

npx @tessl/cli install tessl/pypi-mapie@1.0.0

index.mddocs/

MAPIE

MAPIE (Model Agnostic Prediction Interval Estimator) is a comprehensive Python library for quantifying uncertainties and controlling risks in machine learning models through conformal prediction methods. The library provides model-agnostic tools to compute prediction intervals for regression, prediction sets for classification, and uncertainty quantification for time series, while offering risk control capabilities for complex tasks like multi-label classification and semantic segmentation.

Package Information

  • Package Name: MAPIE
  • Language: Python
  • Installation: pip install MAPIE

Core Imports

import mapie

Common for regression prediction intervals:

from mapie.regression import SplitConformalRegressor, CrossConformalRegressor

Common for classification prediction sets:

from mapie.classification import SplitConformalClassifier, CrossConformalClassifier

Basic Usage

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from mapie.regression import SplitConformalRegressor

# Generate sample data
X = np.random.rand(1000, 5)
y = np.sum(X, axis=1) + np.random.normal(0, 0.1, 1000)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_calib, y_train, y_calib = train_test_split(X_train, y_train, test_size=0.25, random_state=42)

# Create and fit base estimator
base_estimator = RandomForestRegressor(random_state=42)
base_estimator.fit(X_train, y_train)

# Create conformal predictor
mapie_reg = SplitConformalRegressor(estimator=base_estimator, prefit=True)
mapie_reg.conformalize(X_calib, y_calib)

# Make predictions with uncertainty intervals
y_pred, y_intervals = mapie_reg.predict_interval(X_test)
print(f"Predictions: {y_pred[:5]}")
print(f"90% Prediction intervals: {y_intervals[:5]}")

Architecture

MAPIE implements conformal prediction methods based on distribution-free inference with theoretical guarantees:

  • Conformal Prediction Framework: Distribution-free methods providing finite-sample coverage guarantees without assumptions about data distributions
  • Split Conformal Methods: Separate training and conformalization phases for efficiency and flexibility
  • Cross Conformal Methods: Cross-validation based approaches for better data utilization
  • Conformity Scores: Pluggable non-conformity measures that quantify prediction uncertainty
  • Risk Control: Advanced methods for controlling prediction risks in multi-label and complex scenarios

This design enables integration with any scikit-learn compatible estimator while providing rigorous uncertainty quantification for both regression and classification tasks.

Capabilities

Regression Prediction Intervals

Conformal prediction methods for regression that provide prediction intervals with finite-sample coverage guarantees. Supports split conformal, cross conformal, jackknife-after-bootstrap, and time series methods.

class SplitConformalRegressor:
    def __init__(self, estimator=None, confidence_level=0.9, conformity_score="absolute", prefit=True, n_jobs=None, verbose=0): ...
    def fit(self, X_train, y_train, fit_params=None): ...
    def conformalize(self, X_conformalize, y_conformalize, predict_params=None): ...
    def predict_interval(self, X, minimize_interval_width=False, allow_infinite_bounds=False): ...

class CrossConformalRegressor:
    def __init__(self, estimator=None, confidence_level=0.9, conformity_score="absolute", method="plus", cv=5, n_jobs=None, verbose=0, random_state=None): ...
    def fit_conformalize(self, X, y, groups=None, fit_params=None, predict_params=None): ...
    def predict_interval(self, X, aggregate_predictions="mean", minimize_interval_width=False, allow_infinite_bounds=False): ...

class JackknifeAfterBootstrapRegressor:
    def __init__(self, estimator=None, confidence_level=0.9, conformity_score="absolute", method="plus", resampling=30, n_jobs=None, verbose=0, random_state=None): ...
    def fit_conformalize(self, X, y, fit_params=None, predict_params=None): ...
    def predict_interval(self, X, ensemble=True, minimize_interval_width=False, allow_infinite_bounds=False): ...

class TimeSeriesRegressor:
    def __init__(self, estimator=None, method="enbpi", cv=None, n_jobs=None, agg_function="mean", verbose=0, conformity_score=None, random_state=None): ...
    def fit(self, X, y, sample_weight=None): ...
    def partial_fit(self, X, y): ...
    def predict_interval(self, X, confidence_level=None): ...

Regression Methods

Classification Prediction Sets

Conformal prediction methods for classification that provide prediction sets containing the true label with specified probability. Supports split conformal and cross conformal approaches with various conformity scores.

class SplitConformalClassifier:
    def __init__(self, estimator=None, confidence_level=0.9, conformity_score="lac", prefit=True, n_jobs=None, verbose=0, random_state=None): ...
    def fit(self, X_train, y_train, fit_params=None): ...
    def conformalize(self, X_conformalize, y_conformalize, predict_params=None): ...
    def predict_set(self, X, conformity_score_params=None): ...

class CrossConformalClassifier:
    def __init__(self, estimator=None, confidence_level=0.9, conformity_score="lac", cv=5, n_jobs=None, verbose=0, random_state=None): ...
    def fit_conformalize(self, X, y, groups=None, fit_params=None, predict_params=None): ...
    def predict_set(self, X, conformity_score_params=None, agg_scores="mean"): ...

Classification Methods

Risk Control

Advanced risk control methods for multi-label classification scenarios, enabling control of precision and recall metrics with finite-sample guarantees. Implements conformal risk control procedures.

class PrecisionRecallController:
    def __init__(self, estimator=None, metric_control='recall', method=None, n_jobs=None, random_state=None, verbose=0): ...
    def fit(self, X, y, conformalize_size=0.3): ...
    def partial_fit(self, X, y, _refit=False): ...
    def predict(self, X, alpha=None, delta=None, bound=None): ...

Risk Control

Calibration

Probability calibration methods for improving the reliability of probabilistic predictions, particularly for multi-class classification problems with top-label calibration.

class TopLabelCalibrator:
    def __init__(self, estimator=None, calibrator=None, cv="split"): ...
    def fit(self, X, y, sample_weight=None, calib_size=0.33, random_state=None, shuffle=True, stratify=None, **fit_params): ...
    def predict_proba(self, X): ...
    def predict(self, X): ...

Calibration Methods

Evaluation Metrics

Comprehensive evaluation metrics for assessing the quality of conformal prediction intervals and sets, including coverage, width, and calibration metrics.

# Regression metrics
from mapie.metrics.regression import regression_coverage_score, regression_mean_width_score

def regression_coverage_score(y_true, y_intervals): ...
def regression_mean_width_score(y_intervals): ...

# Classification metrics
from mapie.metrics.classification import classification_coverage_score, classification_mean_width_score

def classification_coverage_score(y_true, y_pred_set): ...
def classification_mean_width_score(y_pred_set): ...

# Calibration metrics
from mapie.metrics.calibration import expected_calibration_error, top_label_ece

def expected_calibration_error(y_true, y_scores, num_bins=50, split_strategy=None): ...
def top_label_ece(y_true, y_scores, y_score_arg=None, num_bins=50, split_strategy=None, classes=None): ...

Evaluation Metrics

Utilities and Sampling

Utility functions for data splitting, cross-validation strategies, and bootstrap sampling methods specifically designed for conformal prediction workflows.

def train_conformalize_test_split(X, y, train_size, conformalize_size, test_size, random_state=None, shuffle=True): ...

class Subsample:
    def __init__(self, n_resamplings=30, n_samples=None, replace=True, random_state=None): ...
    def split(self, X, *args, **kwargs): ...

class BlockBootstrap:
    def __init__(self, n_resamplings=30, length=None, n_blocks=None, overlapping=False, random_state=None): ...
    def split(self, X, *args, **kwargs): ...

Utilities and Sampling

Conformity Scores

Custom conformity score classes for regression and classification tasks, enabling pluggable uncertainty quantification methods.

# Base classes
from mapie.conformity_scores import BaseConformityScore, BaseRegressionScore, BaseClassificationScore

class BaseConformityScore(metaclass=ABCMeta): ...
class BaseRegressionScore(BaseConformityScore, metaclass=ABCMeta): ...
class BaseClassificationScore(BaseConformityScore, metaclass=ABCMeta): ...

# Regression conformity scores
from mapie.conformity_scores.bounds.absolute import AbsoluteConformityScore
from mapie.conformity_scores.bounds.gamma import GammaConformityScore
from mapie.conformity_scores.bounds.residuals import ResidualNormalisedScore

class AbsoluteConformityScore(BaseRegressionScore): ...
class GammaConformityScore(BaseRegressionScore): ...
class ResidualNormalisedScore(BaseRegressionScore): ...

# Classification conformity scores
from mapie.conformity_scores.sets.naive import NaiveConformityScore
from mapie.conformity_scores.sets.aps import APSConformityScore
from mapie.conformity_scores.sets.raps import RAPSConformityScore
from mapie.conformity_scores.sets.lac import LACConformityScore
from mapie.conformity_scores.sets.topk import TopKConformityScore

class NaiveConformityScore(BaseClassificationScore): ...
class APSConformityScore(NaiveConformityScore): ...
class RAPSConformityScore(APSConformityScore): ...
class LACConformityScore(BaseClassificationScore): ...
class TopKConformityScore(BaseClassificationScore): ...

Common Types

from typing import Union, Optional, List, Tuple, Dict, Any, Generator
from numpy.typing import NDArray, ArrayLike
from sklearn.base import BaseEstimator, RegressorMixin, ClassifierMixin
from sklearn.model_selection import BaseCrossValidator

# Common parameter types
ConfidenceLevel = Union[float, List[float]]
ConformityScore = Union[str, object]
CVStrategy = Union[int, BaseCrossValidator]
PredictionIntervals = Tuple[NDArray, NDArray]
PredictionSets = Tuple[NDArray, NDArray]