0
# MAPIE
1
2
MAPIE (Model Agnostic Prediction Interval Estimator) is a comprehensive Python library for quantifying uncertainties and controlling risks in machine learning models through conformal prediction methods. The library provides model-agnostic tools to compute prediction intervals for regression, prediction sets for classification, and uncertainty quantification for time series, while offering risk control capabilities for complex tasks like multi-label classification and semantic segmentation.
3
4
## Package Information
5
6
- **Package Name**: MAPIE
7
- **Language**: Python
8
- **Installation**: `pip install MAPIE`
9
10
## Core Imports
11
12
```python
13
import mapie
14
```
15
16
Common for regression prediction intervals:
17
18
```python
19
from mapie.regression import SplitConformalRegressor, CrossConformalRegressor
20
```
21
22
Common for classification prediction sets:
23
24
```python
25
from mapie.classification import SplitConformalClassifier, CrossConformalClassifier
26
```
27
28
## Basic Usage
29
30
```python
31
import numpy as np
32
from sklearn.ensemble import RandomForestRegressor
33
from sklearn.model_selection import train_test_split
34
from mapie.regression import SplitConformalRegressor
35
36
# Generate sample data
37
X = np.random.rand(1000, 5)
38
y = np.sum(X, axis=1) + np.random.normal(0, 0.1, 1000)
39
40
# Split data
41
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
42
X_train, X_calib, y_train, y_calib = train_test_split(X_train, y_train, test_size=0.25, random_state=42)
43
44
# Create and fit base estimator
45
base_estimator = RandomForestRegressor(random_state=42)
46
base_estimator.fit(X_train, y_train)
47
48
# Create conformal predictor
49
mapie_reg = SplitConformalRegressor(estimator=base_estimator, prefit=True)
50
mapie_reg.conformalize(X_calib, y_calib)
51
52
# Make predictions with uncertainty intervals
53
y_pred, y_intervals = mapie_reg.predict_interval(X_test)
54
print(f"Predictions: {y_pred[:5]}")
55
print(f"90% Prediction intervals: {y_intervals[:5]}")
56
```
57
58
## Architecture
59
60
MAPIE implements conformal prediction methods based on distribution-free inference with theoretical guarantees:
61
62
- **Conformal Prediction Framework**: Distribution-free methods providing finite-sample coverage guarantees without assumptions about data distributions
63
- **Split Conformal Methods**: Separate training and conformalization phases for efficiency and flexibility
64
- **Cross Conformal Methods**: Cross-validation based approaches for better data utilization
65
- **Conformity Scores**: Pluggable non-conformity measures that quantify prediction uncertainty
66
- **Risk Control**: Advanced methods for controlling prediction risks in multi-label and complex scenarios
67
68
This design enables integration with any scikit-learn compatible estimator while providing rigorous uncertainty quantification for both regression and classification tasks.
69
70
## Capabilities
71
72
### Regression Prediction Intervals
73
74
Conformal prediction methods for regression that provide prediction intervals with finite-sample coverage guarantees. Supports split conformal, cross conformal, jackknife-after-bootstrap, and time series methods.
75
76
```python { .api }
77
class SplitConformalRegressor:
78
def __init__(self, estimator=None, confidence_level=0.9, conformity_score="absolute", prefit=True, n_jobs=None, verbose=0): ...
79
def fit(self, X_train, y_train, fit_params=None): ...
80
def conformalize(self, X_conformalize, y_conformalize, predict_params=None): ...
81
def predict_interval(self, X, minimize_interval_width=False, allow_infinite_bounds=False): ...
82
83
class CrossConformalRegressor:
84
def __init__(self, estimator=None, confidence_level=0.9, conformity_score="absolute", method="plus", cv=5, n_jobs=None, verbose=0, random_state=None): ...
85
def fit_conformalize(self, X, y, groups=None, fit_params=None, predict_params=None): ...
86
def predict_interval(self, X, aggregate_predictions="mean", minimize_interval_width=False, allow_infinite_bounds=False): ...
87
88
class JackknifeAfterBootstrapRegressor:
89
def __init__(self, estimator=None, confidence_level=0.9, conformity_score="absolute", method="plus", resampling=30, n_jobs=None, verbose=0, random_state=None): ...
90
def fit_conformalize(self, X, y, fit_params=None, predict_params=None): ...
91
def predict_interval(self, X, ensemble=True, minimize_interval_width=False, allow_infinite_bounds=False): ...
92
93
class TimeSeriesRegressor:
94
def __init__(self, estimator=None, method="enbpi", cv=None, n_jobs=None, agg_function="mean", verbose=0, conformity_score=None, random_state=None): ...
95
def fit(self, X, y, sample_weight=None): ...
96
def partial_fit(self, X, y): ...
97
def predict_interval(self, X, confidence_level=None): ...
98
```
99
100
[Regression Methods](./regression.md)
101
102
### Classification Prediction Sets
103
104
Conformal prediction methods for classification that provide prediction sets containing the true label with specified probability. Supports split conformal and cross conformal approaches with various conformity scores.
105
106
```python { .api }
107
class SplitConformalClassifier:
108
def __init__(self, estimator=None, confidence_level=0.9, conformity_score="lac", prefit=True, n_jobs=None, verbose=0, random_state=None): ...
109
def fit(self, X_train, y_train, fit_params=None): ...
110
def conformalize(self, X_conformalize, y_conformalize, predict_params=None): ...
111
def predict_set(self, X, conformity_score_params=None): ...
112
113
class CrossConformalClassifier:
114
def __init__(self, estimator=None, confidence_level=0.9, conformity_score="lac", cv=5, n_jobs=None, verbose=0, random_state=None): ...
115
def fit_conformalize(self, X, y, groups=None, fit_params=None, predict_params=None): ...
116
def predict_set(self, X, conformity_score_params=None, agg_scores="mean"): ...
117
```
118
119
[Classification Methods](./classification.md)
120
121
### Risk Control
122
123
Advanced risk control methods for multi-label classification scenarios, enabling control of precision and recall metrics with finite-sample guarantees. Implements conformal risk control procedures.
124
125
```python { .api }
126
class PrecisionRecallController:
127
def __init__(self, estimator=None, metric_control='recall', method=None, n_jobs=None, random_state=None, verbose=0): ...
128
def fit(self, X, y, conformalize_size=0.3): ...
129
def partial_fit(self, X, y, _refit=False): ...
130
def predict(self, X, alpha=None, delta=None, bound=None): ...
131
```
132
133
[Risk Control](./risk-control.md)
134
135
### Calibration
136
137
Probability calibration methods for improving the reliability of probabilistic predictions, particularly for multi-class classification problems with top-label calibration.
138
139
```python { .api }
140
class TopLabelCalibrator:
141
def __init__(self, estimator=None, calibrator=None, cv="split"): ...
142
def fit(self, X, y, sample_weight=None, calib_size=0.33, random_state=None, shuffle=True, stratify=None, **fit_params): ...
143
def predict_proba(self, X): ...
144
def predict(self, X): ...
145
```
146
147
[Calibration Methods](./calibration.md)
148
149
### Evaluation Metrics
150
151
Comprehensive evaluation metrics for assessing the quality of conformal prediction intervals and sets, including coverage, width, and calibration metrics.
152
153
```python { .api }
154
# Regression metrics
155
from mapie.metrics.regression import regression_coverage_score, regression_mean_width_score
156
157
def regression_coverage_score(y_true, y_intervals): ...
158
def regression_mean_width_score(y_intervals): ...
159
160
# Classification metrics
161
from mapie.metrics.classification import classification_coverage_score, classification_mean_width_score
162
163
def classification_coverage_score(y_true, y_pred_set): ...
164
def classification_mean_width_score(y_pred_set): ...
165
166
# Calibration metrics
167
from mapie.metrics.calibration import expected_calibration_error, top_label_ece
168
169
def expected_calibration_error(y_true, y_scores, num_bins=50, split_strategy=None): ...
170
def top_label_ece(y_true, y_scores, y_score_arg=None, num_bins=50, split_strategy=None, classes=None): ...
171
```
172
173
[Evaluation Metrics](./metrics.md)
174
175
### Utilities and Sampling
176
177
Utility functions for data splitting, cross-validation strategies, and bootstrap sampling methods specifically designed for conformal prediction workflows.
178
179
```python { .api }
180
def train_conformalize_test_split(X, y, train_size, conformalize_size, test_size, random_state=None, shuffle=True): ...
181
182
class Subsample:
183
def __init__(self, n_resamplings=30, n_samples=None, replace=True, random_state=None): ...
184
def split(self, X, *args, **kwargs): ...
185
186
class BlockBootstrap:
187
def __init__(self, n_resamplings=30, length=None, n_blocks=None, overlapping=False, random_state=None): ...
188
def split(self, X, *args, **kwargs): ...
189
```
190
191
[Utilities and Sampling](./utils.md)
192
193
### Conformity Scores
194
195
Custom conformity score classes for regression and classification tasks, enabling pluggable uncertainty quantification methods.
196
197
```python { .api }
198
# Base classes
199
from mapie.conformity_scores import BaseConformityScore, BaseRegressionScore, BaseClassificationScore
200
201
class BaseConformityScore(metaclass=ABCMeta): ...
202
class BaseRegressionScore(BaseConformityScore, metaclass=ABCMeta): ...
203
class BaseClassificationScore(BaseConformityScore, metaclass=ABCMeta): ...
204
205
# Regression conformity scores
206
from mapie.conformity_scores.bounds.absolute import AbsoluteConformityScore
207
from mapie.conformity_scores.bounds.gamma import GammaConformityScore
208
from mapie.conformity_scores.bounds.residuals import ResidualNormalisedScore
209
210
class AbsoluteConformityScore(BaseRegressionScore): ...
211
class GammaConformityScore(BaseRegressionScore): ...
212
class ResidualNormalisedScore(BaseRegressionScore): ...
213
214
# Classification conformity scores
215
from mapie.conformity_scores.sets.naive import NaiveConformityScore
216
from mapie.conformity_scores.sets.aps import APSConformityScore
217
from mapie.conformity_scores.sets.raps import RAPSConformityScore
218
from mapie.conformity_scores.sets.lac import LACConformityScore
219
from mapie.conformity_scores.sets.topk import TopKConformityScore
220
221
class NaiveConformityScore(BaseClassificationScore): ...
222
class APSConformityScore(NaiveConformityScore): ...
223
class RAPSConformityScore(APSConformityScore): ...
224
class LACConformityScore(BaseClassificationScore): ...
225
class TopKConformityScore(BaseClassificationScore): ...
226
```
227
228
## Common Types
229
230
```python { .api }
231
from typing import Union, Optional, List, Tuple, Dict, Any, Generator
232
from numpy.typing import NDArray, ArrayLike
233
from sklearn.base import BaseEstimator, RegressorMixin, ClassifierMixin
234
from sklearn.model_selection import BaseCrossValidator
235
236
# Common parameter types
237
ConfidenceLevel = Union[float, List[float]]
238
ConformityScore = Union[str, object]
239
CVStrategy = Union[int, BaseCrossValidator]
240
PredictionIntervals = Tuple[NDArray, NDArray]
241
PredictionSets = Tuple[NDArray, NDArray]
242
```