0
# Calibration Methods
1

2
Probability calibration methods for improving the reliability of probabilistic predictions, particularly for multi-class classification problems. MAPIE provides top-label calibration techniques that ensure predicted probabilities accurately reflect the true likelihood of predictions.
3

4
## Capabilities
5

6
### Top-Label Calibrator
7

8
Implements top-label calibration for multi-class classification, focusing on calibrating the probability of the most likely predicted class. This approach is particularly effective for scenarios where the confidence in the top prediction is most important.
9

10
```python { .api }
11
class TopLabelCalibrator:
12
    """
13
    Top-label calibration for multi-class classification.
14

15
    Parameters:
16
    - estimator: ClassifierMixin, base multi-class classifier
17
    - calibrator: Union[str, RegressorMixin], calibration method ("sigmoid", "isotonic") or custom regressor
18
    - cv: str, cross-validation strategy ("split", "prefit") (default: "split")
19
    """
20
    def __init__(self, estimator=None, calibrator=None, cv="split"): ...
21

22
    def fit(self, X, y, sample_weight=None, calib_size=0.33, random_state=None, shuffle=True, stratify=None, **fit_params):
23
        """
24
        Fit the classifier and calibrator.
25

26
        Parameters:
27
        - X: ArrayLike, input features
28
        - y: ArrayLike, class labels
29
        - sample_weight: Optional[ArrayLike], sample weights
30
        - calib_size: float, fraction of data for calibration when cv="split" (default: 0.33)
31
        - random_state: Optional[int], random seed for data splitting
32
        - shuffle: bool, whether to shuffle data before splitting (default: True)
33
        - stratify: Optional[ArrayLike], stratification labels (default: None, uses y)
34
        - **fit_params: additional parameters passed to estimator.fit()
35

36
        Returns:
37
        Self
38
        """
39

40
    def predict_proba(self, X):
41
        """
42
        Predict calibrated class probabilities.
43

44
        Parameters:
45
        - X: ArrayLike, test features
46

47
        Returns:
48
        NDArray: calibrated probabilities (shape: n_samples x n_classes)
49
        """
50

51
    def predict(self, X):
52
        """
53
        Predict class labels using calibrated probabilities.
54

55
        Parameters:
56
        - X: ArrayLike, test features
57

58
        Returns:
59
        NDArray: predicted class labels
60
        """
61

62
    # Key attributes after fitting
63
    classes_: NDArray  # Array with class names
64
    n_classes_: int  # Number of classes
65
    single_estimator_: ClassifierMixin  # Fitted base classifier
66
    calibrators: Dict[Union[int, str], RegressorMixin]  # Fitted calibrators per class
67
```
68

69
## Usage Examples
70

71
### Basic Top-Label Calibration
72

73
```python
74
from mapie.calibration import TopLabelCalibrator
75
from sklearn.ensemble import RandomForestClassifier
76
from sklearn.model_selection import train_test_split
77
import numpy as np
78

79
# Prepare multi-class data
80
X, y = load_multiclass_data()
81
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)
82

83
# Create calibrated classifier
84
calibrated_clf = TopLabelCalibrator(
85
    estimator=RandomForestClassifier(n_estimators=100, random_state=42),
86
    calibrator="sigmoid",  # Platt scaling
87
    cv="split"
88
)
89

90
# Fit with automatic calibration split
91
calibrated_clf.fit(X_train, y_train, calib_size=0.3, random_state=42)
92

93
# Get calibrated probabilities
94
y_proba_calibrated = calibrated_clf.predict_proba(X_test)
95
y_pred_calibrated = calibrated_clf.predict(X_test)
96
```
97

98
### Isotonic Calibration
99

100
```python
101
# Use isotonic regression for non-parametric calibration
102
calibrated_clf = TopLabelCalibrator(
103
    estimator=LogisticRegression(max_iter=1000),
104
    calibrator="isotonic",  # Non-parametric calibration
105
    cv="split"
106
)
107

108
# Fit with stratified splitting
109
calibrated_clf.fit(
110
    X_train, y_train,
111
    calib_size=0.4,
112
    stratify=y_train,  # Ensure balanced calibration set
113
    random_state=42
114
)
115

116
# Compare raw vs calibrated probabilities
117
raw_clf = LogisticRegression(max_iter=1000)
118
raw_clf.fit(X_train, y_train)
119

120
y_proba_raw = raw_clf.predict_proba(X_test)
121
y_proba_calibrated = calibrated_clf.predict_proba(X_test)
122
```
123

124
### Pre-fitted Estimator Calibration
125

126
```python
127
# Use pre-fitted estimator
128
base_clf = RandomForestClassifier(n_estimators=50)
129
base_clf.fit(X_train, y_train)
130

131
# Calibrate pre-fitted estimator
132
calibrated_clf = TopLabelCalibrator(
133
    estimator=base_clf,
134
    calibrator="sigmoid",
135
    cv="prefit"  # Use pre-fitted estimator
136
)
137

138
# Fit calibrator only (estimator already fitted)
139
calibrated_clf.fit(X_calib, y_calib)  # Separate calibration data
140
```
141

142
### Custom Calibration Regressor
143

144
```python
145
from sklearn.linear_model import Ridge
146

147
# Use custom regressor for calibration
148
custom_calibrator = Ridge(alpha=1.0)
149

150
calibrated_clf = TopLabelCalibrator(
151
    estimator=RandomForestClassifier(),
152
    calibrator=custom_calibrator,
153
    cv="split"
154
)
155

156
calibrated_clf.fit(X_train, y_train)
157
```
158

159
## Calibration Methods
160

161
### Sigmoid Calibration (Platt Scaling)
162

163
Fits a sigmoid function to map predicted probabilities to calibrated probabilities. Assumes calibration curve has sigmoid shape.
164

165
```python
166
calibrator="sigmoid"
167
```
168

169
**Mathematical Form:**
170
```
171
P_calibrated = 1 / (1 + exp(A * P_raw + B))
172
```
173

174
**Advantages:**
175
- Parametric method with interpretable parameters
176
- Works well when calibration curve is sigmoid-shaped
177
- Requires relatively few calibration samples
178

179
**Best for:**
180
- Small calibration datasets
181
- When the miscalibration follows a sigmoid pattern
182
- Naive Bayes and SVM classifiers
183

184
### Isotonic Calibration
185

186
Non-parametric method that fits a monotonic function to map probabilities to calibrated values.
187

188
```python
189
calibrator="isotonic"
190
```
191

192
**Advantages:**
193
- Non-parametric, no assumptions about calibration curve shape
194
- More flexible than sigmoid calibration
195
- Can handle complex calibration patterns
196

197
**Best for:**
198
- Larger calibration datasets
199
- Tree-based models (Random Forest, Gradient Boosting)
200
- When calibration curve is not sigmoid-shaped
201

202
## Advanced Usage
203

204
### Analyzing Calibration Quality
205

206
```python
207
from sklearn.calibration import calibration_curve
208
import matplotlib.pyplot as plt
209

210
# Compute calibration curves
211
fraction_pos_raw, mean_pred_raw = calibration_curve(
212
    y_test, y_proba_raw[:, 1], n_bins=10
213
)
214

215
fraction_pos_cal, mean_pred_cal = calibration_curve(
216
    y_test, y_proba_calibrated[:, 1], n_bins=10
217
)
218

219
# Plot calibration curves
220
plt.figure(figsize=(10, 6))
221
plt.plot([0, 1], [0, 1], 'k--', label='Perfect calibration')
222
plt.plot(mean_pred_raw, fraction_pos_raw, 's-', label='Raw probabilities')
223
plt.plot(mean_pred_cal, fraction_pos_cal, 's-', label='Calibrated probabilities')
224
plt.xlabel('Mean predicted probability')
225
plt.ylabel('Fraction of positives')
226
plt.legend()
227
plt.title('Calibration Plot')
228
plt.show()
229
```
230

231
### Using Calibration Metrics
232

233
```python
234
from mapie.metrics.calibration import (
235
    expected_calibration_error,
236
    top_label_ece,
237
    kolmogorov_smirnov_statistic
238
)
239

240
# Expected Calibration Error (ECE)
241
ece_raw = expected_calibration_error(y_test, y_proba_raw[:, 1])
242
ece_calibrated = expected_calibration_error(y_test, y_proba_calibrated[:, 1])
243

244
print(f"ECE before calibration: {ece_raw:.4f}")
245
print(f"ECE after calibration: {ece_calibrated:.4f}")
246

247
# Top-label ECE for multi-class
248
top_ece_raw = top_label_ece(y_test, y_proba_raw)
249
top_ece_calibrated = top_label_ece(y_test, y_proba_calibrated)
250

251
print(f"Top-label ECE before: {top_ece_raw:.4f}")
252
print(f"Top-label ECE after: {top_ece_calibrated:.4f}")
253

254
# Kolmogorov-Smirnov test
255
ks_stat_raw = kolmogorov_smirnov_statistic(y_test, y_proba_raw[:, 1])
256
ks_stat_calibrated = kolmogorov_smirnov_statistic(y_test, y_proba_calibrated[:, 1])
257
```
258

259
### Multi-Class Calibration Analysis
260

261
```python
262
# Analyze per-class calibration
263
n_classes = len(calibrated_clf.classes_)
264

265
plt.figure(figsize=(15, 5))
266
for i in range(n_classes):
267
    plt.subplot(1, n_classes, i+1)
268

269
    # Binary indicator for class i
270
    y_binary = (y_test == calibrated_clf.classes_[i]).astype(int)
271

272
    # Calibration curve for class i
273
    fraction_pos, mean_pred = calibration_curve(
274
        y_binary, y_proba_calibrated[:, i], n_bins=10
275
    )
276

277
    plt.plot([0, 1], [0, 1], 'k--', alpha=0.5)
278
    plt.plot(mean_pred, fraction_pos, 's-')
279
    plt.xlabel(f'Mean predicted prob (Class {calibrated_clf.classes_[i]})')
280
    plt.ylabel('Fraction of positives')
281
    plt.title(f'Calibration - Class {calibrated_clf.classes_[i]}')
282

283
plt.tight_layout()
284
plt.show()
285
```
286

287
### Sample Weight Support
288

289
```python
290
# Use sample weights during fitting
291
sample_weights = compute_sample_weight('balanced', y_train)
292

293
calibrated_clf = TopLabelCalibrator(
294
    estimator=RandomForestClassifier(),
295
    calibrator="sigmoid"
296
)
297

298
# Pass sample weights to fit
299
calibrated_clf.fit(
300
    X_train, y_train,
301
    sample_weight=sample_weights,
302
    calib_size=0.3
303
)
304
```
305

306
## Best Practices
307

308
### Choosing Calibration Method
309

310
- **Use sigmoid** when:
311
  - Small calibration dataset (< 1000 samples)
312
  - Base classifier is Naive Bayes or SVM
313
  - Calibration curve appears sigmoid-shaped
314

315
- **Use isotonic** when:
316
  - Larger calibration dataset (> 1000 samples)
317
  - Base classifier is tree-based (Random Forest, XGBoost)
318
  - Calibration curve has complex shape
319

320
### Calibration Set Size
321

322
```python
323
# Rule of thumb: 20-40% for calibration
324
calib_sizes = [0.1, 0.2, 0.3, 0.4, 0.5]
325
ece_scores = []
326

327
for calib_size in calib_sizes:
328
    clf = TopLabelCalibrator(estimator=RandomForestClassifier())
329
    clf.fit(X_train, y_train, calib_size=calib_size)
330
    y_proba = clf.predict_proba(X_test)
331
    ece = expected_calibration_error(y_test, y_proba[:, 1])
332
    ece_scores.append(ece)
333

334
# Find optimal calibration set size
335
optimal_size = calib_sizes[np.argmin(ece_scores)]
336
print(f"Optimal calibration size: {optimal_size}")
337
```
338

339
### Cross-Validation for Calibration
340

341
```python
342
from sklearn.model_selection import cross_val_score
343

344
# Evaluate calibration with cross-validation
345
def calibration_score(estimator, X, y):
346
    """Custom scoring function for calibration quality."""
347
    y_proba = estimator.predict_proba(X)
348
    return -expected_calibration_error(y, y_proba[:, 1])  # Negative ECE
349

350
calibrated_clf = TopLabelCalibrator(
351
    estimator=RandomForestClassifier(),
352
    calibrator="isotonic"
353
)
354

355
scores = cross_val_score(
356
    calibrated_clf, X, y,
357
    cv=5,
358
    scoring=calibration_score
359
)
360

361
print(f"Average calibration score: {np.mean(scores):.4f} ± {np.std(scores):.4f}")
362
```