0
# Metrics and Loss Functions
1
2
Comprehensive metrics for evaluating model performance and loss functions for training across different domains and tasks including classification, regression, segmentation, and language modeling.
3
4
## Capabilities
5
6
### Classification Metrics
7
8
Metrics for evaluating classification model performance across binary and multi-class scenarios.
9
10
```python { .api }
11
def accuracy(inp, targ, axis=-1):
12
"""
13
Classification accuracy metric.
14
15
Parameters:
16
- inp: Model predictions (logits or probabilities)
17
- targ: Target labels
18
- axis: Axis for prediction classes
19
20
Returns:
21
- Accuracy as a tensor
22
"""
23
24
def error_rate(inp, targ, axis=-1):
25
"""
26
Classification error rate (1 - accuracy).
27
28
Parameters:
29
- inp: Model predictions
30
- targ: Target labels
31
- axis: Axis for prediction classes
32
33
Returns:
34
- Error rate as a tensor
35
"""
36
37
def top_k_accuracy(inp, targ, k=5, axis=-1):
38
"""
39
Top-k accuracy for multi-class classification.
40
41
Parameters:
42
- inp: Model predictions
43
- targ: Target labels
44
- k: Number of top predictions to consider
45
- axis: Axis for prediction classes
46
47
Returns:
48
- Top-k accuracy as a tensor
49
"""
50
51
class Precision(Metric):
52
"""Precision metric for classification."""
53
54
def __init__(self, axis=1, pos_label=1, average='binary'): ...
55
56
def reset(self): ...
57
def accumulate(self, learn): ...
58
@property
59
def value(self): ...
60
61
class Recall(Metric):
62
"""Recall metric for classification."""
63
64
def __init__(self, axis=1, pos_label=1, average='binary'): ...
65
66
def reset(self): ...
67
def accumulate(self, learn): ...
68
@property
69
def value(self): ...
70
71
class F1Score(Metric):
72
"""F1 score combining precision and recall."""
73
74
def __init__(self, axis=1, pos_label=1, average='binary'): ...
75
76
def reset(self): ...
77
def accumulate(self, learn): ...
78
@property
79
def value(self): ...
80
81
class MatthewsCorrCoef(Metric):
82
"""Matthews Correlation Coefficient for binary classification."""
83
84
def __init__(self, axis=1): ...
85
86
def reset(self): ...
87
def accumulate(self, learn): ...
88
@property
89
def value(self): ...
90
91
class RocAuc(Metric):
92
"""Area Under the ROC Curve."""
93
94
def __init__(self, axis=1): ...
95
96
def reset(self): ...
97
def accumulate(self, learn): ...
98
@property
99
def value(self): ...
100
```
101
102
### Regression Metrics
103
104
Metrics for evaluating regression model performance and continuous value predictions.
105
106
```python { .api }
107
def rmse(inp, targ):
108
"""
109
Root Mean Square Error.
110
111
Parameters:
112
- inp: Model predictions
113
- targ: Target values
114
115
Returns:
116
- RMSE as a tensor
117
"""
118
119
def mae(inp, targ):
120
"""
121
Mean Absolute Error.
122
123
Parameters:
124
- inp: Model predictions
125
- targ: Target values
126
127
Returns:
128
- MAE as a tensor
129
"""
130
131
def msle(inp, targ):
132
"""
133
Mean Squared Logarithmic Error.
134
135
Parameters:
136
- inp: Model predictions (must be positive)
137
- targ: Target values (must be positive)
138
139
Returns:
140
- MSLE as a tensor
141
"""
142
143
def exp_rmspe(inp, targ):
144
"""
145
Exponential Root Mean Square Percentage Error.
146
147
Parameters:
148
- inp: Model predictions
149
- targ: Target values
150
151
Returns:
152
- Exponential RMSPE as a tensor
153
"""
154
155
def mse(inp, targ):
156
"""Mean Squared Error."""
157
158
def r2_score(inp, targ):
159
"""R-squared coefficient of determination."""
160
161
class ExplainedVariance(Metric):
162
"""Explained variance regression score."""
163
164
def reset(self): ...
165
def accumulate(self, learn): ...
166
@property
167
def value(self): ...
168
```
169
170
### Segmentation Metrics
171
172
Specialized metrics for image segmentation and pixel-level prediction tasks.
173
174
```python { .api }
175
class Dice(Metric):
176
"""
177
Dice coefficient for segmentation.
178
Measures overlap between predicted and target masks.
179
"""
180
181
def __init__(self, axis=1): ...
182
183
def reset(self): ...
184
def accumulate(self, learn): ...
185
@property
186
def value(self): ...
187
188
class JaccardCoeff(Metric):
189
"""
190
Jaccard coefficient (Intersection over Union) for segmentation.
191
"""
192
193
def __init__(self, axis=1): ...
194
195
def reset(self): ...
196
def accumulate(self, learn): ...
197
@property
198
def value(self): ...
199
200
def foreground_acc(inp, targ, bkg_idx=0, axis=1):
201
"""
202
Foreground accuracy ignoring background class.
203
204
Parameters:
205
- inp: Model predictions
206
- targ: Target masks
207
- bkg_idx: Background class index to ignore
208
- axis: Class axis
209
210
Returns:
211
- Foreground accuracy
212
"""
213
214
def Iou(inp, targ, ignore_idx=None):
215
"""Intersection over Union for segmentation."""
216
```
217
218
### Language Model Metrics
219
220
Metrics specialized for language modeling and text generation tasks.
221
222
```python { .api }
223
class Perplexity(Metric):
224
"""
225
Perplexity metric for language models.
226
Measures how well the model predicts the text.
227
"""
228
229
def __init__(self, dim=-1): ...
230
231
def reset(self): ...
232
def accumulate(self, learn): ...
233
@property
234
def value(self):
235
"""Returns perplexity = exp(average_loss)."""
236
237
class BLEU(Metric):
238
"""
239
BLEU score for text generation evaluation.
240
Measures n-gram overlap between generated and reference text.
241
"""
242
243
def __init__(self, n_gram=4, weights=None): ...
244
245
def reset(self): ...
246
def accumulate(self, learn): ...
247
@property
248
def value(self): ...
249
250
def bleu_score(pred_tokens, targ_tokens, n_gram=4):
251
"""Calculate BLEU score for token sequences."""
252
```
253
254
### Loss Functions
255
256
Core loss functions for training models across different tasks and domains.
257
258
```python { .api }
259
class CrossEntropyLossFlat(nn.CrossEntropyLoss):
260
"""
261
Cross-entropy loss with flattened inputs.
262
Handles multi-dimensional inputs by flattening before loss computation.
263
"""
264
265
def __init__(self, weight=None, ignore_index=-100, reduction='mean',
266
label_smoothing=0.0, axis=-1): ...
267
268
def forward(self, input, target): ...
269
270
class BCEWithLogitsLossFlat(nn.BCEWithLogitsLoss):
271
"""Binary cross-entropy with logits, flattened inputs."""
272
273
def __init__(self, weight=None, reduction='mean', pos_weight=None,
274
axis=-1): ...
275
276
class MSELossFlat(nn.MSELoss):
277
"""Mean squared error loss with flattened inputs."""
278
279
def __init__(self, reduction='mean', axis=-1): ...
280
281
class L1LossFlat(nn.L1Loss):
282
"""L1 (mean absolute error) loss with flattened inputs."""
283
284
def __init__(self, reduction='mean', axis=-1): ...
285
286
class FocalLoss(nn.Module):
287
"""
288
Focal loss for handling class imbalance.
289
Focuses learning on hard examples by down-weighting easy examples.
290
"""
291
292
def __init__(self, alpha=1, gamma=2, reduction='mean'):
293
"""
294
Initialize focal loss.
295
296
Parameters:
297
- alpha: Weighting factor for rare class
298
- gamma: Focusing parameter (higher = more focus on hard examples)
299
- reduction: Reduction method ('mean', 'sum', 'none')
300
"""
301
302
def forward(self, input, target): ...
303
304
class LabelSmoothingCrossEntropy(nn.Module):
305
"""
306
Cross-entropy loss with label smoothing regularization.
307
Prevents overfitting by smoothing target distribution.
308
"""
309
310
def __init__(self, eps=0.1, reduction='mean', ignore_index=-100): ...
311
312
def forward(self, input, target): ...
313
314
class DiceLoss(nn.Module):
315
"""
316
Dice loss for segmentation tasks.
317
Optimizes directly for Dice coefficient.
318
"""
319
320
def __init__(self, axis=1, smooth=1e-6): ...
321
322
def forward(self, input, target): ...
323
324
class JaccardLoss(nn.Module):
325
"""Jaccard (IoU) loss for segmentation."""
326
327
def __init__(self, axis=1, smooth=1e-6): ...
328
329
def forward(self, input, target): ...
330
331
class CombinedLoss(nn.Module):
332
"""Combine multiple loss functions with weights."""
333
334
def __init__(self, losses, weights=None): ...
335
336
def forward(self, input, target): ...
337
```
338
339
### Base Metric Classes
340
341
Foundation classes for implementing custom metrics.
342
343
```python { .api }
344
class Metric:
345
"""Base class for metrics that accumulate over batches."""
346
347
def __init__(self): ...
348
349
def reset(self):
350
"""Reset metric state for new epoch."""
351
352
def accumulate(self, learn):
353
"""Accumulate metric for current batch."""
354
355
@property
356
def value(self):
357
"""Get current metric value."""
358
359
@property
360
def name(self):
361
"""Get metric name for display."""
362
363
class AccumMetric(Metric):
364
"""
365
Base class for metrics that accumulate values.
366
Automatically handles accumulation and averaging.
367
"""
368
369
def __init__(self, func, dim_argmax=None, activation=ActivationType.No,
370
thresh=None, to_np=False, invert_arg=False, flatten=True): ...
371
372
class AvgMetric(Metric):
373
"""Average a function across all batches."""
374
375
def __init__(self, func, name=None): ...
376
377
class AvgLoss(Metric):
378
"""Average loss metric."""
379
380
def reset(self): ...
381
def accumulate(self, learn): ...
382
@property
383
def value(self): ...
384
385
class AvgSmoothLoss(Metric):
386
"""Smoothed average loss metric."""
387
388
def __init__(self, beta=0.98): ...
389
390
def reset(self): ...
391
def accumulate(self, learn): ...
392
@property
393
def value(self): ...
394
395
class ValueMetric(Metric):
396
"""Track a single value (like learning rate)."""
397
398
def __init__(self, func, name=None): ...
399
```
400
401
### Metric Utilities
402
403
Utility functions for working with metrics and evaluation.
404
405
```python { .api }
406
def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
407
"""Multi-label accuracy with threshold."""
408
409
def precision_multi(inp, targ, thresh=0.5, sigmoid=True):
410
"""Multi-label precision."""
411
412
def recall_multi(inp, targ, thresh=0.5, sigmoid=True):
413
"""Multi-label recall."""
414
415
def fbeta_multi(inp, targ, beta=2, thresh=0.5, sigmoid=True):
416
"""Multi-label F-beta score."""
417
418
def hamming_loss(inp, targ, thresh=0.5, sigmoid=True):
419
"""Hamming loss for multi-label classification."""
420
421
def confusion_matrix(inp, targ, normalize=None):
422
"""Compute confusion matrix."""
423
424
def classification_report(inp, targ, labels=None):
425
"""Generate classification report with precision, recall, F1."""
426
427
def roc_curve(inp, targ, pos_label=1):
428
"""Compute ROC curve."""
429
430
def auc(fpr, tpr):
431
"""Compute Area Under Curve."""
432
```