0
# Scores
1

2
A comprehensive Python library containing mathematical functions for the verification, evaluation and optimization of forecasts, predictions or models. The package provides over 60 functions specifically designed for scientific applications in meteorology, climatology and oceanography, built on xarray for handling labeled n-dimensional scientific data with support for NetCDF4, HDF5, Zarr and GRIB formats.
3

4
## Package Information
5

6
- **Package Name**: scores
7
- **Language**: Python
8
- **Installation**: `pip install scores`
9
- **Python Requirement**: >=3.10
10

11
## Core Imports
12

13
Basic package import:
14

15
```python
16
import scores
17
```
18

19
Module-specific imports:
20

21
```python
22
import scores.continuous
23
import scores.probability
24
import scores.categorical
25
import scores.processing
26
import scores.sample_data
27
import scores.emerging
28
# Note: spatial must be imported directly
29
from scores import spatial
30
```
31

32
Direct function imports (recommended):
33

34
```python
35
from scores.continuous import mse, rmse, mae, kge, nse
36
from scores.probability import brier_score, crps_for_ensemble
37
from scores.categorical import probability_of_detection
38
from scores.spatial import fss_2d
39
```
40

41
## Basic Usage
42

43
```python
44
import xarray as xr
45
from scores.continuous import mse, rmse, mae
46
from scores.sample_data import simple_forecast, simple_observations
47

48
# Generate sample data for demonstration
49
forecast = simple_forecast()
50
observations = simple_observations()
51

52
# Calculate basic continuous scores
53
mse_score = mse(forecast, observations)
54
rmse_score = rmse(forecast, observations)
55
mae_score = mae(forecast, observations)
56

57
print(f"MSE: {mse_score.values}")
58
print(f"RMSE: {rmse_score.values}")
59
print(f"MAE: {mae_score.values}")
60

61
# Calculate scores with dimension reduction
62
forecast_3d = forecast.expand_dims({"station": 5, "time": 10})
63
observations_3d = observations.expand_dims({"station": 5, "time": 10})
64

65
# Reduce over time dimension only
66
temporal_mse = mse(forecast_3d, observations_3d, reduce_dims="time")
67
print(f"Temporal MSE shape: {temporal_mse.shape}")
68

69
# Apply weighting (e.g., latitude weighting for spatial data)
70
from scores.functions import create_latitude_weights
71
import numpy as np
72

73
lats = np.linspace(-90, 90, forecast_3d.sizes["station"])
74
lat_weights = create_latitude_weights(lats)
75
weighted_mse = mse(forecast_3d, observations_3d,
76
                   reduce_dims=["time", "station"],
77
                   weights=lat_weights)
78
print(f"Weighted MSE: {weighted_mse.values}")
79
```
80

81
## Architecture
82

83
The scores package is organized around forecast evaluation paradigms:
84

85
- **Data Structures**: Built on xarray DataArrays and Datasets for labeled n-dimensional scientific data with metadata preservation
86
- **Dimension Handling**: Flexible dimension reduction (`reduce_dims`) and preservation (`preserve_dims`) with automatic broadcasting
87
- **Weighting Support**: Optional weighting for all metrics via `weights` parameter (e.g., area weighting, population weighting)
88
- **Modular Design**: Separate modules for different forecast types (continuous, categorical, probability, spatial)
89
- **Performance**: Uses bottleneck and scipy for optimized computations, supports Dask for scaling
90

91
## Capabilities
92

93
### Continuous Scores
94

95
Standard metrics for evaluating single-valued continuous forecasts including error measures, bias metrics, efficiency indices, and correlation coefficients.
96

97
```python { .api }
98
from scores.continuous import (
99
    mse, rmse, mae, additive_bias, multiplicative_bias, mean_error, pbias,
100
    kge, nse, pearsonr, spearmanr
101
)
102
```
103

104
[Continuous Scores](./continuous-scores.md)
105

106
### Probability Scores
107

108
Metrics for evaluating probability forecasts, ensemble forecasts, and distributional predictions including Brier score, CRPS (Continuous Ranked Probability Score), and their threshold-weighted variants.
109

110
```python { .api }
111
from scores.probability import (
112
    brier_score, brier_score_for_ensemble,
113
    crps_cdf, crps_for_ensemble, crps_cdf_brier_decomposition,
114
    tw_crps_for_ensemble, tail_tw_crps_for_ensemble
115
)
116
```
117

118
[Probability Scores](./probability-scores.md)
119

120
### Categorical Scores
121

122
Verification metrics for categorical and binary forecasts including contingency table statistics, skill scores, and multicategorical measures.
123

124
```python { .api }
125
from scores.categorical import (
126
    probability_of_detection, probability_of_false_detection,
127
    firm, seeps, BinaryContingencyManager, BasicContingencyManager
128
)
129
```
130

131
[Categorical Scores](./categorical-scores.md)
132

133
### Spatial Scores
134

135
Spatial verification methods that account for spatial structure and displacement errors in gridded forecasts.
136

137
```python { .api }
138
from scores.spatial import fss_2d, fss_2d_binary, fss_2d_single_field
139
```
140

141
[Spatial Scores](./spatial-scores.md)
142

143
### Data Processing
144

145
Tools for data preprocessing, discretization, bootstrapping, and CDF manipulation required for forecast verification workflows.
146

147
```python { .api }
148
from scores.processing import (
149
    comparative_discretise, binary_discretise, binary_discretise_proportion,
150
    broadcast_and_match_nan, isotonic_fit, block_bootstrap
151
)
152
```
153

154
[Processing Tools](./processing-tools.md)
155

156
### Plot Data Generation
157

158
Functions for generating data structures needed for diagnostic plots and visualization including Murphy diagrams, Q-Q plots, and ROC curves.
159

160
```python { .api }
161
from scores.plotdata import murphy_score, murphy_thetas, qq, roc
162
```
163

164
[Plot Data](./plot-data.md)
165

166
### Statistical Tests
167

168
Statistical significance testing for forecast comparisons and confidence interval estimation.
169

170
```python { .api }
171
from scores.stats.statistical_tests import diebold_mariano
172
```
173

174
[Statistical Tests](./statistical-tests.md)
175

176
### Sample Data
177

178
Built-in sample data generation for tutorials, testing, and experimentation with various data formats and structures.
179

180
```python { .api }
181
from scores.sample_data import (
182
    simple_forecast, simple_observations,
183
    continuous_forecast, continuous_observations,
184
    cdf_forecast, cdf_observations
185
)
186
```
187

188
[Sample Data](./sample-data.md)
189

190
### Emerging Scores
191

192
Experimental scoring methods under development and peer review, including risk-based evaluation frameworks designed for decision-oriented forecast assessment.
193

194
```python { .api }
195
from scores.emerging import (
196
    risk_matrix_score, matrix_weights_to_array, weights_from_warning_scaling
197
)
198
```
199

200
[Emerging Scores](./emerging-scores.md)
201

202
### Pandas Integration
203

204
Specialized functions for working with pandas Series data, providing simplified APIs for basic continuous metrics without dimensional operations.
205

206
```python { .api }
207
from scores.pandas.continuous import mse, rmse, mae
208
from scores.pandas.typing import PandasType
209
```
210

211
[Pandas Integration](./pandas-integration.md)
212

213
## Common Data Patterns
214

215
### Dimension Handling
216

217
```python { .api }
218
# Standard dimension parameters (used by all scoring functions)
219
reduce_dims: Optional[Iterable[Hashable]] = None     # Dimensions to reduce
220
preserve_dims: Optional[Iterable[Hashable]] = None   # Dimensions to keep
221
weights: Optional[xr.DataArray] = None               # Optional weighting
222
```
223

224
### Type Definitions
225

226
```python { .api }
227
from scores.typing import (
228
    FlexibleDimensionTypes,  # Iterable[Hashable]
229
    XarrayLike,              # Union[xr.DataArray, xr.Dataset]
230
    FlexibleArrayType        # Union[XarrayLike, pd.Series]
231
)
232
```
233

234
### Utility Functions
235

236
```python { .api }
237
from scores.functions import (
238
    apply_weights,
239
    create_latitude_weights,
240
    angular_difference
241
)
242
```
243

244
## Installation Options
245

246
```bash
247
# Basic installation
248
pip install scores
249

250
# Development tools (testing, linting)
251
pip install scores[dev]
252

253
# Tutorial dependencies (jupyter, matplotlib, plotting tools)
254
pip install scores[tutorial]
255

256
# Everything
257
pip install scores[all]
258

259
# Alternative: Using pixi environment manager
260
pixi install
261
```