0
# Emerging Scores
1

2
Experimental scoring methods under development and peer review. These metrics represent cutting-edge approaches to forecast verification that may become standard methods in future versions.
3

4
## Capabilities
5

6
### Risk Matrix Score
7

8
A decision-oriented scoring framework designed for risk-based evaluation of forecasts with asymmetric loss functions.
9

10
```python { .api }
11
def risk_matrix_score(
12
    fcst: XarrayLike,
13
    obs: XarrayLike,
14
    risk_matrix: Union[Sequence[Sequence[float]], np.ndarray],
15
    *,
16
    fcst_bins: Optional[Sequence[float]] = None,
17
    obs_bins: Optional[Sequence[float]] = None,
18
    reduce_dims: Optional[FlexibleDimensionTypes] = None,
19
    preserve_dims: Optional[FlexibleDimensionTypes] = None,
20
    weights: Optional[xr.DataArray] = None,
21
) -> XarrayLike:
22
    """
23
    Calculate risk matrix score for decision-oriented forecast evaluation.
24

25
    Args:
26
        fcst: Forecast values
27
        obs: Observation values
28
        risk_matrix: 2D matrix defining loss/risk values for each forecast-observation bin combination
29
        fcst_bins: Bin edges for discretizing forecast values
30
        obs_bins: Bin edges for discretizing observation values
31
        reduce_dims: Dimensions to reduce
32
        preserve_dims: Dimensions to preserve
33
        weights: Optional weights
34

35
    Returns:
36
        Risk matrix scores
37

38
    Notes:
39
        - Designed for asymmetric loss situations
40
        - Risk matrix defines penalties for different forecast-observation combinations
41
        - Higher scores indicate greater risk/loss
42
        - Useful for decision-making contexts where different errors have different consequences
43
        - Under peer review - interface may change in future versions
44
    """
45
```
46

47
### Matrix Utilities
48

49
Helper functions for creating and manipulating risk matrices.
50

51
#### Matrix Weights to Array
52

53
Converts matrix weight specifications to array format for computational efficiency.
54

55
```python { .api }
56
def matrix_weights_to_array(
57
    matrix_weights: Union[dict, Sequence[Sequence[float]]],
58
    fcst_categories: Optional[Sequence[str]] = None,
59
    obs_categories: Optional[Sequence[str]] = None,
60
) -> np.ndarray:
61
    """
62
    Convert matrix weights to array format.
63

64
    Args:
65
        matrix_weights: Weight matrix as dict or nested sequences
66
        fcst_categories: Forecast category labels
67
        obs_categories: Observation category labels
68

69
    Returns:
70
        Numpy array representation of weights matrix
71

72
    Notes:
73
        - Standardizes weight matrix format for internal use
74
        - Supports both dictionary and array input formats
75
        - Used internally by risk matrix scoring functions
76
    """
77
```
78

79
#### Warning Scaling Weights
80

81
Creates weight matrices based on warning system scaling parameters.
82

83
```python { .api }
84
def weights_from_warning_scaling(
85
    scaling_factors: Sequence[float],
86
    n_categories: int,
87
    *,
88
    symmetric: bool = False,
89
) -> np.ndarray:
90
    """
91
    Create weight matrix from warning scaling parameters.
92

93
    Args:
94
        scaling_factors: Scaling factors for different warning levels
95
        n_categories: Number of forecast/observation categories
96
        symmetric: Whether to apply symmetric weighting
97

98
    Returns:
99
        Weight matrix for risk-based scoring
100

101
    Notes:
102
        - Generates matrices suitable for warning system evaluation
103
        - Scaling factors control penalty severity for different categories
104
        - Symmetric option creates balanced penalty structure
105
        - Used for meteorological warning verification
106
    """
107
```
108

109
## Usage Patterns
110

111
### Basic Risk Matrix Evaluation
112

113
```python
114
from scores.emerging import risk_matrix_score, matrix_weights_to_array
115
import numpy as np
116

117
# Create sample forecast and observation data
118
forecast = np.random.normal(10, 3, 1000)
119
observation = np.random.normal(10, 2, 1000)
120

121
# Define a simple 3x3 risk matrix for low/medium/high categories
122
# Rows = forecast categories, Columns = observation categories
123
risk_matrix = [
124
    [0.0, 0.5, 2.0],  # Low forecast: small penalty for missing medium/high events
125
    [0.3, 0.0, 1.5],  # Medium forecast: balanced penalties
126
    [1.0, 0.8, 0.0]   # High forecast: large penalty for false alarms
127
]
128

129
# Define bin edges for categorization
130
bins = [5, 12, 15]  # Creates categories: <5, 5-12, 12-15, >15
131

132
# Calculate risk matrix score
133
rms = risk_matrix_score(
134
    forecast, observation,
135
    risk_matrix=risk_matrix,
136
    fcst_bins=bins,
137
    obs_bins=bins
138
)
139

140
print(f"Risk Matrix Score: {rms.values:.3f}")
141
```
142

143
### Weather Warning System Application
144

145
```python
146
from scores.emerging import risk_matrix_score, weights_from_warning_scaling
147

148
# Precipitation forecast evaluation for warning systems
149
precip_forecast = np.random.exponential(3, 500)
150
precip_observed = np.random.exponential(3, 500)
151

152
# Create warning-based weight matrix
153
# Categories: No warning, Advisory, Watch, Warning
154
n_categories = 4
155
scaling_factors = [1.0, 2.0, 4.0, 8.0]  # Increasing penalties for higher warnings
156

157
warning_weights = weights_from_warning_scaling(
158
    scaling_factors,
159
    n_categories,
160
    symmetric=False  # Asymmetric penalties
161
)
162

163
print("Warning Weight Matrix:")
164
print(warning_weights)
165

166
# Define precipitation warning thresholds
167
warning_thresholds = [1.0, 10.0, 25.0]  # mm/hr thresholds
168

169
# Evaluate warning system performance
170
warning_score = risk_matrix_score(
171
    precip_forecast, precip_observed,
172
    risk_matrix=warning_weights,
173
    fcst_bins=warning_thresholds,
174
    obs_bins=warning_thresholds
175
)
176

177
print(f"Warning System Risk Score: {warning_score.values:.3f}")
178
```
179

180
### Decision-Oriented Evaluation
181

182
```python
183
# Economic loss-based evaluation for agricultural applications
184

185
# Temperature forecasts for frost warning
186
temp_forecast = np.random.normal(2, 4, 300)  # Temperature in °C
187
temp_observed = np.random.normal(2, 3, 300)
188

189
# Economic loss matrix (in thousands of dollars)
190
# Categories: Safe (>5°C), Caution (0-5°C), Frost (<0°C)
191
economic_loss_matrix = [
192
    [0,   5,  20],  # Safe forecast: minimal loss if wrong
193
    [2,   0,  15],  # Caution forecast: moderate false alarm cost
194
    [10,  8,   0]   # Frost forecast: high false alarm but necessary protection
195
]
196

197
frost_thresholds = [0, 5]  # Temperature thresholds in °C
198

199
# Calculate economic risk score
200
economic_score = risk_matrix_score(
201
    temp_forecast, temp_observed,
202
    risk_matrix=economic_loss_matrix,
203
    fcst_bins=frost_thresholds,
204
    obs_bins=frost_thresholds
205
)
206

207
print(f"Economic Risk Score: {economic_score.values:.1f} (k$)")
208

209
# Compare to symmetric scoring (traditional approach)
210
symmetric_weights = weights_from_warning_scaling(
211
    [1, 1, 1],  # Equal penalties
212
    3,
213
    symmetric=True
214
)
215

216
symmetric_score = risk_matrix_score(
217
    temp_forecast, temp_observed,
218
    risk_matrix=symmetric_weights,
219
    fcst_bins=frost_thresholds,
220
    obs_bins=frost_thresholds
221
)
222

223
print(f"Symmetric Score: {symmetric_score.values:.3f}")
224
print(f"Asymmetric advantage: {(symmetric_score - economic_score).values:.3f}")
225
```
226

227
### Advanced Risk Matrix Design
228

229
```python
230
# Custom risk matrix for multi-dimensional evaluation
231

232
# Create complex weight matrix with specific penalty structure
233
def create_custom_risk_matrix(n_cats, penalty_type='conservative'):
234
    """Create custom risk matrices for different evaluation contexts."""
235

236
    matrix = np.zeros((n_cats, n_cats))
237

238
    if penalty_type == 'conservative':
239
        # Heavy penalty for missing events, lighter for false alarms
240
        for i in range(n_cats):
241
            for j in range(n_cats):
242
                if i < j:  # Under-forecasting
243
                    matrix[i, j] = (j - i) * 2.0
244
                elif i > j:  # Over-forecasting
245
                    matrix[i, j] = (i - j) * 1.0
246
                else:  # Perfect forecast
247
                    matrix[i, j] = 0.0
248

249
    elif penalty_type == 'balanced':
250
        # Equal penalties for under- and over-forecasting
251
        for i in range(n_cats):
252
            for j in range(n_cats):
253
                matrix[i, j] = abs(i - j)
254

255
    return matrix
256

257
# Test different penalty structures
258
conservative_matrix = create_custom_risk_matrix(4, 'conservative')
259
balanced_matrix = create_custom_risk_matrix(4, 'balanced')
260

261
# Multi-threshold evaluation
262
thresholds = [2, 8, 15, 25]
263

264
conservative_score = risk_matrix_score(
265
    precip_forecast, precip_observed,
266
    risk_matrix=conservative_matrix,
267
    fcst_bins=thresholds,
268
    obs_bins=thresholds
269
)
270

271
balanced_score = risk_matrix_score(
272
    precip_forecast, precip_observed,
273
    risk_matrix=balanced_matrix,
274
    fcst_bins=thresholds,
275
    obs_bins=thresholds
276
)
277

278
print(f"Conservative penalty score: {conservative_score.values:.3f}")
279
print(f"Balanced penalty score: {balanced_score.values:.3f}")
280
```
281

282
### Multi-dimensional Risk Assessment
283

284
```python
285
# Evaluate risk across multiple dimensions
286
forecast_3d = xr.DataArray(
287
    np.random.exponential(2, (50, 10, 15)),  # time, lat, lon
288
    dims=["time", "lat", "lon"]
289
)
290
observation_3d = xr.DataArray(
291
    np.random.exponential(2, (50, 10, 15)),
292
    dims=["time", "lat", "lon"]
293
)
294

295
# Simple 3-category risk matrix
296
simple_risk = [
297
    [0, 1, 3],
298
    [1, 0, 2],
299
    [2, 1, 0]
300
]
301

302
# Risk assessment at each grid point (temporal aggregation)
303
spatial_risk = risk_matrix_score(
304
    forecast_3d, observation_3d,
305
    risk_matrix=simple_risk,
306
    fcst_bins=[1, 5],
307
    obs_bins=[1, 5],
308
    reduce_dims="time"
309
)
310

311
# Overall risk assessment
312
total_risk = risk_matrix_score(
313
    forecast_3d, observation_3d,
314
    risk_matrix=simple_risk,
315
    fcst_bins=[1, 5],
316
    obs_bins=[1, 5],
317
    reduce_dims=["time", "lat", "lon"]
318
)
319

320
print(f"Spatial risk assessment shape: {spatial_risk.shape}")
321
print(f"Total risk score: {total_risk.values:.3f}")
322
print(f"Highest risk location: {spatial_risk.max().values:.3f}")
323
print(f"Lowest risk location: {spatial_risk.min().values:.3f}")
324
```
325

326
## Important Notes
327

328
### Experimental Status
329

330
**⚠️ Warning**: The functions in this module are under active development and peer review. The API may change in future versions of the scores package.
331

332
### Limitations
333

334
1. **Performance**: Risk matrix calculations can be computationally intensive for large datasets
335
2. **Matrix Design**: Requires careful consideration of penalty structure and loss functions
336
3. **Category Definition**: Bin edge selection significantly impacts results
337
4. **Validation**: Limited validation compared to established scoring methods
338

339
### Best Practices
340

341
1. **Matrix Design**: Ensure risk matrix reflects actual decision costs/consequences
342
2. **Threshold Selection**: Use domain expertise to define meaningful category boundaries
343
3. **Validation**: Compare results with traditional scoring methods for context
344
4. **Documentation**: Document matrix design rationale for reproducibility
345

346
### Future Development
347

348
These emerging methods may be integrated into the core scoring modules in future releases. Users should expect potential API changes and should validate results against established methods during the experimental phase.