0
# Elasticsearch Rank Eval Plugin
1

2
The Elasticsearch Rank Eval plugin provides comprehensive ranking evaluation capabilities for search quality assessment. It implements multiple information retrieval metrics including Precision@K, Recall@K, Mean Reciprocal Rank (MRR), Discounted Cumulative Gain (DCG), and Expected Reciprocal Rank (ERR) to evaluate the quality of ranked search results against manually rated document sets.
3

4
## Package Information
5

6
- **Package Name**: org.elasticsearch.plugin/rank-eval-client
7
- **Package Type**: maven
8
- **Language**: Java
9
- **Minimum Java Version**: 11
10
- **Elasticsearch Version**: 7.17.29
11
- **Installation**: 
12
  ```xml
13
  <dependency>
14
      <groupId>org.elasticsearch.plugin</groupId>
15
      <artifactId>rank-eval-client</artifactId>
16
      <version>7.17.29</version>
17
  </dependency>
18
  ```
19

20
## Core Imports
21

22
```java
23
// Main request/response classes
24
import org.elasticsearch.index.rankeval.RankEvalRequest;
25
import org.elasticsearch.index.rankeval.RankEvalResponse;
26
import org.elasticsearch.index.rankeval.RankEvalSpec;
27
import org.elasticsearch.index.rankeval.RankEvalAction;
28

29
// Evaluation metrics
30
import org.elasticsearch.index.rankeval.PrecisionAtK;
31
import org.elasticsearch.index.rankeval.RecallAtK;
32
import org.elasticsearch.index.rankeval.MeanReciprocalRank;
33
import org.elasticsearch.index.rankeval.DiscountedCumulativeGain;
34
import org.elasticsearch.index.rankeval.ExpectedReciprocalRank;
35

36
// Supporting classes
37
import org.elasticsearch.index.rankeval.RatedRequest;
38
import org.elasticsearch.index.rankeval.RatedDocument;
39
import org.elasticsearch.index.rankeval.EvalQueryQuality;
40
```
41

42
## Basic Usage
43

44
```java
45
// Create evaluation metric
46
PrecisionAtK metric = new PrecisionAtK(1, false, 10);
47

48
// Create rated documents for a query
49
List<RatedDocument> ratedDocs = Arrays.asList(
50
    new RatedDocument("test_index", "doc1", 2),
51
    new RatedDocument("test_index", "doc2", 1),
52
    new RatedDocument("test_index", "doc3", 0)
53
);
54

55
// Create search request
56
SearchSourceBuilder searchSource = SearchSourceBuilder.searchSource()
57
    .query(QueryBuilders.matchQuery("title", "elasticsearch"));
58

59
// Create rated request
60
RatedRequest ratedRequest = new RatedRequest(
61
    "test_query", 
62
    Arrays.asList("test_index"), 
63
    searchSource, 
64
    ratedDocs
65
);
66

67
// Create evaluation specification
68
RankEvalSpec spec = new RankEvalSpec(
69
    Arrays.asList(ratedRequest), 
70
    metric
71
);
72

73
// Create and execute rank evaluation request
74
RankEvalRequest request = new RankEvalRequest(spec, new String[]{"test_index"});
75
RankEvalResponse response = client.execute(RankEvalAction.INSTANCE, request).get();
76

77
// Access results
78
double overallScore = response.getMetricScore();
79
Map<String, EvalQueryQuality> queryResults = response.getPartialResults();
80
```
81

82
## Capabilities
83

84
### Request and Response API
85

86
The core API for creating and executing ranking evaluation requests, including request builders, response handling, and evaluation specifications.
87

88
```java { .api }
89
// Main request class
90
public class RankEvalRequest extends ActionRequest implements IndicesRequest.Replaceable {
91
    public RankEvalRequest(RankEvalSpec rankingEvaluationSpec, String[] indices);
92
    public RankEvalSpec getRankEvalSpec();
93
    public void setRankEvalSpec(RankEvalSpec task);
94
}
95

96
// Main response class  
97
public class RankEvalResponse extends ActionResponse implements ToXContentObject {
98
    public double getMetricScore();
99
    public Map<String, EvalQueryQuality> getPartialResults();
100
    public Map<String, Exception> getFailures();
101
}
102

103
// Evaluation specification
104
public class RankEvalSpec implements Writeable, ToXContentObject {
105
    public RankEvalSpec(List<RatedRequest> ratedRequests, EvaluationMetric metric);
106
    public EvaluationMetric getMetric();
107
    public List<RatedRequest> getRatedRequests();
108
}
109
```
110

111
[Request and Response API Documentation](./request-response-api.md)
112

113
### Evaluation Metrics
114

115
Implementation of information retrieval metrics for ranking quality assessment, including Precision@K, Recall@K, MRR, DCG, and ERR with configurable parameters.
116

117
```java { .api }
118
// Base interface for all metrics
119
public interface EvaluationMetric extends ToXContentObject, NamedWriteable {
120
    EvalQueryQuality evaluate(String taskId, SearchHit[] hits, List<RatedDocument> ratedDocs);
121
    default double combine(Collection<EvalQueryQuality> partialResults);
122
}
123

124
// Precision@K implementation
125
public class PrecisionAtK implements EvaluationMetric {
126
    public static final String NAME = "precision";
127
    public PrecisionAtK(int relevantRatingThreshold, boolean ignoreUnlabeled, int k);
128
}
129

130
// Recall@K implementation  
131
public class RecallAtK implements EvaluationMetric {
132
    public static final String NAME = "recall";
133
    public RecallAtK(int relevantRatingThreshold, int k);
134
}
135
```
136

137
[Evaluation Metrics Documentation](./evaluation-metrics.md)
138

139
### Configuration and Utilities
140

141
Supporting classes for document rating, search hit handling, query quality evaluation, and configuration management.
142

143
```java { .api }
144
// Document with rating
145
public class RatedDocument implements Writeable, ToXContentObject {
146
    public RatedDocument(String index, String id, int rating);
147
    public DocumentKey getKey();
148
    public int getRating();
149
}
150

151
// Query with rated documents
152
public class RatedRequest implements Writeable, ToXContentObject {
153
    public RatedRequest(String id, List<String> indices, SearchSourceBuilder request, List<RatedDocument> ratedDocs);
154
    public String getId();
155
    public List<RatedDocument> getRatedDocs();
156
}
157

158
// Query evaluation result
159
public class EvalQueryQuality implements ToXContentFragment, Writeable {
160
    public String getId();
161
    public double metricScore();
162
    public MetricDetail getMetricDetails();
163
}
164
```
165

166
[Configuration and Utilities Documentation](./configuration-utilities.md)
167

168
## Architecture
169

170
The Rank Eval plugin follows Elasticsearch's plugin architecture with these key components:
171

172
- **Plugin Registration**: `RankEvalPlugin` registers actions and REST handlers
173
- **Transport Layer**: `TransportRankEvalAction` handles distributed execution
174
- **REST API**: `RestRankEvalAction` provides HTTP endpoints at `/_rank_eval`
175
- **Metric System**: Pluggable evaluation metrics implementing `EvaluationMetric` interface
176
- **Request Processing**: Evaluation requests processed across cluster nodes with result aggregation
177

178
The plugin supports both individual query evaluation and batch evaluation with template-based queries for systematic ranking quality assessment.