CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/elasticsearch-relevance-tests

Author search-engine relevance regression tests for Elasticsearch, OpenSearch, and Apache Solr. Core workflow on the Elasticsearch Ranking Evaluation API (`POST {index}/_rank_eval`) - judgment lists (query + expected docs at ranks), per-query metrics (Precision@K, Recall@K, MRR, DCG, ERR), reproducible test corpora; pair with Quepid + Splainer for interactive judgment authoring. Per-engine references cover the OpenSearch delta (Search Relevance Workbench, neural query DSL, hybrid BM25 + neural pipelines, ES-to-OS migration parity) and the Apache Solr delta (no _rank_eval: debugQuery score explain, LTR feature/model store REST, eDisMax qf/pf/mm tuning, external nDCG harness). Use before changing analyzers, synonyms, boosts, or query templates on an Elasticsearch, OpenSearch, or Solr index that serves user-facing search, so the NDCG / MRR baseline is captured first.

74

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

rank-eval-guide.mdreferences/

Rank Eval guide - binary thresholds, snapshot corpora, judgment tooling

Supplementary detail for elasticsearch-relevance-tests. The core _rank_eval workflow (judgments, metrics, request, test wrapper, per-query regression) stays in SKILL.md; this file holds the deeper configuration and tooling.

relevant_rating_threshold for binary metrics

Per the Elasticsearch Rank Eval API: Precision/Recall/MRR accept relevant_rating_threshold (default 1). For graded judgments:

"metric": {
  "precision": {
    "k": 10,
    "relevant_rating_threshold": 2,
    "ignore_unlabeled": false
  }
}

Rating >= 2 counted as "relevant"; below counted as "not relevant". The ignore_unlabeled flag controls whether unrated docs in results count against precision.

Reproducible test corpus

Snapshot the index state used for tests:

PUT _snapshot/test_repo/baseline_2026_05_06
{
  "indices": "products",
  "include_global_state": false
}

Restore for each CI run:

- name: Restore index snapshot
  run: |
    curl -X POST localhost:9200/_snapshot/test_repo/baseline_2026_05_06/_restore

Otherwise document changes (new docs, re-indexes) silently shift relevance baselines.

Quepid + Splainer integration

Quepid (open source from OpenSource Connections) provides:

  • Web UI for judges to rate per-query results
  • CSV export -> SKILL.md Step 1 judgment list
  • "Try" tab to test query template changes against current judgments

Splainer explains why a doc ranked where it did - invaluable for debugging unexpected results.

references

opensearch.md

rank-eval-guide.md

solr.md

SKILL.md

tile.json