CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/elasticsearch-relevance-tests

Author search-engine relevance regression tests for Elasticsearch, OpenSearch, and Apache Solr. Core workflow on the Elasticsearch Ranking Evaluation API (`POST {index}/_rank_eval`) - judgment lists (query + expected docs at ranks), per-query metrics (Precision@K, Recall@K, MRR, DCG, ERR), reproducible test corpora; pair with Quepid + Splainer for interactive judgment authoring. Per-engine references cover the OpenSearch delta (Search Relevance Workbench, neural query DSL, hybrid BM25 + neural pipelines, ES-to-OS migration parity) and the Apache Solr delta (no _rank_eval: debugQuery score explain, LTR feature/model store REST, eDisMax qf/pf/mm tuning, external nDCG harness). Use before changing analyzers, synonyms, boosts, or query templates on an Elasticsearch, OpenSearch, or Solr index that serves user-facing search, so the NDCG / MRR baseline is captured first.

71

Quality

89%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

rank-eval-guide.mdreferences/

Rank Eval guide - binary thresholds, snapshot corpora, judgment tooling

Supplementary detail for elasticsearch-relevance-tests. The core _rank_eval workflow (judgments, metrics, request, test wrapper, per-query regression) stays in SKILL.md; this file holds the deeper configuration and tooling.

relevant_rating_threshold for binary metrics

Per the Elasticsearch Rank Eval API: Precision/Recall/MRR accept relevant_rating_threshold (default 1). For graded judgments:

"metric": {
  "precision": {
    "k": 10,
    "relevant_rating_threshold": 2,
    "ignore_unlabeled": false
  }
}

Rating >= 2 counted as "relevant"; below counted as "not relevant". The ignore_unlabeled flag controls whether unrated docs in results count against precision.

Reproducible test corpus

Snapshot the index state used for tests:

PUT _snapshot/test_repo/baseline_2026_05_06
{
  "indices": "products",
  "include_global_state": false
}

Restore for each CI run:

- name: Restore index snapshot
  run: |
    curl -X POST localhost:9200/_snapshot/test_repo/baseline_2026_05_06/_restore

Otherwise document changes (new docs, re-indexes) silently shift relevance baselines.

Quepid + Splainer integration

Quepid (open source from OpenSource Connections) provides:

  • Web UI for judges to rate per-query results
  • CSV export -> SKILL.md Step 1 judgment list
  • "Try" tab to test query template changes against current judgments

Splainer explains why a doc ranked where it did - invaluable for debugging unexpected results.

references

opensearch.md

rank-eval-guide.md

solr.md

SKILL.md

tile.json