CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/elasticsearch-relevance-tests

Author Elasticsearch relevance regression tests using the Ranking Evaluation API (`POST {index}/_rank_eval`) - judgment lists (query + expected docs at ranks), per-query metrics (Precision@K, Recall@K, MRR, DCG, ERR), reproducible test corpora; pair with Quepid + Splainer for interactive judgment authoring. Use before changing analyzers, synonyms, boosts, or query templates on an Elasticsearch index that serves user-facing search, so the NDCG / MRR baseline is captured first.

79

Quality

99%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

rank-eval-guide.mdreferences/

Rank Eval guide - binary thresholds, snapshot corpora, judgment tooling

Supplementary detail for elasticsearch-relevance-tests. The core _rank_eval workflow (judgments, metrics, request, test wrapper, per-query regression) stays in SKILL.md; this file holds the deeper configuration and tooling.

relevant_rating_threshold for binary metrics

Per the Elasticsearch Rank Eval API: Precision/Recall/MRR accept relevant_rating_threshold (default 1). For graded judgments:

"metric": {
  "precision": {
    "k": 10,
    "relevant_rating_threshold": 2,
    "ignore_unlabeled": false
  }
}

Rating >= 2 counted as "relevant"; below counted as "not relevant". The ignore_unlabeled flag controls whether unrated docs in results count against precision.

Reproducible test corpus

Snapshot the index state used for tests:

PUT _snapshot/test_repo/baseline_2026_05_06
{
  "indices": "products",
  "include_global_state": false
}

Restore for each CI run:

- name: Restore index snapshot
  run: |
    curl -X POST localhost:9200/_snapshot/test_repo/baseline_2026_05_06/_restore

Otherwise document changes (new docs, re-indexes) silently shift relevance baselines.

Quepid + Splainer integration

Quepid (open source from OpenSource Connections) provides:

  • Web UI for judges to rate per-query results
  • CSV export -> SKILL.md Step 1 judgment list
  • "Try" tab to test query template changes against current judgments

Splainer explains why a doc ranked where it did - invaluable for debugging unexpected results.

references

rank-eval-guide.md

SKILL.md

tile.json