How to Actually Measure Search Relevance

Everyone wants better search relevance, but “better” is not a plan. Before you can improve ranking, you have to be able to say whether one version is actually better than another, and by how much. That means measuring it, and measuring it in a way that is repeatable rather than a gut feeling about a handful of queries you happen to remember. This post is a practical tour of the offline metrics that let you do that.

Offline and online measurement

There are two broad ways to measure relevance. Online measurement watches real users through A/B tests and behavioral signals, and it is the ground truth for whether a change helped. Offline measurement scores your ranking against a fixed set of judged queries, without putting anything in front of users.

Offline metrics are the focus here. They are fast, repeatable, and safe to run against every change before it ships, which makes them the workhorse of day-to-day relevance work. The catch is that they are only an approximation of what users actually want, so the honest workflow is to iterate offline and then confirm the winners online.

You need judgments first

Every offline metric is computed against judgments, which are relevance labels for query and document pairs. Judgments come in two shapes.

Binary judgments say a document is either relevant or not for a query. Graded judgments use a scale, often 0 to 3 or 0 to 4, to say how relevant a document is, so “exactly what I wanted” scores higher than “sort of related.” Graded judgments carry more information but cost more to produce.

Judgments can come from human raters or be derived from user behavior like clicks and conversions. Where they come from is its own topic, and one that learning to rank and User Behavior Insights both deal with. Tools like Quepid are built for gathering those judgments and scoring a ranking against them with the metrics below. For this post, assume you have a set of judged queries and want to score a ranking against them.

The metrics

Each metric below is computed per query and then averaged across your whole query set. The interesting differences are in what each one rewards, and a lot of that comes down to how much credit a result gets based on where it sits in the ranking.

Precision@k and Recall@k

Precision@k is the fraction of the top k results that are relevant. If 7 of your top 10 are relevant, precision@10 is 0.7. Recall@k is the fraction of all the relevant documents that appear in the top k. If there are 20 relevant documents in total and 8 of them show up in the top 10, recall@10 is 0.4.

Use precision@k when users only look at the first page or two and you care how many of those results are good. Use recall@k when finding everything relevant matters, such as legal discovery or a research search where a missed document is a real failure.

The tradeoff is that both ignore order within the top k. A relevant result at rank 1 and one at rank 10 count exactly the same, which is rarely how users experience a ranked list. Both also depend heavily on the choice of k, and both use binary relevance, so they cannot tell a great result from a merely acceptable one.

Mean reciprocal rank (MRR)

Mean reciprocal rank looks only at the first relevant result. For each query you take the reciprocal of the rank of the first relevant hit, so a first-place hit scores 1, second place scores 0.5, third scores 0.33, and so on. You then average that across queries.

Use MRR when there is essentially one right answer and getting it near the top is the whole game. Known-item lookups, navigational queries, and question answering all fit this shape.

The tradeoff is that MRR ignores everything after the first relevant result. If a query has ten good answers, MRR does not care whether you surface one of them or all ten, which makes it a poor fit for exploratory or multi-answer search.

Mean average precision (MAP)

Mean average precision extends precision to the whole ranked list. For a single query, average precision is the average of the precision values measured at each rank where a relevant document appears. MAP is that number averaged across all your queries.

Use MAP when relevance is binary, queries have several relevant documents, and you want a single number that rewards ranking all of them highly rather than just the first. It is a long-standing default in information retrieval research.

The tradeoff is that MAP is still binary, so it cannot express degrees of relevance. It also gets harder to reason about intuitively than precision or MRR, since a single score folds together the positions of many relevant documents.

Normalized discounted cumulative gain (NDCG)

NDCG is the metric most modern relevance work reaches for, because it handles graded relevance and discounts results by position. It builds on discounted cumulative gain (DCG), which sums the relevance of each result while dividing by a log of its rank, so results further down contribute less:

DCG@k = sum over i = 1..k of (2^rel_i - 1) / log2(i + 1)

DCG on its own is not comparable across queries, because a query with more relevant documents can pile up a higher score. NDCG fixes that by dividing DCG by the ideal DCG (IDCG), which is the DCG you would get from the best possible ordering. The result is a number between 0 and 1, where 1 means your ranking matches the ideal.

A quick example makes it concrete. Suppose a query returns five results with these graded labels on a 0 to 3 scale:

rank 1: 3    rank 2: 2    rank 3: 0    rank 4: 1    rank 5: 2

DCG@5  = 7/1.00 + 3/1.58 + 0/2.00 + 1/2.32 + 3/2.58 = 10.49
IDCG@5 (ideal order 3, 2, 2, 1, 0)                  = 10.82
NDCG@5 = 10.49 / 10.82                              = 0.97

Use NDCG when you have graded judgments and position matters, which covers most web, product, and content search. The main tradeoff is that it needs graded judgments, which are more expensive to collect than binary ones, and the score is less immediately intuitive than something like precision@k.

Choosing the right metric

The right metric follows from what your users are trying to do.

• If there is one correct answer and it needs to be near the top, use MRR.
• If users scan a top-k list and you care how many results are good, use precision@k.
• If missing any relevant document is costly, use recall@k.
• If relevance is binary but queries have many relevant documents, use MAP.
• If you have graded judgments and position matters, use NDCG, which is the safe default for most relevance work.

You do not have to pick only one. A common practice is to track NDCG as the primary metric and watch precision@k and recall@k alongside it, since they are easier to explain to stakeholders and catch different kinds of regressions.

Tradeoffs to keep in mind

Whichever metric you choose, a few things are worth remembering.

The cutoff k shapes the answer. A small k hides recall problems, and a large k includes results real users never scroll to, so pick a k that reflects how your users actually behave.

Judgments are expensive and incomplete. You will never judge every document for every query, and most tools treat unjudged documents as not relevant, which can unfairly penalize a ranking that surfaced a genuinely good result nobody labeled. Be aware of how much of your result set is actually judged.

Averages hide distributions. A healthy mean NDCG can still hide a cluster of queries that perform terribly. Look at the worst-performing queries, not just the average, because those are where users get frustrated.

Offline is not online. Offline metrics approximate user satisfaction, and they can move in ways that do not translate to real behavior. Use them to filter and iterate, then validate the winners with an online A/B test before you trust them.

Summary

Measuring search relevance well is the difference between guessing and knowing. Precision and recall tell you how many results are good and how many you found. MRR rewards getting the one right answer to the top. MAP scores the whole ranking under binary relevance. NDCG handles graded relevance with a position discount and is the default for most modern work. Pick the metric that matches what your users are trying to do, watch the worst queries rather than just the average, and confirm your offline wins with an online test.

If you are working on search relevance and want help setting up a measurement process you can trust, please reach out.