What is Learning to Rank?

Most search engines rank results with a scoring function like BM25 . It’s a good default: it rewards documents that contain the query terms, weighs rarer terms more heavily, and accounts for document length. But it only knows about the text. It has no idea that, for your users, a highly-rated product beats a poorly-rated one, that a recent article beats a stale one, or that a certain category of result almost always gets clicked. Learning to rank is how you teach the search engine those things.

The core idea

Learning to rank (LTR) uses machine learning to order search results. Instead of hand-tuning a scoring formula, you train a model on examples of good and bad rankings, and the model learns a ranking function from the patterns in that data.

The pieces you need are:

• Judgments that say how relevant a document is for a given query.
• Features that describe each query-document pair with numbers.
• A model trained on those features and judgments to predict relevance.

Let’s take them one at a time.

Judgments: what does “good” look like?

A judgment is a grade for a document given a query, for example “for the query running shoes, this product is a 3 out of 4.” There are two ways to get them.

Explicit judgments come from humans rating results. They’re accurate but slow and expensive, so you’ll only ever have them for a small set of queries.

Implicit judgments are derived from user behavior: clicks, add-to-carts, purchases, and the results people skip over. There’s an enormous amount of this data, and it reflects what users actually did. Capturing it in a structured way is its own problem, which is exactly what OpenSearch User Behavior Insights (UBI) is built for. Most real LTR systems lean heavily on implicit judgments because they scale.

Features: describing a query-document pair

A model can’t learn from raw text, so each query-document pair is described by a set of features, each a single number. Features usually fall into a few groups:

• Query-dependent features, like the BM25 score of the query against the title, or against the body.
• Document features, like a product’s rating, its popularity, or how recently it was published.
• Query features, like the length of the query.

Choosing good features is most of the work in LTR. The model can only be as smart as the signals you give it.

The model

With judgments and features in hand, you build a training set: for each judged query-document pair, the feature values plus the relevance grade. A learning-to-rank algorithm trains on that set to produce a model that predicts a relevance score for any new query-document pair.

The approaches are usually grouped as pointwise, pairwise, and listwise, depending on whether the model looks at one document at a time, pairs of documents, or a whole ranked list during training. In practice, gradient-boosted decision tree models such as LambdaMART are a common and strong choice.

How it works inside a search engine

You don’t run an expensive model over every document in your index. LTR is almost always a two-phase process:

1. Retrieve a set of candidate documents cheaply with a normal query like BM25.
2. Rerank just the top candidates by running the learned model over their features.

This keeps things fast: the cheap query narrows millions of documents down to a few hundred, and the model only has to score those.

Learning to rank in OpenSearch

The OpenSearch Learning to Rank plugin implements exactly this. It lets you define feature sets, log the feature values for your queries, upload a trained model, and then rerank results with it using a rescore query. Conceptually, the reranking step looks like this:

{
  "query": {
    "match": { "title": "running shoes" }
  },
  "rescore": {
    "query": {
      "rescore_query": {
        "sltr": {
          "params": { "keywords": "running shoes" },
          "model": "my_ltr_model"
        }
      }
    }
  }
}

The baseline match query retrieves candidates, and the sltr rescore query applies your model to reorder the top results. The plugin handles logging features and scoring with the model so you don’t have to build that machinery yourself.

If you’re on Elasticsearch, the same approach is available through the Elasticsearch Learning to Rank plugin , which the OpenSearch plugin descends from. The sltr rescore query and the feature-logging workflow will look familiar.

When to consider learning to rank

Learning to rank isn’t free. It needs judgments, some feature engineering, and a training loop you’ll want to revisit over time, so it’s worth reaching for when the payoff justifies that effort. A few situations where it tends to earn its keep:

• You have a lot of search traffic. Implicit judgments come from user behavior, so the more clicks and conversions you see, the better your training data.
• Relevance directly drives the outcome. E-commerce is the classic case, where a better ranking means more revenue, but the same holds anywhere users give up when the top results are wrong.
• You’ve outgrown hand-tuned boosting. When a query juggles so many signals, like recency, popularity, price, ratings, and text match, that tuning weights by hand has become a game of whack-a-mole, a model can balance them for you.
• You have signals a plain text query can’t easily use. Popularity, margin, and personalization are awkward to fold into BM25 but natural as LTR features.
• Different users want different things for the same query. A model trained on behavior can capture patterns that a single static formula can’t.

It’s also worth knowing when to hold off. If you have low query volume, you won’t gather enough judgments to train on. If your catalog is small and simple, BM25 with a few boosts may already do the job. And if you can’t capture user behavior yet, start there first, since that’s what User Behavior Insights is for, and revisit LTR once you have the data.

The costs to weigh

Before you commit, it helps to be clear-eyed about what LTR asks of you. It’s a powerful tool, but it isn’t set-and-forget.

• It needs training data. You have to gather judgments, and good implicit judgments take real query volume and a way to capture behavior. Without them the model has nothing to learn from.
• Feature engineering is most of the work. The model is only as good as the signals you feed it, and working out which features matter is hands-on and iterative.
• Models drift. Your catalog, your users, and their intent all change, so a model that was great six months ago can quietly go stale. LTR is a loop you maintain, not a one-time project, so you’ll retrain and re-evaluate on a schedule.
• It adds moving parts. Feature logging, model storage, a rescore step at query time, and a training pipeline are all new pieces to operate and monitor.
• It’s harder to debug. When a hand-tuned formula misbehaves you can read it. When a model ranks something oddly, working out why takes more effort.
• You have to measure it honestly. Without solid offline metrics and online A/B testing, it’s easy to ship a change that looks better but isn’t.

None of this should scare you off. It’s just why LTR pays off most when the use cases above line up. Go in expecting to maintain it, and it will reward you.

Alternatives to consider

LTR isn’t the only way to improve relevance, and sometimes a simpler tool gets you most of the way there. Before reaching for a model, it’s worth weighing a few options:

• Better analyzers and synonyms. A lot of what looks like bad relevance is really a tokenization or synonym problem. Fixing how text is analyzed, adding synonyms, and handling stemming often helps more than any ranking change.
• Query-time boosting. Boosting fields, recency, or popularity with a handful of weights is quick to set up and easy to reason about. If a few boosts do the job, you may not need a model at all.
• Query rules and curation. Pinning or burying specific results for known queries is a precise way to handle head queries and merchandising without training anything. Tools like Querqy make this kind of rule-based rewriting and curation easier to manage.
• Vector and hybrid search. Embedding-based semantic search, often combined with BM25 in a hybrid query, captures meaning that keyword matching misses. It’s a different lever than LTR, and the two can complement each other.
• Cross-encoder or LLM reranking. You can rerank the top results with a pretrained cross-encoder or an LLM instead of a model you train yourself. That trades training effort for inference cost and less control over the signals.

None of these are mutually exclusive with LTR. In practice, strong relevance usually comes from stacking several of them, and LTR is the piece that learns to balance the signals once you have enough behavior to train on.

Summary

Learning to rank replaces a hand-tuned scoring formula with a model that learns what relevance means for your users. It comes down to three things: judgments that grade relevance, features that describe each query-document pair, and a model trained to put them together. The OpenSearch Learning to Rank plugin gives you a practical way to run it, and User Behavior Insights gives you the implicit judgments to train it on. If you want to go deeper on applying LTR in real document workflows, see my post on neural networks and learning to rank .

If you’re trying to improve search relevance and wondering whether learning to rank is worth it for your use case, please reach out.