What is Learning to Rank?

Most search engines rank results with a scoring function like BM25. It’s a good default: it rewards documents that contain the query terms, weighs rarer terms more heavily, and accounts for document length. But it only knows about the text. It has no idea that, for your users, a highly-rated product beats a poorly-rated one, that a recent article beats a stale one, or that a certain category of result almost always gets clicked. Learning to rank is how you teach the search engine those things.

The core idea

Learning to rank (LTR) uses machine learning to order search results. Instead of hand-tuning a scoring formula, you train a model on examples of good and bad rankings, and the model learns a ranking function from the patterns in that data.

The pieces you need are:

• Judgments that say how relevant a document is for a given query.
• Features that describe each query-document pair with numbers.
• A model trained on those features and judgments to predict relevance.

Let’s take them one at a time.

Judgments: what does “good” look like?

A judgment is a grade for a document given a query, for example “for the query running shoes, this product is a 3 out of 4.” There are two ways to get them.

Explicit judgments come from humans rating results. They’re accurate but slow and expensive, so you’ll only ever have them for a small set of queries.

Implicit judgments are derived from user behavior: clicks, add-to-carts, purchases, and the results people skip over. There’s an enormous amount of this data, and it reflects what users actually did. Capturing it in a structured way is its own problem, which is exactly what OpenSearch User Behavior Insights (UBI) is built for. Most real LTR systems lean heavily on implicit judgments because they scale.

Features: describing a query-document pair

A model can’t learn from raw text, so each query-document pair is described by a set of features, each a single number. Features usually fall into a few groups:

• Query-dependent features, like the BM25 score of the query against the title, or against the body.
• Document features, like a product’s rating, its popularity, or how recently it was published.
• Query features, like the length of the query.

Choosing good features is most of the work in LTR. The model can only be as smart as the signals you give it.

The model

With judgments and features in hand, you build a training set: for each judged query-document pair, the feature values plus the relevance grade. A learning-to-rank algorithm trains on that set to produce a model that predicts a relevance score for any new query-document pair.

The approaches are usually grouped as pointwise, pairwise, and listwise, depending on whether the model looks at one document at a time, pairs of documents, or a whole ranked list during training. In practice, gradient-boosted decision tree models such as LambdaMART are a common and strong choice.

How it works inside a search engine

You don’t run an expensive model over every document in your index. LTR is almost always a two-phase process:

1. Retrieve a set of candidate documents cheaply with a normal query like BM25.
2. Rerank just the top candidates by running the learned model over their features.

This keeps things fast: the cheap query narrows millions of documents down to a few hundred, and the model only has to score those.

Learning to rank in OpenSearch

The OpenSearch Learning to Rank plugin implements exactly this. It lets you define feature sets, log the feature values for your queries, upload a trained model, and then rerank results with it using a rescore query. Conceptually, the reranking step looks like this:

{
  "query": {
    "match": { "title": "running shoes" }
  },
  "rescore": {
    "query": {
      "rescore_query": {
        "sltr": {
          "params": { "keywords": "running shoes" },
          "model": "my_ltr_model"
        }
      }
    }
  }
}

The baseline match query retrieves candidates, and the sltr rescore query applies your model to reorder the top results. The plugin handles logging features and scoring with the model so you don’t have to build that machinery yourself.

Summary

Learning to rank replaces a hand-tuned scoring formula with a model that learns what relevance means for your users. It comes down to three things: judgments that grade relevance, features that describe each query-document pair, and a model trained to put them together. The OpenSearch Learning to Rank plugin gives you a practical way to run it, and User Behavior Insights gives you the implicit judgments to train it on. If you want to go deeper on applying LTR in real document workflows, see my post on neural networks and learning to rank.

If you’re trying to improve search relevance and wondering whether learning to rank is worth it for your use case, use the Contact Jeff button up top.