Neural Sparse Search in OpenSearch

Most relevance conversations end up as a choice between two options. BM25 is fast, cheap, and easy to reason about, but it only matches the words that are actually there. Dense vector search captures meaning, but it needs an approximate nearest neighbor index, a lot of memory, and it’s hard to inspect when a result looks wrong. Neural sparse search is a third option that sits between the two, and it deserves more attention than it usually gets.

One of my areas of work is search relevance, so this post is a practical introduction to neural sparse search . It covers what it is, when to reach for it, and what it looks like to set up.

What neural sparse search is

Neural sparse search is a form of learned sparse retrieval. A model reads your text and produces a sparse set of token and weight pairs. Instead of the handful of words that literally appear, you get an expanded set of terms the model thinks are relevant, each with a learned weight. A document about a heart attack might pick up weighted terms like “cardiac”, “myocardial”, and “infarction” even if those exact words never appear.

The important part is where those terms live. They’re stored in a sparse field and retrieved through the same inverted index that powers BM25. You get much of the semantic benefit of a neural model while still riding the mature, efficient machinery OpenSearch already has. There’s no separate vector index to build and hold in memory, and because the matches are real terms with weights, you can look at them and understand why a document scored the way it did.

The two modes, and why the choice matters

Neural sparse search runs in one of two modes, and picking the right one is most of the decision.

In bi-encoder mode, the model encodes both your documents at ingest time and your queries at search time. This gives the best relevance, because the query gets the same learned expansion the documents do. The cost is that every query runs model inference, which adds latency and compute to the hot path.

In doc-only mode, the model still encodes your documents at ingest time, but queries are expanded with a lightweight tokenizer or analyzer at search time instead of the full model. Query latency drops close to a plain lexical search, since there’s no model inference per query. You give up some relevance compared to the bi-encoder, but for many workloads the tradeoff is well worth it.

A simple way to decide is to ask where your budget is. If query latency and query-time compute are precious, doc-only is usually the right starting point. If you need the best possible relevance and can absorb the inference cost, use the bi-encoder.

When neural sparse search is useful

Neural sparse search is not the answer to every relevance problem, but there are several situations where it’s a strong fit.

• Your queries and documents use different words for the same thing. A user searching for “heart attack” should match a clinical note that says “myocardial infarction”. BM25 misses that. Neural sparse expands both sides so they meet.
• You want semantic recall without the memory bill of dense vectors. On a large corpus, a dense k-NN index can be expensive to keep in RAM. Neural sparse rides the inverted index, so it’s far lighter on memory.
• You’re running on CPU. Dense ANN search and model inference often want a GPU. Neural sparse works well on CPU, and doc-only mode keeps query-time work minimal.
• You need to explain why a result matched. Because matches are weighted terms rather than opaque vectors, you can inspect and debug relevance. That matters in regulated settings and on any team that has to justify ranking decisions.
• You have a latency-sensitive query path. Doc-only mode gives you semantic matching at close to plain lexical query speed, which is hard to get from dense search.
• Your content is full of domain jargon. Technical and specialized corpora, where terminology shifts and synonyms abound, are exactly where learned term expansion earns its keep.

If none of those describe you, plain BM25 or a dense vector approach might be the better call. Neural sparse shines when you want meaning-aware matching but also care about memory, cost, interpretability, or query latency.

What it looks like in OpenSearch

The workflow has a few pieces. You deploy a sparse encoding model, encode documents as they’re ingested, and then query against the sparse field.

First, create an ingest pipeline that runs the sparse_encoding processor to turn text into a sparse vector:

curl -X PUT "http://localhost:9200/_ingest/pipeline/neural-sparse-pipeline" \
  -H 'Content-Type: application/json' -d '
{
  "description": "Encodes passage_text into a sparse vector",
  "processors": [
    {
      "sparse_encoding": {
        "model_id": "<your-model-id>",
        "field_map": {
          "passage_text": "passage_embedding"
        }
      }
    }
  ]
}'

Then create an index that uses the pipeline by default and maps the sparse field:

curl -X PUT "http://localhost:9200/my-index" \
  -H 'Content-Type: application/json' -d '
{
  "settings": {
    "default_pipeline": "neural-sparse-pipeline"
  },
  "mappings": {
    "properties": {
      "passage_text": { "type": "text" },
      "passage_embedding": { "type": "rank_features" }
    }
  }
}'

Indexing a document is unremarkable, and the pipeline encodes the text for you:

curl -X POST "http://localhost:9200/my-index/_doc" \
  -H 'Content-Type: application/json' -d '
{
  "passage_text": "Aspirin can lower the risk of a heart attack."
}'

Searching uses the neural_sparse query. In bi-encoder mode you pass the same model that encoded your documents, and it expands the query too:

curl -X POST "http://localhost:9200/my-index/_search" \
  -H 'Content-Type: application/json' -d '
{
  "query": {
    "neural_sparse": {
      "passage_embedding": {
        "query_text": "medication to prevent myocardial infarction",
        "model_id": "<your-model-id>"
      }
    }
  }
}'

For doc-only mode, you drop the query-time model in favor of an analyzer, so the query is tokenized rather than run through the full model:

curl -X POST "http://localhost:9200/my-index/_search" \
  -H 'Content-Type: application/json' -d '
{
  "query": {
    "neural_sparse": {
      "passage_embedding": {
        "query_text": "medication to prevent myocardial infarction",
        "analyzer": "standard"
      }
    }
  }
}'

The exact model, field type, and analyzer options move between OpenSearch versions, so check the neural sparse documentation for the version you’re running before you wire it up.

The tradeoffs to weigh

Neural sparse search is not free, and it’s worth going in with eyes open.

• Your index gets bigger. Expanding each document into many weighted terms grows the index on disk compared to plain text.
• Bi-encoder queries cost inference. Running the model on every query adds latency and compute. Doc-only mode avoids this, at some relevance cost.
• You have a model to manage. Deploying, versioning, and updating the sparse encoding model is real operational work.
• Relevance depends on the model. The quality of your results is only as good as the model doing the expansion, so the choice of model matters.

Where it fits with everything else

Neural sparse search isn’t an island. It combines well with the rest of your relevance stack. A common pattern is a hybrid query that blends neural sparse with BM25, or even with dense vectors, so you get lexical precision and semantic recall together. On top of that, User Behavior Insights lets you measure whether a change actually helped real users, and learning to rank can rerank the top results once you have the behavior to train on. Neural sparse is one strong retrieval option among several, and the best systems usually stack a few of them.

Summary

Neural sparse search gives you meaning-aware retrieval that still runs on the inverted index, stays interpretable, and is light on memory. It’s a particularly good fit when your queries and documents use different words for the same idea, when you can’t afford a dense vector index, or when you need to explain your relevance. Pick doc-only mode when query latency matters and the bi-encoder when you want the best relevance you can get.

If you’re working on OpenSearch relevance and want to talk through whether neural sparse search fits your stack, please reach out.