Using Modern NLP Models from Apache OpenNLP with Apache Solr

Apache OpenNLP takes a “classical” approach to NLP, one that predates today’s deep learning methods. That turns out to be a feature: it’s fast without a GPU, its performance can be on par with newer methods, and it has a healthy user and developer community. But NLP has changed enormously over the last decade, and a common question is how to bring those modern models into a search engine like Apache Solr.

This post is based on a talk I gave with Eric Pugh on exactly that. The short version: you can run modern transformer models inside Solr at index time, through Apache Lucene and Apache OpenNLP, without standing up any new services.

A little history

OpenNLP and Solr have a long shared timeline. OpenNLP started on SourceForge around 2002 and joined the Apache incubator in 2010. The Lucene ticket to integrate OpenNLP (LUCENE-2899) was resolved in 2017, and Solr 7.3.0 shipped the OpenNLP update processor soon after. OpenNLP 2.0 arrived in 2022 and introduced support for ONNX Runtime, which is the piece that makes everything below possible.

In the years between, practically everything in NLP changed: the adoption of deep neural networks, the Python ecosystem of PyTorch and friends, embeddings, and a steady stream of new state-of-the-art results. The motivation here is to make that progress usable from the Java and Solr side of the world.

ONNX Runtime as the bridge

ONNX Runtime is an open-source, high-performance inference engine that executes machine learning models in the Open Neural Network Exchange (ONNX) format. It acts as a common execution layer, so models trained in various frameworks (like PyTorch or TensorFlow) can be exported to ONNX and run anywhere ONNX Runtime runs, including from Java.

That is the whole trick. Training stays in Python, the model is exported to ONNX, and OpenNLP runs inference on it directly. No model server sits in between.

How the model runs

A transformer classification model takes a few tensors as input: input_ids, attention_mask, and token_type_ids. It returns logits, which are the probability of each label for each token in the input. There is one logit per label, and the labels themselves are defined in the model’s config file, set at training time and typically expressed in inside-outside-beginning (IOB or BIO) format.

So for an input like “George Washington was president,” the model produces a label for each token:

George   Washington   was   president
B-PER    I-PER         O     O

Diagram of model inputs and outputs. The tokens George, Washington, was, president feed in, and the model outputs a row of logits per token; the highest logit in each row maps to a label such as B-PER, I-PER, or O. — For each token, the highest logit selects the label. The label set comes from the model’s config.

If you want to see a model’s inputs and outputs concretely, Netron is a handy viewer for neural network and machine learning models.

Getting a model in ONNX format

You have two options. Convert an existing model to ONNX:

optimum-cli export onnx --model <model-id> --task text-classification \
  ./onnx_model_directory

Or just grab a model that is already published in ONNX format, such as onnx-community/bert-base-multilingual-uncased-sentiment-ONNX on Hugging Face. It’s worth checking whether an ONNX version already exists before converting one yourself.

One important point: no training happens in OpenNLP. Models are trained with PyTorch or TensorFlow, converted to ONNX, and then used from OpenNLP.

Wiring it into Solr

With ONNX support in place, we can use these models in Solr through Lucene through OpenNLP. The integration is a Solr UpdateProcessorFactory that classifies text as documents are indexed, for tasks like sentiment analysis or categorization. It’s configured with four things:

The model to use.
The vocabulary file for tokenization.
The source field to classify.
The destination field for the result.

Solr updateRequestProcessorChain configuration for OpenNLP document classification, with callouts pointing to the model file, the vocabulary file, the source field, and the destination field. — The Solr update processor configuration: model, vocabulary, source field, and destination field.

After indexing, each document carries a classification (for example, sentiment) derived from its text.

On the OpenNLP side, running inference means building the input tensors and reading back the output array:

final Map<String, OnnxTensor> inputs = new HashMap<>();
inputs.put("input_ids", OnnxTensor.createTensor(env,
    LongBuffer.wrap(tokens.getIds()), new long[]{1, tokens.getIds().length}));
inputs.put("attention_mask", OnnxTensor.createTensor(env,
    LongBuffer.wrap(tokens.getMask()), new long[]{1, tokens.getMask().length}));
inputs.put("token_type_ids", OnnxTensor.createTensor(env,
    LongBuffer.wrap(tokens.getTypes()), new long[]{1, tokens.getTypes().length}));

final float[][][] vectors = (float[][][]) session.run(inputs).get(0).getValue();

On the Solr side, the update processor instantiates OpenNLP’s document categorizer, runs the inference, and reads the best category:

documentCategorizerDL = new DocumentCategorizerDL(
    modelFile.toFile(),
    vocabFile.toFile(),
    getCategories(),
    new AverageClassificationScoringStrategy(),
    new InferenceOptions());

final double[] result = documentCategorizerDL.categorize(new String[] {fullText});

String bestCategory = documentCategorizerDL.getBestCategory(result);

The Solr-side work lives in apache/solr#1999 (SOLR-17023), which adds support for document classification using OpenNLP and ONNX.

Why this is nice

The best part is what it doesn’t change. It has no impact on how you train models, so you keep using your favorite Python tools. It simply makes those models available to OpenNLP, and therefore to every project that depends on OpenNLP, including Lucene and Solr.

Next steps

There’s more to do on both sides:

On the OpenNLP side: support more NLP tasks via ONNX Runtime (document classification and NER work today) and improve the handling of model properties, such as sentiment classifications.
On the Solr side: take advantage of those OpenNLP improvements to obtain the classifications directly from OpenNLP instead of listing them in the Solr config, and add support for NER ONNX models.
Everywhere: think more carefully about when to lean on a GPU versus a CPU.

Try it out

OpenNLP supports models in ONNX format, and through Lucene that capability is available to Solr in the linked PR. There’s plenty of interesting work left, so if you’d like to get involved, the Apache OpenNLP team would love to have you. Reach out any time using the Contact Jeff button up top.