Adding ONNX Runtime Support to Apache OpenNLP

A lot of NLP development has shifted to the Python ecosystem over the last few years. That’s where the transformer models are trained, where Hugging Face lives, and where most of the tutorials point you. But Java is still one of the most widely used languages in production, and Java developers still have a real need for robust NLP, often inside existing systems like Apache Solr, Apache Lucene, and Apache UIMA.

As the project chair for Apache OpenNLP, I wanted to close that gap. So for OpenNLP 2.0 I added support for running Hugging Face transformer models directly in Java by integrating ONNX Runtime. This post walks through why I did it and how it works.

The problem

The motivating use case for me was document classification during indexing in Apache Solr: categorizing documents as they’re ingested to improve search relevance and result quality. The state-of-the-art models for that kind of task are transformers, and they’re trained in PyTorch or TensorFlow on the Python side.

Historically, a Java application had two options: reimplement the model (not realistic), or stand up a separate Python service just to serve inference and call it over the network. That second option works, but it adds an entirely new service to deploy, scale, monitor, and secure, all to run a model that could, in principle, run right inside the JVM.

Why ONNX Runtime

ONNX Runtime is a cross-platform accelerator for machine learning models trained from all the popular deep learning frameworks. Two things made it the right foundation:

It has a Java API, so inference can happen in-process, with no separate model server.
It’s framework agnostic. You train and evaluate your model with whatever tools you prefer, export it to the ONNX format, and run it. There’s no retraining and no lock-in to a specific toolchain.

That last point mattered a lot to me. A guiding principle of the OpenNLP integration is that it doesn’t force you to use any particular tools or frameworks to create and evaluate your models.

How it works

The workflow has three steps: train in Python, convert to ONNX, and deploy in Java.

1. Train a model (Python)

Nothing about OpenNLP changes how you train. Here’s the familiar Hugging Face path for a sentiment classifier on the IMDB movie reviews dataset:

from datasets import load_dataset
imdb = load_dataset("imdb")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=2)

trainer = Trainer(model=model, args=training_args, ...)
trainer.train()
trainer.save_model("distilbert-imdb")

2. Convert to ONNX

Hugging Face’s transformers library can export the trained checkpoint to ONNX directly:

python -m transformers.onnx --model=local-pt-checkpoint/ \
  --feature sequence-classification onnx

You now have a .onnx model file and its vocabulary: the two artifacts OpenNLP needs.

3. Deploy in Java (Apache OpenNLP)

Add the opennlp-dl dependency, which contains the ONNX Runtime integration:

<dependency>
  <groupId>org.apache.opennlp</groupId>
  <artifactId>opennlp-dl</artifactId>
  <version>2.0.0</version>
</dependency>

Then point OpenNLP at the model and vocabulary and classify text:

import opennlp.dl.doccat.DocumentCategorizerDL;

File model = new File("/path/to/model.onnx");
File vocab = new File("/path/to/vocab.txt");
Map<Integer, String> classifications = Map.of(0, "positive", 1, "negative");

DocumentCategorizerDL categorizer =
    new DocumentCategorizerDL(model, vocab, classifications,
        new AverageClassificationScoringStrategy(),
        new InferenceOptions());

double[] result = categorizer.categorize(new String[]{"I am happy"});

The returned array holds a probability for each class; the highest value is the predicted category. That’s the whole thing: a transformer trained in Python, running inline in a Java application, with no model server in between.

What OpenNLP 2.0 supports

At release, the ONNX Runtime integration covered:

Sequence classification: document categorization and sentiment analysis
Token classification: named-entity recognition

Parts-of-speech tagging and language detection via ONNX Runtime were on the roadmap for later releases.

Why it matters

For Java teams, this removes a real barrier. You get state-of-the-art transformer models:

Without retraining: export what you already have and run it.
Without an external service: inference happens in the JVM, so there’s one fewer system to operate and secure.
Without changing your workflow: keep training and evaluating with your preferred tools.

I’m proud of how this landed in OpenNLP 2.0, and it was great to collaborate with the ONNX Runtime team at Microsoft on it. If you’re working on NLP in Java and want to bring modern transformer models into your stack, I’d love to hear about it. Use the Contact Jeff button up top.