Apache OpenNLP and LLMs: Where Does OpenNLP Fit In?

With large language models, NLP has exploded into the forefront of almost everything. That raises a fair question, and it’s one I get asked a lot as the PMC chair of Apache OpenNLP: where does a classic Java NLP toolkit fit in a landscape now dominated by LLMs?

I dug into that question with an experiment. Here’s what I found. This post accompanies the talk I gave on the same topic at Apache Community over Code in Halifax in October 2023.

A quick refresher

Apache OpenNLP is a machine learning based toolkit for processing natural language text. It handles the common NLP tasks: tokenization, sentence segmentation, named-entity extraction, chunking, language detection, parts-of-speech tagging, and document classification. The project has deep roots: the pre-Apache code on SourceForge goes back to 2003, it joined the ASF incubator in 2010, and became a top-level project in 2012.

Large language models (BERT, GPT-4, LLaMA, and friends) are language models characterized by their large size. They cover a lot of the same ground OpenNLP does (NER, classification, POS, tokenization) and then push well beyond it into text generation, summarization, natural language understanding, code generation, and conversational AI.

So the LLM-plus-Python ecosystem can do everything OpenNLP can, plus more. Why would anyone still reach for OpenNLP? To answer that honestly, I ran a comparison.

The experiment: named-entity recognition

A disclaimer up front: this is one unscientific test, one use case, one experiment. It isn’t a rigorous benchmark, and some things could surely be optimized on both sides. But it’s enough to highlight the tradeoffs.

I trained a person-entity NER model two ways:

• Apache OpenNLP: the built-in TokenNameFinder, with default first-run parameters (10 iterations, cutoff of 1), trained on CPU.
• A BERT-based model: using the excellent Python SpanMarker library with its default parameters (1 epoch, learning rate 0.00005), trained on GPU.

To be clear, none of this is a critique of SpanMarker. It’s a great NLP library with LLM support and a fine choice for a Python app. It’s just a useful stand-in for “the modern LLM approach.”

Both used the English person-entity subset of the multiNERD dataset. For OpenNLP I converted that data to its training format, where entities are marked inline:

<START:person> John Smith <END> is a person.

The conversion tooling and training code are on GitHub in my opennlp-formats repository.

The results

The BERT-based model is more accurate, about five points higher across the board. But look at what that accuracy costs: roughly 36x the training time and 662x the training cost, and it needs a GPU to get there. OpenNLP trained in three minutes on a cheap CPU instance for a third of a cent.

The point isn’t that one is always better than the other. The point is to surface the tradeoff so you can make an informed call for your own situation.

So what does it mean?

It comes down to what you value most: training and evaluation time, cost, or precision. Is a ~5% gain in accuracy worth 662x the cost and 36x the time? The honest answer is: it depends.

• What’s your current stack and architecture? What are your future plans?
• Is $2.25 significant? On its own, no. At scale, it can be.
• Does your model use a much larger dataset that takes longer to train?
• Do you need to retrain frequently because of model degradation?
• Do you need many models, such as separate ones per language or per entity type?

Multiply that $2.25 across frequent retraining and many models, and the cost gap starts to matter.

Inference tells a similar story. Both approaches are fast at inference time, but OpenNLP doesn’t need a GPU and generally needs far fewer resources. Its model files are tiny (a few KB versus hundreds of MB), with smaller CPU and memory requirements to match.

But what about Java devs who want LLMs?

Sometimes you have a genuine reason to use an LLM (fine-tuning, or a task OpenNLP doesn’t cover), but you still want to run inference from a JVM application. You don’t have to stand up a separate Python service to do it.

OpenNLP 2.0 introduced support for ONNX Runtime. You can train a model in Python, convert it to ONNX, and run inference directly from OpenNLP. Python folks stay in Python, Java folks stay in Java, and the model is served straight from the JVM, with no new services required. The integration currently supports NER, document classification, and sentence-embedding models, with more NLP tasks in progress. (Want to help? The team would love to have you.)

Converting a model is a single command:

python3 -m transformers.onnx -m nlptown/bert-base-multilingual-uncased-sentiment \
  --feature sequence-classification exported

From there you feed the tokenized inputs to an ONNX Runtime session in Java:

final Map<String, OnnxTensor> inputs = new HashMap<>();
inputs.put("input_ids", OnnxTensor.createTensor(env,
    LongBuffer.wrap(tokens.getIds()), new long[]{1, tokens.getIds().length}));
inputs.put("attention_mask", OnnxTensor.createTensor(env,
    LongBuffer.wrap(tokens.getMask()), new long[]{1, tokens.getMask().length}));
inputs.put("token_type_ids", OnnxTensor.createTensor(env,
    LongBuffer.wrap(tokens.getTypes()), new long[]{1, tokens.getTypes().length}));

final float[][][] vectors = (float[][][]) session.run(inputs).get(0).getValue();

Then walk the output array to find the highest-scoring label for each token. Best of all, this plugs into the existing OpenNLP interfaces, so it fits code you may already have.

The takeaway: two choices

So where does Apache OpenNLP fit in an LLM-dominated world? You have two good options, and they aren’t mutually exclusive:

1. Use OpenNLP’s native models. They’re still a solid choice for many NLP tasks: training is fast, needs no GPU, and costs very little.
2. Use LLMs with OpenNLP via ONNX Runtime when a task genuinely calls for them, and keep serving everything from Java.

LLMs are remarkable, but “more capable” and “right for the job” aren’t always the same thing. For a lot of real-world NLP, a fast, cheap, GPU-free model is exactly what you need, and when it isn’t, OpenNLP can still be the front door to the model you do need.

If you’d like to get involved, the Apache OpenNLP team would love to have you.