Open Sourcing NLP Building Blocks

This post was originally published on Medium and moved here later. The post was updated during the move.

I’ve put NLP Building Blocks on GitHub. It’s five microservices that were previously commercial products available through DockerHub and the AWS and Azure marketplaces, and they’re all Java, all Spring Boot, all stateless behind a REST API, and all built on Idyl NLP, which itself sits on top of Apache OpenNLP.

• Idyl E3, entity extraction.
• Renku, language detection.
• Sonnet, tokenization.
• Prose, sentence detection.
• Verso, text preprocessing.

They are no longer maintained, and they’re on GitHub because working code sitting on a hard drive is worse than working code somebody might get some use out of. What I think is still worth talking about is the shape of the thing, since splitting it up this way is the decision I’d actually defend.

Five services instead of one

The obvious way to build this is one NLP service with an endpoint per capability. Tokenize here, extract entities there, all in one process, one deployment, one thing to run and monitor. We didn’t do that, and the reason comes down to models rather than code.

The model is the expensive part of any NLP service. A deep learning entity model wants real memory and takes real time per document, while tokenization is close to free. When those live in the same process you can’t scale them independently, so you end up provisioning for the entity model and then paying that price on every tokenization request too. Separating them means running three instances of the slow thing and one of the cheap thing, which is a considerably better use of the hardware.

Model lifecycle pushed in the same direction. Entity models get retrained often, because that’s where the domain specificity lives and domains keep moving, whereas a sentence detector that’s decent tends to stay decent for years. Bundle them together and updating the entity model means restarting the sentence detector for no reason at all.

Language detection ended up being its own case. It sits at the front as a router, so it sees every document including the ones that get dropped immediately afterward, and its traffic pattern looks nothing like anything downstream of it.

Where this gets you in trouble

A pipeline that preprocesses, tokenizes, detects sentences, and then extracts entities is four network calls per document instead of four method calls. That’s the whole tradeoff, and depending on your workload it’s either irrelevant or fatal. If you’re handling documents one at a time in response to a user doing something, the overhead vanishes into the noise and you get the operational benefits more or less for free. If you’re batch processing millions of documents it does not vanish at all, and you want the library inside your own process rather than a service across a network. I wouldn’t build a bulk processing pipeline this way.

There’s a second cost that’s easier to overlook, which is that you’ve turned a library call into a distributed system. Timeouts, retries, partial failures, and documents that make it through three stages and die on the fourth are all your problem now. None of that existed when it was a method call, and none of it shows up in the benchmark you ran before deciding this was a good idea.

The part I’d redo

Model training lived inside the services. Idyl E3 has model generator classes for entities, parts of speech, and lemmas, plus scheduled tasks to run them, so the same application that serves a model is also the one that builds it. At the time that felt convenient, because it meant one thing to deploy rather than two.

In practice it means your inference service and your training job share a JVM, share dependencies, and share a release cycle, and all three of those want to move at completely different speeds. Training is bursty, memory hungry, and allowed to fail without anyone losing sleep over it, while serving is none of those things. If I were doing it again the training would be its own container writing artifacts somewhere the serving side reads from, with nothing shared between them beyond the model format.

The rule I’d take away

Split NLP capabilities into separate services when they have different scaling profiles or different model lifecycles, and keep them together when they’re always used on the same document in the same request and you care about throughput. That sounds obvious written down, but it wasn’t obvious to me at the start, and I had built several of these before the pattern was clear enough to state in a sentence.

The code is on GitHub under Apache 2.0. If you’re building on Apache OpenNLP and want to see one way of structuring things it might save you some time, even if the conclusion you come to is that you’d rather keep the whole lot in one process.

If you’re working on NLP infrastructure and want to talk through how to carve it up, please reach out.