Applied MLOps: Keeping Models Fresh on Kubernetes

This post was originally published on Medium and moved here later. The post was updated during the move.

David Smithbauer and I gave a talk at Berlin Buzzwords last week, Applied MLOps to Maintain Model Freshness on Kubernetes. This is the written version. All the code is in the repository if you’d rather skip to that.

What we wanted to poke at is that models go stale, and in search they go stale faster than people expect. The weights don’t rot. The world moves and the model doesn’t notice.

The standard answer is a retraining schedule. That works, and it costs you compute, time, and fresh labels, and the labels are usually the part you don’t have.

So we went looking for how current we could stay without retraining anything.

The demo

Rank movie search results by whatever people are talking about on Twitter right now.

If Christmas is trending, a search for family movies should put Christmas movies at the top. In July it shouldn’t. And nobody is going to hand-maintain that, or train a Christmas movie classifier, then a Halloween movie classifier, then one for whatever turns up next week.

Apache Flink reads the tweet stream, counts hashtags, and keeps running totals in a Redis sorted set. Getting the current top hashtags back out is one ZREVRANGEBYSCORE.

Elasticsearch holds a few thousand TMDB movies and their overviews.

What sits between those two is the part worth talking about.

Zero-shot is what makes it cheap

The classifier is an NLI model, BERT fine-tuned on MNLI, used zero-shot. The label is an input at inference time rather than something baked in when the model was trained.

Hand it a movie overview and the candidate label “christmas” and you get a number between 0 and 1 for how well those two go together. “Jingle All the Way” comes back above 0.9. “Space Jam” doesn’t.

Which means a hashtag that started trending an hour ago works exactly as well as one from last year. There’s nothing to retrain, because the thing that changes is an argument instead of a weight.

So the pipeline takes the top trending hashtag, runs every indexed overview through the classifier with that hashtag as the label, and writes the score onto the document in a field named after it. classification_christmas. Then you sort on that.

Search the Family genre, sort descending on classification_christmas, and family Christmas movies come back first.

That pass touches every document in the index, so it’s a nightly job rather than something that happens per query. How often you actually want to run it depends on how much your content cares what’s trending.

The pipeline parts

Locally it’s all Docker Compose. Flink, Elasticsearch, Redis, the classifier, MySQL, and Quepid come up together, which matters more than it sounds like it should. You can’t reason about a system like this if you can’t run all of it at once.

On Kubernetes, KServe serves the classifier, the Elastic resources are managed through GitOps instead of by hand, and model artifacts are versioned with DVC or pulled from the HuggingFace hub. Training runs as its own container and writes artifacts out for something else to pick up.

I don’t think the specific choices there matter much. What matters is that every step between training and serving is something you can trigger, version, and run again.

Did it actually work

Here’s the part that tends to go missing from MLOps talks.

You can automate retraining and deployment beautifully and still have no idea whether the new model is better than the old one. For a ranking model, better doesn’t mean accuracy on a held-out set. It means the search results improved, which is a harder question.

So we built judgments into the demo. We scored the results for our test searches by hand, put them in MySQL, and used Quepid for the scoring workflow. A small service runs a search, matches each result against the judgments, and computes DCG and NDCG .

Train a model, deploy it, re-run the classify pass, recompute, compare against the baseline. That’s the loop.

The failures you turn up doing this are more interesting than the number itself. Searching Family movies for “christmas” gives back 40 results, one of which is a film called “Savannah” that has nothing to do with Christmas. There’s a character in it named Christmas Moultrie. A few others just happen to be set around Christmas. A metric ticking upward wouldn’t have told us about any of that.

Things I’d want to fix

This is a conference demo. It’s meant to show the shape of the thing and it has never been anywhere near a production incident.

Zero-shot output turned out to be more sensitive to label phrasing than I expected. “christmas” and “a Christmas movie” don’t score the same way. There’s prompt tuning hiding inside what looks like a config value, and we didn’t do much of it.

The judgments are the bottleneck, which is always true of this kind of work. We counted any document not in the judgment table as a zero. That’s convenient and it’s wrong, and it’s still good enough for comparing two versions of a model against each other.

A field per trending topic means the index grows along with the topic list, and we didn’t build anything to retire the old ones.

The classify pass is also real compute. Every topic you support is another full sweep of the corpus through a transformer, on a schedule.

Anyway

Even if you never build this exact thing, the general move seems worth stealing. Find the part of your model that has to change when the world changes, and see whether you can get it out of the weights and into an input.

And build the measurement before the automation. Otherwise what you’ve got is a very reliable way to ship models without knowing whether they’re any good.

If you’re working on search relevance or trying to get a model pipeline under control, please reach out.