Leveraging Neural Networks and Learning-to-Rank in Document Workflows

This post is based on the talk I gave at Lucidworks Activate 2019 in Washington, DC. It looks at a specific question: in an environment of technical documents that, once written, follow a defined workflow, can we offer real-time assistance to authors to help their documents exit that workflow with a positive resolution?

The short answer is yes. Here I’ll walk through how a neural network classifier was built to predict a document’s final state, and how learning-to-rank was used to suggest phrases as authors write, all in a field dominated by highly technical text.

Background

The setting has a few defining characteristics. There is a workflow, the documents are highly technical, and a full trip through the workflow can take a while to complete.

A useful analogy is the Plinko game from The Price is Right. A workflow is a lot like Plinko: there are several terminal states of differing value, and if we had more control over the chip on the way down, we could improve our odds of landing on the big prize.

The document workflow itself looks like this: a document goes from New to In Progress to Submitted, and from Submitted it ends in one of three terminal states: Rejected, Approved, or Cancelled. Approved is the state we want, and documents can sit in the Submitted state for a long time, sometimes weeks.

The documents themselves are worth describing:

• Technical in nature, and likely to contain many non-dictionary terms.
• Typically less than 500 characters long.
• Written by a single author, but with many authors overall, so writing styles and experience levels vary.
• Accompanied by metadata as key/value pairs.

The goal and the approach

We wanted more documents to reach the Approved final state. Documents aren’t inherently good or bad, or high or low quality; the idea was to use our observations of Approved documents to improve the workflow, and to give authors feedback. The hypothesis was that with feedback, authors would produce documents with a higher probability of being Approved.

The approach had three steps:

• Predict the final state. Use past documents to build a classifier that predicts the final state of new documents.
• Help authors improve. Provide auto-complete for phrases while a document is being edited.
• Integrate it. Make both available to existing systems.

Step 1: Document classification

The classifier estimates the probability of each final state for an in-progress document, and the probability of reaching Approved is provided back to the author.

The training data was three sets of documents: those currently Approved, Cancelled, and Rejected. Each was formatted as one document per line, preceded by its state:

__approved__ this is a document
__cancelled__ this is another document
__rejected__ this is a third document

I trained the classifier with Facebook Research’s fastText, splitting the data 80/20 into train and test. Preprocessing removed English stop words, dropped tokens shorter than three characters, lowercased everything, and stripped punctuation. The resulting classifier reached 82% accuracy.

fastText was a good fit here for a few reasons. It doesn’t require a GPU, it’s a library for efficient learning of word representations and sentence classification, and it ships pre-trained English and multilingual vectors along with language-identification models.

It also handles out-of-vocabulary words well, which matters a lot for technical text where terminology changes over time. fastText uses character-level n-grams (I used 3 to 6 characters), and word vectors are computed as a sum of those parts. Words like “system,” “systems,” and “systematic” look similar, so when “systematically” shows up, its vector sums up similarly. Concretely, fastText processes “technology” as:

[te, tec, ech, chn, hno, nol, olo, ogy, gy]

That differs from the original word2vec, and it’s the same character-level idea that BERT and ELMo also build on.

For training I used 100 epochs, word n-grams of size 5, a 0.1 learning rate, a minimum of 3 and maximum of 6 characters, a minimum of 10 word occurrences, and hierarchical softmax as the loss function. Evaluation was a scripted grid search, with experiments run concurrently across systems and their parameters and validation results stored in a database so runs could go unattended. fastText also has built-in hyperparameter optimization to find the best F1 score.

A note on managing risk. The prediction values are meant as a guide for authors, not a definitive measure of a document’s quality, and that distinction was communicated through training and documentation. We didn’t want a consistently low score to block a document’s progress, because documents and terminology change over time and a prediction is just a prediction. During training we worked to prevent overfitting and to maximize recall.

Step 2: Phrase auto-suggest with learning-to-rank

Across the corpus, documents share common trends. Approved documents have a much higher ratio of technical words to dictionary words, and certain phrases recur. The idea for step two was to take a set of candidate n-grams and return them ranked, so an author sees the most relevant continuation of what they’re typing.

The source documents were two sets: successful documents already in the Approved state, and Submitted documents that the classifier predicted would be Approved, with a configurable threshold providing the cutoff.

Training the learning-to-rank model went like this:

1. Extract bigrams and trigrams from the source documents and collect their frequencies. For a large corpus this can be time consuming.
2. Take prefixes of length 3 from each n-gram and order them by frequency.
3. Use the prefixes, the candidate suggestions, and additional attributes to define the features.
4. Train the LTR model, retraining periodically as more documents become available.

Using the model at authoring time works in reverse. As an author edits, the first three letters after whitespace are used to fetch auto-complete suggestions through a web service call. The service finds matching n-gram phrases based on the prefix, ranks the candidates by applying the model, and returns a ranked list.

The LTR features included the frequency of the candidate suggestion, the edit distance between the prefix and the candidate, and the candidate’s relevance to the in-progress document. Relevance was determined by keyword extraction per document topic, other document metadata, and expert spot review. Finding the most optimal set of features was still a bit of an open question.

Step 3: Integration

Both pieces were exposed as simple REST services.

For prediction, model evaluation was wrapped in a REST service. In-progress documents are sent to it, and it responds with the predicted probability for each final state, which the author can use to improve the document.

For auto-suggest, an API endpoint accepts a prefix and the document, gets a list of candidate n-grams, ranks them based on the document, and returns the top subset.

The API is small:

POST /api/predict
(document as the request body)

{"Approved": 0.92, "Cancelled": 0.04, "Rejected": 0.04}


GET /api/suggestions?prefix={prefix}

{"jumps over the", "ate some cake", "fell asleep", ...}

Tools

Two libraries did the heavy lifting:

• jforests for learning-to-rank, using LambdaMART.
• fastText for document classification.

Summary

In the end, this described a document workflow, built a classifier to predict a document’s final state, created a phrase auto-suggester with learning-to-rank, and integrated both into the document authoring process. The result gives authors real-time, data-driven feedback while they write, with the goal of more documents reaching a positive resolution.

If you’re working on something similar, I’d enjoy comparing notes. Use the Contact Jeff button up top.