Building a Right-to-Be-Forgotten Skill for the OpenSearch Agent Skills Hackathon

The OpenSearch Agent Skills Hackathon asked for agent skills that solve a real OpenSearch problem and ship as production-ready code with a short demo video, and the one I entered handles right-to-be-forgotten requests. The code is on GitHub, the submission thread has the discussion that shaped it, and there is a demo video.

The problem I picked

Deleting every document about one person is hard even when you have their name and a unique ID. The identifier sits in a user.id field on some records, in the body text of others, and in the from and cc fields of anything shaped like a message. At least the work is mechanical, since a regex finds the literal string and you can check the result by searching for it again.

The other half never announces itself. Somewhere in an incident review is a sentence like “the solo senior frontend engineer on-call during the #4091 outage who resigned at the end of March,” with no name, no ID, and nothing to match on. If exactly one person fits that description then the sentence identifies them as precisely as their badge number does, but a PII scanner sees nothing to redact and reports a clean index. That is a search problem wearing a compliance hat, and because it needs both retrieval and judgment rather than pattern matching, it looked like a better fit for a skill than for another CLI tool. So the skill runs two passes, a direct one that matches literal identifiers and needs no reasoning at all, and an indirect one that is the rest of this post.

What the skill actually is

An agent skill is a folder, and mine holds a SKILL.md that tells the agent what it is and what order to do things in, a scripts/ directory of Python that the agent calls, and a knowledge/ directory it reads when it needs background on indirect identification or the regulation. Drop that folder into Claude Code, Cursor, Kiro, Copilot, Windsurf, Gemini CLI, or Codex and the agent picks it up, with no runtime to stand up, no service to host, and nothing to deploy.

The front matter is doing more work than it looks like, because the description field is how the agent decides whether this skill applies to whatever you just asked. Mine enumerates the vocabulary a real request tends to arrive in, including GDPR, CCPA, right to be forgotten, DSAR, data subject request, PII removal, redaction, and Article 17, and it ends with an instruction to activate even when the user never says the word OpenSearch. Nobody opens a conversation with “run a hybrid query against my log indices,” they open it with “we got a request, scrub this person from our logs,” and the skill has to trigger on the second one.

The practical consequence is that the interface is a sentence rather than a command, and a session tends to open with roughly what the person handling the request would have written in an email anyway. In the demo it looks like this:

We’ve had a right-to-be-forgotten request. The person is a senior frontend engineer who owned Checkout, was sole on-call during incident #4091, and resigned end of March. Scrub them from logs-application-*. Redact, don’t delete.

From that the agent pulls out the profile, the index pattern, and the action, asks for anything it still needs, and then works through the phases in SKILL.md in order. Nobody types a script name at any point, which is the part that makes it a skill rather than documentation for a CLI.

Drawing the line between the script and the agent

The design question that mattered most was which half of the work belongs to deterministic code and which half belongs to the model, because getting it wrong leaves you with either a chatbot wrapping a CLI or a script with an unreliable narrator. Everything reproducible ended up in Python, including the hybrid query, the direct identifier search, the threshold filtering, and the generated update DSL. Those all have correct answers and need to produce the same output twice, and a model that paraphrases a Painless script instead of emitting it verbatim is a liability.

One thing lives in the agent, which is deciding whether a retrieved document is actually about this person. Consider three documents that all rank well against the profile above. One says “Engineer Tom Jones deployed a checkout hotfix during #4091,” which names a different person. Another says “the senior frontend engineer on the Search team shipped autocomplete,” which is the right role on the wrong team. A third says “a frontend intern joined the checkout team in June,” which is wrong on both seniority and timeline. All three share vocabulary with the profile, and no threshold on a relevance score separates them from the document that is actually the person.

So SKILL.md hands the agent a judgment prompt and asks for strict JSON back, covering whether the document is identifiable, at what confidence, and which exact substrings do the identifying. Those snippets have to be verbatim, since they are what redaction replaces and one that is not a real substring makes the update silently do nothing. A precision mode then thresholds the confidence scores, defaulting to a loose 0.60, because an over-flagged document gets caught by the human review while a missed one just stays in the index. In the demo that leaves the agent with 20 candidates out of 940 retrieved documents to decide on.

Why hybrid retrieval feeds the judgment

The candidates come from hybrid search, because the two clauses fail in opposite directions and this problem needs both of them. The lexical clause catches sharp anchors like the incident number, the service name, and the date, while the neural clause catches the same fact expressed in different words, so that “lead FE who owned the cart UI” can still reach “senior frontend engineer on the checkout squad.”

The embedding side runs on a local pretrained model deployed through ML Commons, currently all-MiniLM-L6-v2. Portability across distributions was one of the hackathon judging criteria, but it was also the only choice I could defend on its own terms, since shipping the contents of a personal data investigation to a hosted embedding API in order to satisfy an erasure request is not a trade worth making. If no model is deployed at all, the skill falls back to BM25 and reports that in its output rather than quietly returning less than you asked for.

The skill never writes to your cluster

The skill does not do any deleting, instead emitting a shell script with one command per flagged document, each targeting an exact index and _id rather than a query, each carrying a comment explaining why that document is in the list, and each followed by a read-back command so you can confirm the result afterward. Any index you name as being under legal hold is refused outright, you read the script yourself, and you are the one who runs it. Alongside the script the skill writes a hash-chained certificate recording what the run decided, which is the piece that outlives the chat transcript and can be handed to somebody later.

Redaction rather than deletion is the default action, because replacing the identifying spans with [GDPR_REDACTED] removes the person while leaving the rest of the log line intact and still operationally useful. It also makes a false positive survivable, since redacting a phrase from somebody else’s record is recoverable in a way that deleting their record is not.

How to evaluate it

The direct pass validates itself, since it finds documents containing a literal string and anyone can confirm a hit by searching for the same string. The judgment pass has no such key, because whether “the sole senior frontend engineer on Checkout who resigned in March” identifies someone depends on how many people fit the description, which is exactly what the document does not say. Labelling that by hand would mean an assessor reading every document and performing the identification the skill exists to remediate.

What I landed on is mask and recover, which needs a corpus with two channels, one naming people in structured form like an email header and one describing them in prose. Discovery only searches the prose, so the structured channel can supply labels without contaminating the search. Pick a subject, record every document whose text holds one of their identifiers, strip the identifiers out, and index the result separately. The label is then a fact rather than a judgment, and whether the pipeline still finds those documents measures whether the residual context identifies the person. I ran it on two corpora that have both channels, the Enron email archive and US court opinions from the CourtListener bulk export, and most of the work turned out to be in the ways this quietly breaks.

Mask broadly, label narrowly

One variant list cannot do both jobs, which took me longer to see than it should have. Masking has to remove anything that might conceivably refer to the subject, because a surviving variant makes the document trivially retrievable and invalidates every number computed from it, and since over-masking costs you nothing but residual context, masking should be greedy.

Labelling has to work the other way around, counting a document as a positive only when it holds a variant that nobody else in the population produces. Seven people in the Enron corpus share one subject’s given name, and labelling that subject on the full alias list with the given name included made 92% of their positives into documents about somebody else. The retrieval numbers computed on top of that were answering a different question than the one I thought I was asking, and they looked perfectly reasonable while doing it.

Failing the run when the mask leaks

If a single variant survives the mask then every number downstream of it is meaningless, so a survivor fails the run outright rather than producing a score with a caveat attached. The audit checks for surviving variants, document ids that embed a name, header fields carried across into the masked index, and the mask marker itself, and that last one is the trap I walked into. My first pass replaced each name with a [MASKED] token, which is tidy and also appears in exactly the documents that contained a name, which is the definition of a positive. The count made it obvious once I looked, with the marker sitting in 743 documents against 711 that had been masked, so masking now removes the variant and leaves nothing behind.

The synthetic corpus needs the same care for a different reason, since the agent is both the thing under test and the thing reporting the score. Document ids are opaque digests rather than labels like sub-1, the ground truth goes to a file instead of the terminal, and SKILL.md tells the agent not to read it. That last part is a soft control, but a corpus that does not label itself is the half that actually holds.

A result that did not survive

An early run on 300 subjects put hybrid retrieval 7.0 points ahead of BM25 at k=50, which read like the first real evidence that hybrid surfaces documents the lexical clause misses, and that is the mechanism claim the whole skill rests on. I reported it with a caveat that it was not yet settled, and it turned out not to be. Running it against nine times as many subjects shrank the gap to 2.2 points with overlapping confidence intervals and reversed the ordering entirely at k=10. Here is where it ended up, measured across 2,692 case-law subjects over a 5,470-document index.

BM25 wins decisively at k=1 and k=5, and from k=10 onward the two are statistically indistinguishable, so the honest reading is that hybrid buys parity at depth rather than an edge and that lexical retrieval is the better choice when you need precision at the top. That is a considerably less exciting claim than the one I started with, and it is the one the data actually supports. Hybrid stays in because the judgment stage reads every candidate, so recall at depth is what matters and rank 1 is not worth much on its own.

Why RRF instead of normalization

Testing hybrid against BM25 alone did turn up one result that stuck. Min-max normalization scales each clause within its own result set, so an uninformative neural clause gets its noise stretched across the full range and then averaged into the lexical score, which pushes good lexical hits down for no reason at all. Reciprocal rank fusion works on ranks instead, so a weak clause can only contribute bounded noise.

The size of that effect surprised me. On 300 subjects, normalization gave hybrid 5.7% at k=5 against RRF’s 35.3%, and on Enron the same change lifted hybrid from 22.4% to 31.6%. RRF is the default in the skill now, by way of the score ranker processor, and the rank constant barely matters once you are there, with values of 10, 20 and 60 differing by under half a point. The effect is large enough to be worth checking on any corpus where one clause is frequently uninformative, because this is not a subtle degradation.

The corpus is the variable

Here is the finding I did not expect to be the headline, and the one that changed what the skill does before it searches anything. Whether this works on your index has almost nothing to do with the pipeline and almost everything to do with whether your documents describe people at all, and the two real corpora I measured the same way sit two orders of magnitude apart.

Court opinions carry 21.0 descriptive references per document across 98.7% of documents, because an opinion’s entire purpose is to recount what a person did while calling them “the defendant.” Mask the party’s name out and 76% of those opinions are still a description of somebody. Enron email, by contrast, carries 0.20 descriptive references per document. Role-reference language shows up in 0.39% of messages there, and sampling those found that they either name the person in the same sentence, describe a generic role, or describe somebody with no roster entry at all. The judgment pass flagged nothing on Enron at the default threshold, and it was right not to.

Those two corpora bracket the range and I measured nothing in between them, so the skill now runs an assessment before it searches anything, sampling the index and reporting its density against those reference points. It takes whatever index pattern the request already named, so it costs nothing to run first:

uv run python scripts/forget_me.py assess --index "logs-application-*"

The verdict comes back as rich, sparse, or absent, and the skill is required to report it alongside any empty result, which is the rule I care most about in the whole thing. “We found nothing” and “this data does not describe people” produce identical output while meaning opposite things to whoever reads the report, since the first is a clean bill of health and the second is a statement about your corpus that says nothing whatsoever about the person who filed the request.

Summary

Dividing the work this way held up well, with the Python handling retrieval and filtering while the agent handled the one step that comes down to judgment. On another skill I would go looking for that boundary first, since it is what determines whether an agent is adding anything a command line tool could not have done on its own.

The numbers around it need reading carefully. Precision of 1.00 and recall of 0.91 at the balanced threshold with no false positives across the eight decoy categories looks good, but all of it comes from the synthetic corpus, where the documents were written to be found. Those results show the agent can tell near-misses apart when the signal is there while saying nothing about how often it is there, and masking the two real corpora has a similar limit, since a sentence written without a name reads differently from one with the name taken out. What I would rather have is the kind of corpus the skill targets, meaning operational records like logs, tickets, and incident reviews, and I have yet to find a public example with an intact naming channel to supply the labels.

The evaluation took longer than the pipeline did and changed what the skill claims, which I did not expect going in. It retracted a headline result, turned up the fusion defect, and made clear that the corpus decides whether any of this is possible before the pipeline gets a vote. The skill is an illustration of the mechanics rather than a compliance product, since detection is probabilistic and whether a given erasure is required or complete is a call for counsel rather than a tool. The repository has the skill, the seed scripts, and an EVALUATION.md with the method and the numbers in more detail than this post.