Jeff ZemerickContact Jeff

← All posts

Redacting PII at Scale with Phileas

Data PrivacyNLP

Sensitive data has a way of ending up where it shouldn’t. It leaks into application logs, support tickets, chat transcripts, free-text form fields, and the prompts and training data we feed to large language models. Names, Social Security numbers, credit card numbers, phone numbers, email addresses, and medical details all flow through systems that were never designed to protect them.

Phileas is an open-source project I founded to deal with exactly this problem. It finds and removes personally identifiable information (PII) and protected health information (PHI) from text, and it’s built to do it at scale. This post is an introduction to how it works and why redaction is harder, and more important, than it first looks.

Why redaction is harder than it looks

At a glance, redacting PII sounds like a job for a few regular expressions. Match a Social Security number, replace it with XXX-XX-XXXX, done. In practice it falls apart quickly.

Some PII has a predictable structure that regular expressions handle well: credit card numbers, phone numbers, ZIP codes, email addresses. But a lot of it doesn’t. Names, cities, and organizations have no fixed format, and the only way to recognize them reliably is with natural language processing. And even once you’ve found something sensitive, you have to decide what to do with it, and that decision depends on what the redacted text is for.

That’s the real challenge. Redaction isn’t one problem; it’s two: finding the sensitive data, and transforming it in a way that preserves whatever value the downstream consumer still needs.

Filters: finding the sensitive data

Phileas approaches the “finding” half with a set of filters, where each filter is responsible for identifying one type of sensitive information. There are filters for structured identifiers like SSNs, credit cards, phone numbers, and email addresses, and there are NLP-based filters for the things that have no fixed pattern, like people’s names.

Because every filter is independent and configurable, you enable only the ones you need. A healthcare system worried about PHI has very different requirements from a company scrubbing credit card numbers out of its application logs. You tune the set of filters to the data you actually have.

Take a support ticket like this:

Hi, this is Jane Doe. My account email is jane.doe@example.com and my
card 4111 1111 1111 1111 was charged twice this month. You can reach me
at (555) 123-4567.

Phileas’s filters identify each piece of sensitive data in that sentence: the name (Jane Doe), the email address, the credit card number, and the phone number. It then hands them off to be transformed.

Redaction strategies: deciding what to do with it

Once a filter identifies sensitive text, the more interesting question is what to replace it with. Simply deleting everything is easy, but it often destroys data you still need. Phileas supports a range of strategies so you can match the transformation to the use case:

  • Redaction. Replace the value with a placeholder, so John Smith becomes something like {{{REDACTED-name}}}. Simple and unambiguous.
  • Replacement. Substitute a realistic-looking fake value. This is useful when downstream systems or models need text that still looks like a name or a date in order to behave correctly.
  • Hashing and encryption. Transform the value in a way that removes the original but can still be matched or, where appropriate, reversed.

The right choice depends entirely on what happens next. Redacting for a public document is different from de-identifying a dataset you still want to analyze.

To make that concrete, here’s the same support ticket under each strategy.

Redaction replaces each value with a labeled placeholder:

Hi, this is {{{REDACTED-name}}}. My account email is {{{REDACTED-email-address}}}
and my card {{{REDACTED-credit-card}}} was charged twice this month. You can
reach me at {{{REDACTED-phone-number}}}.

Replacement swaps in realistic-looking fake values, so the text still reads naturally and downstream models still see something shaped like a name or a card:

Hi, this is Maria Alvarez. My account email is user837@example.net and my
card 4242 4242 4242 4242 was charged twice this month. You can reach me at
(555) 908-2231.

Hashing replaces each value with a deterministic token. The original is gone, but the same input always produces the same output:

Hi, this is a1b2c3d4. My account email is 7f9e2ac1 and my card 5e8842fb was
charged twice this month. You can reach me at 0c1d9a7e.

Consistency is the hard part

Here’s a subtle requirement that separates real redaction from a quick script: consistency. If Jane Doe appears five times in a document, or across a million documents, you often need it replaced with the same token every time. Analysts still need to see that the same person shows up in multiple records, even if they can no longer see who that person is.

For example, if a log stream mentions the same customer twice:

2025-06-17 09:14  user Jane Doe failed login from host web-03
2025-06-17 09:15  user Jane Doe locked out after 3 attempts

consistent replacement turns both mentions into the same token, so an analyst can still see it’s one person across both lines without seeing who it is:

2025-06-17 09:14  user person_8f21 failed login from host web-03
2025-06-17 09:15  user person_8f21 locked out after 3 attempts

Getting that right across large volumes of text, deterministically, is one of the things Phileas is designed to handle. Consistent replacement preserves the relationships in your data while still removing the identities.

Why this matters more in the age of LLMs

Redaction used to be mostly about compliance: HIPAA, GDPR, CCPA, and keeping PII out of logs. Those drivers haven’t gone anywhere. But large language models have added a new and urgent reason to care.

Every prompt you send to a hosted model, and every document you use to fine-tune or ground one, is a potential leak of sensitive data. Redacting PII before it reaches the model lets you take advantage of LLMs without handing over your users' personal information. So instead of sending the raw ticket to a summarization model, you send the redacted version:

Summarize this support ticket:
"Hi, this is {{{REDACTED-name}}}. My account email is {{{REDACTED-email-address}}}
and my card {{{REDACTED-credit-card}}} was charged twice this month..."

The model gets everything it needs to be useful, and none of the personal data ever leaves your control. The same engine that keeps SSNs out of your logs can keep them out of your prompts.

Built to scale

Phileas is designed to run over large volumes of text, in batch or in streaming pipelines, which is what “at scale” really means. It’s been adopted in commercial products, including the Graylog log management platform, where redacting sensitive data out of high-volume log streams is exactly the kind of workload it was built for. It’s available across multiple language ecosystems, so it can fit into the stack you already have.

Summary

Removing PII from text is a two-part problem: finding the sensitive data, which takes both pattern matching and NLP, and transforming it in a way that preserves the value your downstream systems still need. Phileas is an open-source engine built to do both, consistently and at scale, whether you’re scrubbing logs, de-identifying a dataset, or keeping personal data out of your LLM prompts.

You can find the project on GitHub. If you have a data privacy or redaction problem you’re trying to solve, use the Contact Jeff button up top.