Event-Driven Document Processing on AWS Lambda

Most of the document processing pipelines I get called in to look at started as a cron job. Something polls a bucket every five minutes, picks up whatever appeared, and works through it in a loop. That holds up fine until the day somebody uploads forty thousand files at once, and then you find out that the loop is single threaded, the polling interval has become meaningless, and nobody can tell you which files were processed because the only record is a log line.

Event driven architecture is the standard answer, and on AWS the default assembly is Lambda with a queue in front of it. The individual services are easy enough that the architecture looks trivial in a diagram, which is exactly why so many of these end up broken in the same four or five ways. What follows is the shape I keep coming back to and the decisions inside it that actually matter.

Put a queue between the event and the function

You can wire S3 notifications directly to a Lambda function, and for a toy pipeline that is genuinely fine. I would not build anything I cared about that way, because it leaves you no place to stand when something goes wrong.

The route I’d rather take is S3 to EventBridge to SQS to Lambda. EventBridge earns its place by letting you route on the content of the event, so PDFs can go to one queue and images to another without a dispatcher function existing solely to look at file extensions and re-publish. Adding a second consumer later is a rule change rather than a code change, which is the sort of thing that decides whether a pipeline stays clean after two years of feature requests.

The queue is the part people skip, and it’s doing more than buffering. It gives you a visible backlog, which means an operator can see that forty thousand files arrived and that the system is nine minutes behind rather than broken. It gives you retries with backoff that you didn’t write. It gives you somewhere for poison messages to go. Most importantly it decouples the rate work arrives from the rate you’re able to do it, and that gap is where nearly all the pain in these systems lives.

Lambda scales, and your downstream doesn’t

This is the failure I see most often and it’s the one that does real damage. It also tends to arrive as a surprise, because nothing in the Lambda configuration hints that scaling out is a thing you might want less of.

Lambda’s whole appeal is that it scales out without you asking. A thousand messages land on the queue and AWS will happily run a thousand concurrent executions. If each of those opens a database connection, or calls a third party API with a rate limit, or writes to a search cluster sized for normal traffic, you have built an extremely reliable way to take down the systems around you. The Lambda function is fine throughout. Everything it talks to is not.

The fix is boring and you should apply it before you need it. Set maximum concurrency on the event source mapping so the queue only ever drives that many parallel executions. Pick the number from what the downstream can absorb rather than from what feels fast, and treat backlog as the normal, healthy state during a burst rather than an incident. A queue that drains in twenty minutes without breaking anything is a better outcome than one that drains in ninety seconds and pages three teams.

Reserved concurrency on the function is the blunter version of the same idea, and it does double duty by guaranteeing that this workload cannot consume the whole account’s concurrency pool and starve everything else in the region. Both are worth understanding, because the account level limit is shared and it will find you eventually.

Assume every message arrives twice

SQS standard queues are at least once delivery. That is not an edge case to handle someday, it’s a property of the system you are building on, and any handler that isn’t idempotent will eventually do something twice.

The mechanism I reach for is a conditional write to DynamoDB keyed on the event identifier. Before doing the work, try to insert a record for that event with a condition that it must not already exist. If the condition fails, some other invocation already handled this and you can return successfully without repeating the work. It costs a single small write and it converts a whole category of subtle duplication bugs into a non-issue.

import boto3
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table("processed-events")

def already_processed(event_id: str) -> bool:
    try:
        table.put_item(
            Item={"eventId": event_id, "ttl": expiry_timestamp()},
            ConditionExpression="attribute_not_exists(eventId)",
        )
        return False
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return True
        raise

Put a TTL on those records so the table doesn’t grow forever, and set it comfortably longer than your maximum retry window. If you’d rather not maintain this yourself, the idempotency utility in Powertools for AWS Lambda implements the same pattern behind a decorator, including caching the previous response so a duplicate returns the original result rather than an empty success.

One bad message shouldn’t resend the batch

By default, a Lambda reading a batch of ten SQS messages either succeeds for all of them or fails for all of them. One malformed document in the batch means the other nine come back and get processed again, which is unpleasant on its own and considerably worse if you skipped the previous section.

The setting that fixes this is partial batch responses . You enable ReportBatchItemFailures on the event source mapping and return the message identifiers that failed, and only those go back on the queue. It’s two lines of code and one line of Terraform, and I have never regretted turning it on.

resource "aws_lambda_event_source_mapping" "documents" {
  event_source_arn = aws_sqs_queue.documents.arn
  function_name    = aws_lambda_function.processor.arn
  batch_size       = 10

  function_response_types = ["ReportBatchItemFailures"]

  scaling_config {
    maximum_concurrency = 20
  }
}

Behind the queue you want a dead letter queue with a redrive policy, so a message that fails repeatedly stops cycling and lands somewhere an operator can look at it. The part teams forget is that a DLQ with nothing watching it is just a slower way to lose data. Alarm on its depth being greater than zero, and make redriving it a normal operational task rather than an archaeology project.

Choreography or orchestration

Everything above is choreography. Components emit events, other components react, and no single thing knows the whole flow. That’s the right model when the steps are genuinely independent and you want to add consumers without touching existing code.

It’s the wrong model when the work is a sequence with branches, retries at different points, and a meaningful notion of the whole thing having failed. Extract text, then classify, then index, then notify, with different handling at each stage, is a workflow rather than a set of independent reactions. Trying to express that in events gives you logic distributed across six functions and no single place to answer whether document 12345 finished.

Step Functions is the tool for that half, and the honest split is that events are for decoupling across boundaries and state machines are for sequences within one. Most real pipelines want both, with events carrying work between subsystems and a state machine running the multi-step processing inside each one.

Where Lambda stops being the answer

Lambda has a fifteen minute ceiling and ten gigabytes of memory, and those two numbers draw the boundary of what belongs here. Neither is a limit you should be designing right up against, so treat them as somewhat closer than they look.

Document processing tends to run into the time limit through one enormous input rather than through steady load, so the usual fix is decomposition. Split a thousand page PDF into per-page messages and process them independently, which is better architecture anyway because it makes retries cheap and progress visible. If the work genuinely cannot be split, that job wants Fargate or Batch and no amount of tuning will change it.

Cost has a crossover point too. Lambda is dramatically cheaper than a running container for spiky, occasional work, and it stops being cheaper somewhere around the point where you’re processing continuously. If your concurrency chart is a flat line rather than a series of spikes, price the equivalent Fargate service before you renew the assumption. I’ve seen pipelines running Lambda at steady state for years because the original workload was bursty and nobody revisited it after the traffic pattern changed.

Heavy model inference is the other case. If each invocation loads a large model, you pay that cost on every cold start, and you’ll spend more effort fighting initialization time than the serverless model saves you. That work wants something with a warm process and a loaded model sitting in memory.

What I’d check first

If you already have one of these and you want to know whether it’s sound, there are four questions that separate the pipelines that survive a bad day from the ones that don’t. Is there a queue between the event source and the function, or is the event source invoking directly. Is the concurrency capped at something the downstream systems can actually absorb. Is the handler idempotent, with something durable recording what’s already been done. And is anything alarming on the dead letter queue.

Most of the broken pipelines I’ve looked at answered no to at least three of those. All four are considerably cheaper to fix on a quiet afternoon than during the incident that eventually reveals them.

If you’re building or inheriting something like this and want another set of eyes on it, please reach out.