Your RAG Pipeline Has a Search Problem

re:Invent dropped a lot of RAG plumbing this year. Bedrock got a Rerank API on December 1st. Knowledge Bases picked up GraphRAG in preview on Neptune Analytics, and RAG evaluation in preview alongside it.

Good stuff. None of it is what’s wrong with your RAG application.

What’s wrong is retrieval. RAG is a search problem with a language model on the end of it, and the people building these things mostly arrive from an ML background rather than a search one. So the search half gets treated as a solved commodity, which it very much is not.

The model only sees what you hand it

Retrieval runs first. By the time the model is involved, the set of facts available to it is already decided and can’t be added to.

So when the passage that answers the question doesn’t make it into the retrieved set, you aren’t going to get a good answer. You get one of two bad ones. Either the model says it doesn’t know, which users read as the product being broken, or it fills the gap from whatever it absorbed during pretraining and says it with total confidence. That second one is where a lot of the hallucinations people complain about actually come from. The model isn’t malfunctioning. It’s answering a question it was never given the material for.

You can’t prompt your way out of that, though people try. Bigger model, sterner instructions about not making things up, lower temperature, more examples. Every one of those operates on the last stage of a pipeline that lost the information three stages earlier.

The default is five chunks

Bedrock Knowledge Bases returns five results by default. numberOfResults, if you’re calling Retrieve or RetrieveAndGenerate yourself.

Five is a perfectly reasonable default. It’s also a hard cap on what your application can possibly know about any given question, and I don’t think many people building on it have looked at it that way.

Which points at a different question to ask. Not whether the answers were good, but whether the answer was ever in the box to begin with. That one you can measure.

Recall@5 of 0.62 means 38 percent of your users get a bad answer regardless of anything you do downstream. It takes an afternoon to compute that number. Almost nobody does.

Raising k helps, and it isn’t free. More chunks means more tokens and more cost, and past a certain point the right chunk ends up buried in the middle of a large context where the model skims over it. Retrieving something and having the model use it are two different things. That’s what reranking is for, further down.

Chunking is a retrieval decision

Chunking gets treated like preprocessing. Plumbing you configure once on the way to the interesting part.

It’s the biggest lever you have. It decides what a document even is from the retriever’s point of view, and everything else happens downstream of that.

Bedrock’s default is roughly 300 tokens split on sentence boundaries. If your content is FAQ shaped, question and answer sitting near each other, that’s fine and you can skip the rest of this section. If it’s procedural documentation that runs two pages, it’s actively bad. The answer gets sliced in half, neither half looks especially relevant on its own, and so neither half ranks.

Hierarchical chunking is what most documentation corpora actually want. You retrieve on small child chunks, which embed precisely, and the parent gets swapped in before the content reaches the model. One thing to watch is that it changes what numberOfResults means, since children sharing a parent collapse into a single result and you get back fewer than you asked for.

Semantic chunking splits on embedding distance instead of a token count, tuned with a buffer size and a breakpoint percentile. It comes closest to dividing a document the way a person would. It also runs a model over your entire corpus at ingest, which on a large corpus is a real line item.

You can also turn chunking off and treat each document as one chunk. That only makes sense if you’ve split the documents yourself first, which is a legitimate thing to do when you understand your content better than a generic chunker does. Often you do.

There’s no way to pick between these by reasoning about it. You have to try them and measure, which is the part everyone skips.

Measure retrieval separately from the answer

Teams that evaluate at all usually evaluate end to end. Ask it a question, look at the answer, call it good or bad.

That’s the least informative measurement available, because a bad answer hides two unrelated failures. Either the retriever never surfaced the right passage, or it did and the model made a mess of it. The fixes have nothing to do with each other. Chunking and hybrid search address the first. Prompting and model choice address the second. Not knowing which one you have is how a month disappears into the wrong half of the system.

The RAG evaluation that landed in Knowledge Bases at re:Invent does split them, which is the best thing about it. Retrieval gets scored on context relevance and coverage. Generation gets correctness, completeness, and faithfulness, which is the hallucination check, along with responsible AI metrics like harmfulness and stereotyping. It’s in preview as I write this, so plan accordingly.

I’d build your own harness anyway. Not instead of that one, alongside it. You need real questions, pulled from support tickets or chat logs or wherever your users actually are, and for each one a note on which passage in the corpus genuinely answers it. Then recall@k against that. Fifty questions is enough to be useful. It’s a day or two of boring work, and afterward every argument about chunking has a number attached to it instead of opinions.

Be a little careful with the LLM-as-a-judge scoring underneath these evaluations. It’s a real technique and it scales in a way human raters never will. It’s also a model grading a model, with its own blind spots, some of which line up uncomfortably well with the blind spots of the thing being graded. Score a sample by hand and see whether you agree with the judge before you start making decisions from what it says.

Now the knobs make sense

Once retrieval is measurable, the features AWS keeps announcing stop being a menu of things to try and start being answers to specific problems.

• Hybrid search. Set overrideSearchType to HYBRID and the knowledge base searches raw text alongside the embeddings. Semantic retrieval is bad at exact strings, and error codes, part numbers, and policy IDs are exact strings that users paste in character for character. The vector store has to support it, so OpenSearch Serverless or Aurora with a filterable text field.
• Metadata filtering. Cheapest available win and the most commonly skipped, because it means populating metadata at ingest and that’s tedious. If the question is about the 2024 handbook, filter to the 2024 handbook rather than hoping the embedding works it out.
• Reranking. The fix for retrieving broadly and then drowning the model in it. Pull back fifty candidates, rerank with Amazon Rerank 1.0 or Cohere Rerank 3.5, pass along the top few. You get the recall of a large k with the precision of a small one, at the cost of another call in the request path.
• GraphRAG. Genuinely interesting for questions that need facts connected across documents, since it walks out from the chunks it matched instead of just handing them over. It’s also preview, it means running Neptune Analytics, and it targets one specific failure. Check that multi-hop questions are where you’re actually losing before you go there.

What I’d do first

Roughly in this order.

1. Get fifty real questions. Support tickets, chat logs, sales calls. Not questions your team invented, those are always easier than the real ones.
2. For each one, find the passage that answers it. By hand. It’s tedious and there’s no way around it.
3. Measure recall@k on what you have now, at 5, 10, and 20.
4. If recall@5 is poor, that’s the whole project. Chunking first, then hybrid search, then metadata filters.
5. Prompts, models, and reranking come after the right chunks are reliably showing up. Not before.

Most of the RAG work I’ve come across runs that list bottom to top. It starts at five, stays there for a month, and never gets to one through three at all.

Summary

The generative half gets the attention because it’s the half that talks. The retrieval half is where the quality is decided, and it’s a search problem with all the ordinary search problems in it. Chunking, recall, exact match, ranking. AWS shipped good tools for it at re:Invent, and they’re much easier to use well once you can tell whether they’re helping.

Start with the question set.

If you’re building RAG on AWS and the answers aren’t what you expected, please reach out. It’s usually retrieval.