Retrieval-augmented generation has become a standard pattern for building LLM applications that need to reference specific knowledge bases. The basic architecture is straightforward: retrieve relevant documents based on the query, inject them as context into the prompt, generate a response using that context. But making this work well requires thoughtful prompt engineering, particularly around how you use the limited context window.

The fundamental constraint is token budget. Current LLMs have context windows ranging from a few thousand to hundreds of thousands of tokens, but just because you can stuff 100k tokens into the prompt doesn’t mean you should. Cost scales with token count, latency increases, and more importantly, LLM performance often degrades when the context window is filled with excessive information.

The challenge is that retrieval systems often return more context than you need. A semantic search might return 10-20 documents that match the query. If you concatenate all of them into the prompt, you’re using thousands of tokens, most of which aren’t actually necessary for answering the question. The LLM has to sift through all that context to find the relevant parts, which works but isn’t optimal.

Better approaches involve being selective about what context you include. Rank retrieved documents by relevance score and include only the top N. Use chunk-level retrieval rather than full documents, so you’re adding paragraphs instead of pages. Implement a token budget for context and truncate when you hit it, ensuring you always have room for the actual query and response.

The prompt structure matters for helping the LLM use context effectively. Simply dumping retrieved text into the prompt and asking a question doesn’t give the model clear instruction about how to handle the context. More effective prompts explicitly frame the context and instruct the model on how to use it.

A pattern that works well: start with clear instructions about the task, then provide the retrieved context in a clearly demarcated section, then present the user query, then provide explicit instructions about how to generate the response using the context. This structure helps the model understand what information is available and how it should be applied.

For example: “You are answering questions based on a knowledge base. The relevant context is provided below. Use only information from the provided context to answer the question. If the context doesn’t contain enough information to answer completely, say so.”

That framing does several things. It establishes the task type. It indicates where the relevant information is. It sets expectations about using only provided context (important for preventing hallucination). And it provides fallback behavior when context is insufficient.

The specificity of that instruction matters. Without explicitly saying “use only information from the provided context,” many models will supplement with their training data knowledge, which might be outdated or incorrect for your specific use case. The more explicit you are about the boundaries, the more reliably the model stays within them.

Context formatting also affects how well the model can use it. Unstructured text dumps are harder to parse than structured formats. If your retrieved documents have metadata—titles, source information, dates—include that in a consistent format. It helps the model understand what each piece of context represents.

I’ve seen significant improvements from adding simple structure like:

Document 1 [Source: Annual Report 2025, Page 12]:
[content]

Document 2 [Source: Policy Handbook, Section 3.2]:
[content]

This makes it easier for the model to reference specific sources in its response and helps it weight information appropriately based on source type.

The ordering of context documents isn’t random. Models exhibit recency bias—information presented later in the prompt has more influence than information presented earlier. If you have a relevance ranking from your retrieval system, put the most relevant documents last, just before the query. This positions the best information in the most influential part of the context.

Some RAG systems implement iterative refinement: retrieve initial context, generate a draft response, identify gaps, retrieve additional specific information, refine the response. This uses the context window more efficiently than front-loading everything, because you only add context that turns out to be needed.

The tradeoff is latency—multiple retrieval and generation steps take longer than a single pass. For use cases where quality matters more than speed, it’s worthwhile. For interactive applications where sub-second response time is important, single-pass RAG with well-designed prompts is more appropriate.

Another consideration is handling contradictory context. Retrieval systems sometimes return documents that conflict with each other—different policy versions, inconsistent data, competing viewpoints. Your prompt should give the model guidance on how to handle this.

You might instruct it to: note the contradiction and present multiple perspectives, prioritize more recent information over older, or flag the inconsistency and request clarification. Without explicit instruction, the model will make arbitrary choices about which information to use.

Token counting is important for staying within budget. Most LLM APIs charge by token, and if you’re generating responses at scale, cost control requires managing prompt size. Implement token counting before sending prompts, and truncate or filter context to stay under your budget.

But don’t just truncate at arbitrary boundaries. If you need to reduce context, remove entire documents or chunks rather than cutting documents mid-sentence. Partial context is worse than no context—it can create misleading fragments that the model misinterprets.

For systems handling varied query types, dynamic prompt construction helps. Different queries need different amounts of context and different instruction framing. A factual lookup needs less instruction than a complex analytical question. Build prompts programmatically based on query characteristics rather than using a single template for everything.

The instruction about what to do when context is insufficient matters more than many implementations acknowledge. Models default to generating plausible-sounding responses even when they don’t have good information. Explicitly instructing them to admit insufficient context prevents confident-sounding hallucinations.

I use phrasing like: “If the provided context doesn’t contain enough information to fully answer the question, explain what information is available and what’s missing. Don’t make assumptions or use information not present in the context.”

This produces responses that accurately reflect knowledge gaps rather than papering over them with generated text. Users can then refine their query or understand the limitations of the available information.

Testing prompt variations with consistent query sets helps identify what actually improves performance. Change one aspect of the prompt structure, run the same queries, compare outputs. Prompt engineering is empirical—what works depends on your model, your data, and your use case. Best practices provide starting points, not final solutions.

For anyone building RAG systems, spending time on prompt design pays off more than optimizing retrieval algorithms in many cases. A mediocre retrieval system with excellent prompts outperforms excellent retrieval with mediocre prompts. The prompt is where you translate retrieved information into useful responses, and that translation quality determines user experience.