RAG in practice: when to use it (and when not to)

RAG has become the default answer to everything. "How do I make the AI know about my data?" — "RAG!". "How do I reduce hallucination?" — "RAG!". And most of the time that's right — but RAG is also one of the most misused things around right now, thrown at problems that don't call for it. This post is the honest guide: what RAG really is, when it solves your problem, and when you're building expensive infrastructure for nothing.

What RAG is, without the hype

A language model knows what was in its training data and nothing beyond that. It doesn't know your internal documents, your catalog, your company's policy. And its context is limited — you can't just paste 10,000 pages into the conversation.

RAG (Retrieval-Augmented Generation) solves this by fetching, at question time, the relevant pieces of information and injecting them into the model's context. The flow:

1. You break your knowledge base into pieces (chunks) and generate embeddings — numerical representations of the meaning of each piece.
2. You store those vectors in a vector database.
3. When a question comes in, you turn it into a vector and search for the most semantically similar pieces.
4. You inject those pieces into the prompt and the model answers based on them.

The result: the LLM answers about your data, up to date, citing the source — without making things up. It's "giving the model the right cheat sheet before the exam."

When RAG actually solves your problem

RAG shines when you have a lot of external knowledge, that changes, and that the model needs to consult:

• Knowledge base / support: hundreds of articles, manuals, FAQs. The agent finds the right passage and answers accurately.
• Documents that change: policies, catalogs, prices. Since RAG doesn't train anything, you update the document and the answer changes instantly.
• Private knowledge: your company's data that was never in any training set.
• The need to cite the source: RAG knows where it pulled the answer from, which gives you traceability.

If your problem is "the AI needs to answer based on a lot of my documents that change," RAG is the tool.

When NOT to use RAG (the part nobody talks about)

Here's the counterpoint that saves months of unnecessary work:

If the information fits in the context, you don't need RAG. Modern models have huge context windows. If your knowledge is a 20-page document, put it in the prompt and be done. RAG for that is using a cannon to kill a fly.

If the answer comes from a system, use function calling, not RAG. "What's the status of my order?" is not a semantic-search question — it's a database query. The right tool is function calling: the agent calls get_order(id) and gets the exact data. RAG would search for "documents similar to the question," which is wrong for structured data.

If you don't have many documents, RAG is over-engineering. Embeddings, a vector database, an ingestion pipeline, reindexing — that's real infrastructure. It only pays off when the volume justifies it.

The filter question: does my answer depend on searching for meaning across many texts, or on looking up a specific piece of data? If it's searching texts, RAG. If it's looking up data, function calling.

The details that decide whether your RAG works

A badly built RAG is worse than no RAG — it brings back the wrong piece with confidence. What separates a RAG that works:

• Smart chunking. How you split the documents matters more than it seems. Chunks that are too large dilute relevance; too small lose context. Respect the document's structure (sections, paragraphs).
• Embedding quality. The embedding model defines what "similar" means. A good embedding model is half the battle.
• Retrieve enough, not the maximum. Injecting 50 chunks clogs the context and confuses the model. Retrieve the few most relevant ones.
• Reranking. A second pass that reorders the results by real relevance greatly improves precision.
• Evaluate. If you don't measure whether the RAG is bringing back the right passages, you don't know whether it works. Measure it.

The product view

RAG is not a goal — it's a means, like every AI technology. The mistake I see most is teams building an elaborate RAG pipeline before asking whether the problem calls for it. Often, what solves it is function calling into a system, or simply putting the document in the context. RAG comes in when the knowledge is large, textual, and changing — and then it's powerful.

As always: the right question isn't "which trendy AI technique am I going to use?". It's "what's the problem, and what's the simplest tool that solves it?". Sometimes it's RAG. Often it isn't. Knowing the difference is what separates the people who ship from the people who just follow hype.

I build AI systems that solve the right problem with the right tool — RAG, agents, integration — in production. If you're not sure whether you need RAG, let's talk.