RAG explained: How AI understands your documents without giving them away

The problem: AI knows nothing about your company

Updated on 23 September 2026: the former practical example is now labelled as a worked example with assumed values, and unsourced cost figures for fine-tuning have been removed.

Retrieval Augmented Generation (RAG) turns AI into an expert on your documents. Find out how vector databases, semantic search and source citations work.

Large language models, for example from OpenAI, Mistral AI or Anthropic, are impressive. They can summarise texts, answer questions, write code and explain complex relationships. But they have a fundamental weakness: They know nothing about your company.

This is due to the way these models are trained. An LLM (large language model) is trained on publicly available texts from the internet: Wikipedia, books, websites, forums. It knows German employment law, the capital of France and the syntax of Python. But it does not know:

• Your internal operations manual
• Your holiday policy
• The contract with your supplier
• Your standard operating procedures (SOPs)
• The minutes of your last board meeting

The obvious idea would be to simply retrain the model with your documents, known as fine-tuning. But fine-tuning has serious drawbacks:

• Costs: A fine-tuning run for a large language model involves considerable computing and staff effort, depending on the model and the volume of data, and has to be repeated with every update
• Data protection: With fine-tuning, your data is burned into the model's weights. You can no longer remove individual documents without retraining the entire model. This directly contradicts the right to erasure (Art. 17 GDPR)
• Hallucination: Fine-tuning does not reliably reduce hallucinations. The model does “know” more, but it can still output false information with high confidence, without it being traceable where the answer comes from
• Timeliness: Documents can change between fine-tuning and deployment. The model then works with outdated knowledge without this being apparent

A different approach is needed: one that combines the strength of language models (language understanding, text generation) with current company knowledge, without burning data into the model. That is exactly what RAG is. You can find a short definition in the glossary under RAG.

What is RAG? Simply explained

Retrieval Augmented Generation (RAG) was first described scientifically in 2020 by Patrick Lewis and colleagues at Facebook AI Research, now Meta AI (Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, in: Advances in Neural Information Processing Systems 33, NeurIPS 2020). The basic idea is elegantly simple:

Instead of making the model know everything, we give it the relevant information with each question.

Imagine a researcher in a library. The researcher (the language model) is intelligent and can understand complex relationships. But they have not memorised every book in the library. Instead, for every question they go into the library, pick out the relevant books, read the appropriate sections and then formulate a well-founded answer with source citations.

This is exactly how RAG works, in three steps:

Step 1: Search (Retrieval)

The user's question is converted into a mathematical representation (known as an embedding, more on this shortly) and compared with all documents in the knowledge base. The most relevant text passages are identified and retrieved. This step typically takes less than 100 milliseconds, even with hundreds of thousands of documents.

Step 2: Fetch (Retrieve & Rank)

The retrieved text passages are sorted by relevance and narrowed down to the most important ones. A reranking algorithm ensures that the truly relevant passages are at the top, not just those that happen to contain similar words. Typically, the 5 to 15 most relevant passages are selected.

Step 3: Generate

The language model receives the user's question together with the retrieved text passages as context. It then formulates an answer based on the sources provided. Every statement can be traced back to a specific document.

The key difference from fine-tuning: The documents remain external. They are not integrated into the model but retrieved when needed. This means: you can add, update or delete documents at any time. The change takes effect immediately, without the model having to be retrained.

Vector databases: How machines understand meaning

The heart of every RAG pipeline is the vector database. To understand why it is so important, you need to understand how machines “understand” language, or more precisely: how semantic similarity is calculated.

A text embedding is a mathematical representation of a text as a vector, a list of numbers, typically between 384 and 1536 dimensions. These numbers encode the meaning of the text, not its words. Two sentences that say the same thing have similar vectors, even if they use completely different words.

An example: the sentences “What is the holiday policy?” and “What rules apply to vacation absences?” do not share a single meaningful word. A classic keyword search would not find the second sentence when searching for “holiday policy”. A vector database, on the other hand, recognises that both sentences are semantically almost identical, because their embeddings lie close together in high-dimensional space.

The similarity between vectors is calculated mathematically from the angle in high-dimensional space. The closer two vectors are, the more similar the meaning of the texts they represent. Modern vector databases achieve response times in the millisecond range, even with millions of documents.

A modern open-source solution is used as the vector database, offering the following features:

• Millisecond search across millions of vectors thanks to efficient indexing methods
• Filtering capability: Combining vector search with classic filters (e.g. “Find similar documents, but only from the Finance department”)
• Tenant isolation: Each company gets its own collection. Data from different tenants can never mix
• Self-hosting: The entire vector database runs on your own server. No data leaves your infrastructure

The quality of embeddings is compared using the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al. 2023), which measures embedding models across several task types and languages, including classification, clustering and retrieval. For German-language documents, the retrieval scores matter most, because they describe exactly what a knowledge search has to do: find the right passages for a question.

RAG vs. fine-tuning: Why your data stays with the customer

The distinction between RAG and fine-tuning is not only technically relevant, it also has direct consequences under data protection law. Let us compare the two approaches side by side:

Fine-tuning: data becomes part of the model

With fine-tuning, your company documents are used to adjust the weights of a neural network. The information is burned into the model. Afterwards, it no longer exists as a separate record, but as distributed patterns across billions of parameters. This has several consequences:

• No targeted deletion possible: If an employee wants to exercise their right to erasure under Art. 17 GDPR, you cannot remove a single document from the model. You would have to retrain the entire model without this document, a process involving considerable effort
• Risk of data leakage: Fine-tuned models can be made to reproduce training data verbatim through clever prompting (so-called “training data extraction attacks”, Carlini et al. 2021). Confidential information can reach unauthorised users this way
• No tenant separation: If the same model is fine-tuned for several companies, there is a risk that knowledge from company A appears in answers for company B

RAG: data stays external and controllable

With RAG, the documents are stored in a separate database and only provided to the model when needed. The separation between model and data is fully preserved:

• Targeted deletion possible at any time: Removing a document from the vector database takes milliseconds. The next query will no longer use this document as a source. Art. 17 GDPR can be fully implemented
• No data leakage: The language model has no permanent access to your documents. It only sees the passages that are relevant to the current question. Prompting attacks cannot extract any training data, because no training data exists in the model
• Complete tenant separation: Each company has its own collection in the vector database. Access control takes place at database level, physically separated, not just logically
• Immediately up to date: When a document is updated, the old embedding vector is replaced by the new one. The next query immediately works with the current version

For companies operating under the GDPR, RAG is the architecturally cleaner approach. Fine-tuning can have advantages in certain scenarios, for example for adapting the model's style. For working with confidential company data, RAG is the obvious route: the documents remain outside the model, deletable and separated by tenant.

Source citations: every answer shows where it comes from

One of the biggest problems with AI systems is the so-called hallucination: the model generates an answer that sounds plausible but is factually wrong. With general knowledge, this is annoying; with business decisions, it can have disastrous consequences. Imagine an employee asks the AI about the correct notice period and receives a wrong answer.

RAG solves this problem through transparent source citations. Because the model bases its answer on specific text passages retrieved from the knowledge base, every statement can be traced back to its source document.

In practice, it looks like this:

• The user asks a question: “How many days of holiday am I entitled to?”
• The RAG pipeline finds the relevant section in the operations manual (page 23, section 4.2: “Holiday entitlement”)
• The model answers: “According to the operations manual (section 4.2), you are entitled to 30 days of holiday per calendar year based on a 5-day week.”
• Alongside the answer, the source is displayed: document name, section, page number. The user can open the original document with one click and verify the statement

This mechanism creates trust. The user does not have to trust the AI blindly. They can check every answer themselves. And if the model does not find a suitable source, it should answer honestly: “I could not find any information on this in the knowledge base.” An honest “I don't know” answer is infinitely more valuable than a hallucinated “answer”.

Transparency is not just a feature. It is a trust architecture. Companies that use AI for internal knowledge processes must be able to ensure that no false information circulates unchecked within the company. Source citations are the foundation for this.

Worked example: searching a 500-page manual

A worked example with assumed values makes the processes tangible. Imagine the following scenario:

A medium-sized company with 200 employees has an operations manual of 500 pages. It contains everything: working hours, holiday policies, travel expense policies, IT security regulations, compliance requirements, organisational charts, process descriptions. The manual is updated twice a year.

Until now: a new employee has a question about claiming business travel expenses. They search the 500-page PDF for keywords, find three possible passages, read the context of each, are unsure which rule currently applies, and finally ask a colleague who is not sure either. Assumed time required: 15 to 30 minutes for a simple question.

With RAG: the employee types into the AI assistant: “How do I claim expenses for a business trip in my own car?”

What happens in the background, with typical orders of magnitude assumed for this example:

• 0-50 ms: The question is converted into an embedding vector. At the same time, the system uses HyDE (Hypothetical Document Embeddings) to generate a hypothetical answer segment in order to improve the search
• 50-150 ms: The vector database searches all sections of the manual. Semantic search also finds matches under “travel cost reimbursement” and “mileage claims”, terms the employee did not use
• 150-300 ms: A reranking algorithm sorts the results by relevance. The travel expense policy (section 7.3) is identified as the most relevant match
• 300-2000 ms: The language model formulates a precise answer: “According to the travel expense policy (section 7.3), a flat rate of EUR 0.30 per kilometre is reimbursed for business trips in your own car. Claims are submitted using form F-RK-03 on the intranet. The following must be submitted: reason for the trip, route, odometer reading before/after the journey. Submission deadline: 4 weeks after the end of the trip.”

Assumed total duration in the example: under 2 seconds. The answer is precise and up to date, and refers to the source document. The employee can open the cited passage with one click and verify it.

Assuming an average of 3 such questions per week for a company with 200 employees, that makes 600 queries per week. If each of them takes only 30 seconds instead of 20 minutes, the calculated time saving is over 190 hours per week, the equivalent of almost 5 full-time positions. This is an example calculation with assumed values, not a measurement from customers.