RAG guide · 12 min

Build a reliable, citable RAG knowledge base

Connecting AI to documents does not automatically make answers accurate. Quality depends on corpus governance, permissions, chunking, retrieval, citations and tests that include questions with no answer.

An information specialist organises versioned sources connected to an AI answer with visible citations.
Key points

The short answer

Start with a limited corpus and a named owner. Remove stale versions, retain source metadata and permissions, test retrieval separately from generation, display cited passages and teach the system to say when the corpus is insufficient.

  • Govern the corpus
  • Test retrieval separately
  • Make citations verifiable

Turn files into usable knowledge

  1. Sorting documents with folders and an e-reader
    AI-generated illustration

    1. Define the scope

    List users, allowed questions, included documents and excluded decisions. Assign a corpus owner and an update frequency.

  2. AI-generated illustration

    2. Prepare sources

    Remove duplicates and obsolete versions; retain title, author, date, language and access level. Check extraction from tables, notes, scans and attachments.

  3. AI-generated illustration

    3. Enforce permissions

    Retrieval must never bypass document rights. Apply controls before retrieval, log useful access events and test with several user profiles.

  4. AI-generated illustration

    4. Tune retrieval

    Select chunking, metadata, lexical or vector search and passage count for the documents. First measure whether the right passages are found, independently from final prose.

  5. Editorial illustration

    5. Produce a traceable answer

    Show source title, version and supporting passage. Separate quotation from synthesis, prohibit invented references and return an explicit insufficiency message when evidence is weak.

  6. Document research with an atlas and books in a library
    AI-generated illustration

    6. Evaluate and maintain

    Test simple, ambiguous, contradictory, outdated and out-of-scope questions. Track retrieval precision, supported answers, permission failures, freshness and correction time.

Diagnose an answer citing the wrong passage

Separate three checks. Does the original contain the answer? Did conversion and chunking preserve it with its heading and reference? Did the system retrieve the passage and answer faithfully? Record each stage rather than changing the model when a lost table caused the import problem.

Test case: an exception in a footnote

Write a general rule and a footnote exception in a fictional document. Ask a question that depends on the exception. Keep the document, converted text, retrieved chunks and answer. Reject an answer citing the rule but omitting the exception, even if its link opens the right file. Use the decision record to document the fix and retest.

Set the unit before counting

Decide whether you are evaluating entire documents or chunks. Several chunks from one document do not count as several retrieved documents. Build a small corpus with known relevant items and keep the same result limit across candidates. The precision and recall calculator below separates irrelevant hits from missed documents; it does not grade generated answers.

Check the file and then the chunk

Prepare two versions of the same document and one file with the wrong extension. Check that ingestion identifies the format, rejects or reroutes the ambiguous case, and passes each chunk’s identifier, version and section downstream. Verify those fields in retrieval results before evaluating generated text.

Put the method to work

Library with books and a research notebook
AI-generated illustration

Practical case

Build a small corpus of twenty dated documents, including two that conflict and one restricted by user role.

Wooden desk with a laptop and monitor
AI-generated illustration

Evidence to keep

Test passage retrieval, citation attribution, access rights and replacement of an outdated document.

Creative studio with a camera and drawing tablet
AI-generated illustration

Make the decision

Expand the corpus only when answers lead to the right passage and the restricted document remains hidden.

Verification workshop

Three questions exposing incomplete retrieval

Prepare these fictional documents in an isolated test corpus. The outputs below are editorial expectations, not measurements of a tool.

Editorial illustration

Rule and exception

Input to prepare
A rule permits exchanges within thirty days; a separate paragraph excludes personalised products. Ask about a personalised item.
Expected result
The answer cites both passages and identifies the applicable exclusion.
If the check fails
Inspect retrieved fragments. If the exception is missing, repair retrieval before changing wording.
AI-generated illustration

Obsolete version

Input to prepare
Two documents share a title: an older fifteen-day version and an active thirty-day version. Add validity metadata.
Expected result
The answer uses the active version and identifies its cited document.
If the check fails
Check validity filtering and removal of stale fragments. Do not rely on the model to guess the version.
Open desktop computer showing memory and cooling
AI-generated illustration

Absent answer

Input to prepare
Ask about international delivery when the corpus only describes local exchanges.
Expected result
The system states that information is missing and what requires another source.
If the check fails
Keep this as an abstention case; reject an answer inventing a policy.

Decision rule. Accept the workflow only when each decisive claim links to current, authorised evidence. Replay all three cases after changing fragmentation.

Four layers to control

AI-generated illustration

Corpus

Are documents correct, current, readable and attributed?

AI-generated illustration

Access

Does each person retrieve only material they may view?

AI-generated illustration

Retrieval

Do useful passages rank above merely similar passages?

Monitor showing enlarged leaf detail, pen tablet and magnifier in a night studio.
AI-generated illustration

Answer

Is every material claim supported and easy to verify?

6 starting points

Document tools, platforms and local options

These families cover source-grounded notebooks, enterprise platforms and local components. Verify connectors, permissions, data residency and citation mechanisms.

How is this selection produced?

These profiles are linked to the guide’s subject in our editorial catalogue. Inclusion is neither a ranking nor a performance measurement on your documents. Methodology.

Explore the full category

Related tool families

Frequently asked questions

How does RAG differ from training?

RAG retrieves passages at answer time without necessarily changing the model. Training adjusts behaviour or parameters and requires a different data and evaluation lifecycle.

Is a vector database always required?

No. Lexical search can work better for exact references. Hybrid retrieval is often useful, but it should be justified by tests on the real corpus.

How should expired documents be handled?

Keep a validity date, owner and archival rule. Remove the version from the active index while retaining records when documentary obligations require them.

The references below expand on the concepts and checks discussed. Scenarios and trial frameworks remain editorial proposals; provider documentation describes its own product rather than an independent benchmark.

Source note

Judge a RAG knowledge base by the passages retrieved

In RAG, answers first depend on indexed documents and passages retrieved for the question. Microsoft describes retrieval, context augmentation and generation. Index documents with identifiable titles, versions and URLs so each answer can be traced to the source actually used.

Test representative questions, including questions the knowledge base cannot answer. Check the passage’s relevance before answer fluency. The model can err even with context; clickable citations and abstention when evidence is insufficient help review.

Separate two possible failures: the correct passage was not retrieved, or it was available but misused in the answer. Keep expected and retrieved passages for each question. This avoids changing model instructions when the defect lies in the index or corpus.

Include questions involving a precise date, an exception and missing information. Check conflicting document versions: identify the version used and disclose the conflict instead of merging them. For private corpora, verify access rights during retrieval.

Corpus
Current documents, access rights and source identifiers.
Retrieval
Relevant passages for representative questions.
Answer
Verifiable citations and abstention when sources are missing.

Is a displayed citation enough to guarantee an answer?

Open the passage and check whether it supports the exact claim, with its conditions and date. A reference can be relevant to the topic without proving the conclusion.

Try the reasoning

The correct answer exists in the RAG corpus but no relevant passage is retrieved. Where should the defect be isolated?

Reasoning: Test retrieval separately from writing. An incorrect answer can originate in indexing, retrieval or generation.

Primary source : Microsoft Learn — Retrieval augmented generation and indexes

Official sources

Does a search retrieve the right documents?

Evaluate retrieval before judging the generated answer. In a bounded corpus, identify relevant documents for a question, then count relevant hits, irrelevant hits and missed relevant documents. Count each document once and use the same maximum result count when comparing two systems.

Relevant documents and retrieved documents The overlap represents relevant documents retrieved. Missed relevant documents are on the left; irrelevant results are on the right. Areas are schematic and not proportional to counts. Missed Relevant Irrelevant 231
Counts follow your inputs; areas remain schematic. Fictional initial values: five relevant documents, three retrieved, plus one irrelevant result.
Precision: 75% · Recall: 60% · F1: 66.7%

Calculation runs locally in your browser without storage. Without JavaScript, use the formulas below; displayed values remain those of the example.

Development workstation with a code screen
AI-generated illustration
Precision: avoid noise
Relevant retrieved ÷ all retrieved documents. Here: 3 ÷ (3 + 1) = 75%.
AI-generated illustration
Recall: avoid omissions
Relevant retrieved ÷ all relevant documents in the corpus. Here: 3 ÷ (3 + 2) = 60%.
AI-generated illustration
F1
2 × relevant retrieved ÷ (2 × relevant retrieved + irrelevant retrieved + relevant missed). Here: 6 ÷ 9 ≈ 66.7%.

A zero denominator produces “undefined”. Without an inventory of relevant documents, recall is unknown: do not infer it across the entire web. These measures assess neither answer truthfulness nor the severity of a missed document; omitting a decisive exception remains a blocking error.

Metric definitions: scikit-learn documentation

Continue with another guide

Keep evidence from this trial

Connect this material to a decision and a retest condition. Prepare a decision record →