How AI assistants answer from your own documents (retrieval-augmented generation, explained)

The short answer

A general AI chatbot answers from what it learned in training, and it knows nothing about your leave policy, your product specifications or your contracts. Retrieval-augmented generation, usually shortened to RAG, adds a search step: before the model answers, the system finds the most relevant passages in your approved documents and asks the model to answer from those passages, with citations.

Done well, it turns a long search into a cited answer in seconds. It reduces made-up answers but does not eliminate them. A good assistant also respects permissions, keeps its sources current, says "I could not find that" when it should, and is tested on real questions before anyone relies on it.

How RAG works, step by step

1. Collect the sources. Approved documents such as policies, manuals, procedures and contracts are gathered from where they live: SharePoint, file shares, a document system or a database.
2. Split and index them. Each document is split into passages small enough to search precisely. Each passage is indexed by its words and by an embedding, a numerical representation of meaning that lets the system match "time off" to a question about "vacation". Its permissions, owner and date are stored with it.
3. Retrieve. When someone asks a question, the system searches for the passages most likely to answer it, filtered to what that person is allowed to see.
4. Generate. The question and the retrieved passages go to the language model, with instructions to answer only from those passages and to cite them.
5. Show the sources. The answer appears with links to the passages used, so the person can open the original and check.

Microsoft's Azure AI Search documentation describes the same pattern as grounding a model's responses in your own content, and lists challenges that include understanding vague questions, reaching content across several systems, sending only the most relevant passages and enforcing access control (Microsoft Learn).

Most of the quality comes from steps 1 to 3. If the right passage is never retrieved, the best model in the world cannot use it.

Why it reduces made-up answers, but does not eliminate them

RAG helps because the model works from your text rather than from memory, and because citations let people check. OWASP lists retrieval-augmented generation as a way to reduce hallucination and misinformation, alongside cross-checking against trusted sources and human oversight (OWASP).

It still goes wrong in predictable ways:

• The wrong passage is retrieved, such as last year's version of a policy or a similar clause from a different contract.
• The right passage is missed, because the question uses different words or the answer is spread across several documents.
• The model fills the gaps, blending retrieved text with general knowledge or misreading a table.
• The source itself is wrong or out of date. RAG faithfully repeats whatever you feed it.

So design for it: the assistant should say when it cannot find an answer, show citations every time, and leave answers that matter (legal, financial, safety or HR decisions) for a person to check.

Permissions and keeping sources current

An assistant that searches everything for everyone is a data leak with a friendly interface. OWASP warns that weak access controls in retrieval systems can let a model reveal sensitive content to the wrong user, and that documents can carry hidden instructions; it recommends permission-aware retrieval and validating sources before they are added (OWASP). In practice:

• Retrieval applies the same access rights as the source system, checked on every question.
• Sensitive collections, such as HR files, salary data and legal matters, are excluded or separately controlled.
• Documents are checked before they are indexed, including for hidden text.
• Sources are re-indexed on a schedule or when they change, and retired documents are removed.
• Every source has an owner, and administrators can see what is included and when it was last updated.

Stale content is the quality problem we see most often: two versions of the same procedure, both indexed, giving two different answers.

When Microsoft 365 Copilot is enough

Microsoft 365 Copilot grounds its answers in your emails, chats and documents, and Microsoft states that it only surfaces organizational data that each user has at least view permission to see (Microsoft Learn). That makes tidy permissions essential before rollout, as our Copilot readiness guide explains.

Many organizations use both: Copilot for everyday productivity, and a custom assistant for one high-value collection of documents.

What good evaluation looks like

"Not found" questions are useful in their own right: they show you where your documentation has gaps.

Typical first use cases

• A policy library. Staff questions about HR, finance, travel and IT policies, answered with a link to the clause. High volume, low risk and easy to check.
• Product and procedure manuals. Sales, service and operations teams finding specifications, troubleshooting steps and procedures without chasing the one person who knows.
• Contract clause search. Finding renewal, termination, liability or privacy clauses across agreements, with the clause quoted and linked. A lawyer still interprets it.

When to bring in help

A small trial with an off-the-shelf tool on non-sensitive documents is a fine way to learn. If your content lives in Microsoft 365 and permissions are in good order, Copilot may be all you need.

Bring in a professional when the assistant will reach confidential or personal information, when permissions are complex, when sources span several systems, or when people will act on its answers. PIPEDA and provincial privacy laws, such as Quebec's Law 25, apply to personal information in the documents an assistant searches (this is general information, not legal advice).

With our generative AI development service we define the questions, sources and permissions with you, build retrieval that respects access rights and cites every answer, test it against your real questions before launch and monitor accuracy and running costs after it. You get an assistant your staff can check, and a team accountable for keeping it right. RAG works the same way with an open-weight model on your own servers when documents must not leave your environment.

Sources and further reading

Product capabilities and guidance change. These are the primary sources this article relies on, checked on the review date above.

1. RAG and generative AI (Azure AI Search), Microsoft Learn
2. Data, privacy and security for Microsoft Copilot, Microsoft Learn
3. LLM08:2025 Vector and Embedding Weaknesses, OWASP Gen AI Security Project
4. LLM09:2025 Misinformation, OWASP Gen AI Security Project

This article is general information, not legal, accounting or security advice for your specific situation. Examples are hypothetical unless stated otherwise.