Skip to content

How to Build a RAG Chatbot Over Your Own Documents: Concept, Architecture and Code Sketch

Gulab Chand Tejwani

Gulab Chand Tejwani

PMP · PL-300 · Microsoft Certified Trainer

· 12 min read

RAGGenerative AIChatbotLLMEmbeddingsVector Search
How to Build a RAG Chatbot Over Your Own Documents: Concept, Architecture and Code Sketch

A RAG chatbot answers questions from your own documents by searching them first and then asking a language model to answer using only what it found. You build one in five steps: split documents into chunks, turn each chunk into an embedding (a list of numbers that captures meaning), store the embeddings in a vector index, find the chunks closest to each question, and send those chunks plus the question to the model with an instruction to cite them. Retrieval-augmented generation is usually cheaper, faster to update and easier to trust than fine-tuning a model on your data.

I'm Gulab Chand Tejwani, a Microsoft Certified Trainer at TechNet Consultancy in Anand. RAG is the capstone project in our generative AI course, and it's also the most common AI feature businesses ask us to build. Here's how it works, a code sketch you can adapt, and the mistakes I see most.

What problem does RAG solve?

A language model knows what was in its training data, up to a cut-off date. It doesn't know your HR policy, your product manuals or last month's meeting notes. Ask it anyway and it may produce a confident, plausible and wrong answer.

You could paste the whole document into the prompt. That works for one short file. It breaks down with hundreds of documents: you hit the context limit, you pay for every token on every question, and answers get worse when the relevant paragraph is buried in noise.

RAG fixes this by sending only the few passages that matter. The model's job shrinks from "know everything" to "read these five paragraphs and answer", which is something models do well.

How does a RAG pipeline work?

There are two phases.

Indexing (done once, then whenever documents change):

  1. Load documents: PDFs, Word files, web pages, database rows.
  2. Chunk them into passages of a few hundred words, with a little overlap so sentences aren't cut in half at the edges.
  3. Embed each chunk with an embedding model. Similar meanings produce vectors that sit close together.
  4. Store the vectors with the chunk text and metadata (file name, page, date) in a vector store.
RAG indexing phase in four steps: load, chunk, embed, store
Phase 1 runs once, then again whenever documents change.

Answering (every question):

  1. Embed the question with the same embedding model.
  2. Retrieve the top few chunks by similarity, usually cosine similarity.
  3. Build a prompt containing the instructions, the retrieved chunks and the question.
  4. Generate the answer with a language model, including citations to the chunks.
RAG answering phase in four steps: embed question, retrieve, build prompt, generate with citations
Phase 2 runs for every question. The model only ever sees the few chunks you retrieved.

That's the whole idea. Everything else is making each step better.

A code sketch in Python

This sketch keeps the moving parts visible. It uses NumPy for the similarity search and two placeholder functions, embed() and generate(), that you connect to your provider of choice: OpenAI, Azure OpenAI, Anthropic, Google, or a local model through Ollama. Each provider's SDK has a call for embeddings and a call for chat; the shapes are similar.

import numpy as np
from pypdf import PdfReader

# --- connect these two to your AI provider ---------------------------------
def embed(texts: list[str]) -> np.ndarray:
    """Return one embedding vector per text, shape (len(texts), dims)."""
    raise NotImplementedError

def generate(prompt: str) -> str:
    """Send the prompt to a chat model and return its reply."""
    raise NotImplementedError
# ---------------------------------------------------------------------------

def load_chunks(path: str, size: int = 900, overlap: int = 150):
    chunks = []
    for page_no, page in enumerate(PdfReader(path).pages, start=1):
        text = " ".join((page.extract_text() or "").split())
        start = 0
        while start < len(text):
            chunks.append({"text": text[start:start + size],
                           "source": f"{path} p.{page_no}"})
            start += size - overlap
    return chunks

def build_index(paths: list[str]):
    chunks = [c for p in paths for c in load_chunks(p)]
    vectors = embed([c["text"] for c in chunks])
    vectors = vectors / np.linalg.norm(vectors, axis=1, keepdims=True)
    return chunks, vectors

def retrieve(question: str, chunks, vectors, k: int = 4):
    q = embed([question])[0]
    q = q / np.linalg.norm(q)
    scores = vectors @ q                      # cosine similarity
    best = np.argsort(scores)[::-1][:k]
    return [(chunks[i], float(scores[i])) for i in best]

def answer(question: str, chunks, vectors) -> str:
    hits = retrieve(question, chunks, vectors)
    context = "\n\n".join(
        f"[{n}] ({c['source']})\n{c['text']}" for n, (c, _) in enumerate(hits, 1)
    )
    prompt = f"""Answer the question using only the numbered sources below.
Cite sources like [1] or [2]. If the sources don't contain the answer,
say "I couldn't find that in the documents." Do not use outside knowledge.

Sources:
{context}

Question: {question}"""
    return generate(prompt)

chunks, vectors = build_index(["leave-policy.pdf", "travel-policy.pdf"])
print(answer("How many days of casual leave do I get?", chunks, vectors))

Notes on the choices:

  • Chunking by characters with overlap is the simplest method that works. Splitting on headings or paragraphs usually does better, especially for manuals and policies.
  • Normalising vectors lets a plain dot product act as cosine similarity.
  • Keeping the source with every chunk is what makes citations possible. Never throw metadata away.
  • The "couldn't find that" instruction is your main defence against made-up answers. Test it with questions the documents can't answer.

How should you chunk documents?

Chunking is the step people rush and later regret. A chunk should be big enough to contain a complete answer and small enough that retrieval stays precise.

StrategyHow it worksWorks well for
Fixed size + overlapEvery N characters or tokens, overlapping a littleQuick prototypes, uniform text
By structureSplit on headings, sections or paragraphs; keep the heading with its textPolicies, manuals, documentation
By sentence groupsA few sentences per chunkFAQs, short articles
Parent-childSearch small chunks, send their larger parent section to the modelLong technical documents
Common chunking strategies

Whatever you choose, add the document title and section heading to each chunk's text before embedding. "Section 4.2: Casual leave" in the chunk helps retrieval enormously when someone asks about casual leave.

Here's a structure-aware version of the chunker from the sketch, for Markdown or text with headings:

import re

def chunk_by_headings(text: str, source: str, max_chars: int = 1500):
    parts = re.split(r"\n(?=#{1,3} )", text)   # split before #, ## or ### headings
    chunks = []
    for part in parts:
        heading = part.splitlines()[0].lstrip("# ").strip()
        body = part.strip()
        for i in range(0, len(body), max_chars):
            chunks.append({
                "text": f"{heading}\n{body[i:i + max_chars]}",
                "source": f"{source} > {heading}",
            })
    return chunks

Each chunk now carries its heading, and the source shows the section, which makes citations far more useful to the reader.

What about .NET developers?

The same design works in C#. Microsoft.Extensions.AI provides common interfaces such as IChatClient and IEmbeddingGenerator so you can swap providers without rewriting your app, and Microsoft Agent Framework (1.0 since April 2026) builds agents on top of those. For storage, EF Core 10 supports vector search on SQL Server 2025 and Azure SQL Database, so a .NET team can keep embeddings next to their existing data. Our .NET Blazor training covers the C# app side, and the ASP.NET Core Web API tutorial is a good home for a /ask endpoint.

Which vector store should you use?

For a prototype, an in-memory array like the sketch is fine up to a few thousand chunks. Beyond that, pick based on what you already run:

  • PostgreSQL with pgvector if you use Postgres
  • SQL Server 2025 or Azure SQL vector support if you're a Microsoft shop
  • Azure AI Search, which combines keyword and vector search (hybrid search)
  • Dedicated vector databases such as Qdrant, Weaviate or Pinecone

Don't start with the database. Start with good chunks and an evaluation set. The store is the easiest part to change later.

How do you make RAG answers better?

In rough order of payoff:

  1. Clean your documents. Scanned PDFs with no text layer return nothing. Headers and footers repeated on every page pollute results.
  2. Chunk by structure. Keep a heading with its paragraph. Tables need special handling; flattening them to text often scrambles them.
  3. Use hybrid search. Combine vector similarity with keyword search so exact terms such as product codes or section numbers are found.
  4. Re-rank. Retrieve 20 chunks, then use a re-ranking model to pick the best 4.
  5. Filter with metadata. If a user asks about the 2026 policy, only search 2026 documents.
  6. Show sources in the UI. Users trust answers they can check, and they report bad ones.

How do you know your RAG chatbot works?

Write 30 to 50 real questions with the correct answer and the page it comes from. Include questions the documents can't answer. Then measure two things separately:

  • Retrieval: did the right chunk appear in the top results?
  • Answer quality: was the answer correct, grounded in the sources and properly cited?

When an answer is wrong, this split tells you whether to fix search or fix the prompt. Re-run the set whenever you change chunking, models or prompts. Prompt-testing habits from my prompt engineering guide apply directly.

What are the risks?

  • Access control. If a user can't open a document, the chatbot mustn't quote it to them. Filter retrieval by the user's permissions.
  • Prompt injection. Documents can contain text like "ignore previous instructions". Treat retrieved text as data, keep instructions separate, and don't give the model powerful tools without checks.
  • Stale content. Re-index when documents change and show the document date in citations.
  • Privacy. Check where your provider processes and stores data before sending internal documents.

RAG or fine-tuning?

Use RAG when the problem is knowledge: facts that change, documents that grow, answers that need sources. Use fine-tuning when the problem is behaviour: a very specific style or format that prompting can't achieve. Many teams never need fine-tuning. Some use both.

When to use RAG and when to use fine-tuning
Knowledge problems suit RAG. Behaviour and style problems are where fine-tuning earns its cost.
Paste into the promptRAGFine-tuning
Setup effortNoneModerateHigh
Handles many documentsNo (context limit)YesNot as a knowledge store
Updating contentPaste againRe-index changed filesRetrain
CitationsPossibleNaturalHard
Best forOne short documentCompany knowledge, manuals, policiesStyle, format, narrow repeated tasks
Three ways to give a model your knowledge

What does a real business RAG project look like?

A typical brief we get: an internal assistant that answers staff questions from a few hundred policy and process documents. The work usually splits like this:

  1. Inventory the documents. Which are current, who owns them, which are scanned images that need OCR.
  2. Agree permissions. HR documents for HR, finance for finance. This drives the metadata you store.
  3. Build the test set with the people who answer these questions today. Forty real questions and their correct answers.
  4. Prototype quickly, then measure retrieval and answers against the test set.
  5. Pilot with a small group, showing sources on every answer and a thumbs-down button that captures the question.
  6. Set an owner and a re-index schedule, so the assistant doesn't drift out of date.

Notice how little of that is about the model. The model is the easy part. Content, permissions and testing decide whether people trust it.

Learn to build it with guidance

Our Generative AI course builds a RAG knowledge assistant as a project, live online or in our classroom in Vallabh Vidyanagar, Anand. If you're new to the field, start with Generative AI course for beginners 2026. And if your company wants a document assistant built on its own data, our development team builds them: see software and web development.

FAQ

What does RAG stand for?

Retrieval-augmented generation. The model's answer (generation) is improved (augmented) by first searching your documents for relevant passages (retrieval).

Do I need a vector database to build RAG?

Not to start. A few thousand chunks fit comfortably in memory, as in the sketch above. Move to a vector store or a database with vector support when your data grows or several users need it.

Can RAG still hallucinate?

Yes, less often but not never. It can misread a chunk or answer from general knowledge. Instructions to use only the sources, citations users can check, and a test set of questions reduce the risk.

Which embedding model should I use?

Use the embedding model offered by your main AI provider to start, and test it on your own questions. Use the same model for documents and questions, and re-embed everything if you switch.

Can I build a RAG chatbot without coding?

Several no-code tools and Microsoft Copilot Studio let you point an assistant at documents. They're good for simple internal use. Coding gives you control over chunking, permissions, evaluation and cost, which matters for customer-facing systems.

Choose optional cookies

We use necessary cookies for account security. With your permission, we also use analytics and advertising cookies on marketing pages. See our Privacy Policy.