How to Build a RAG System on Your Company Data

What a RAG system is and why AI agents need one

Retrieval augmented generation is a pattern, not a product. When someone asks a question, the system searches your own documents and records, passes the most relevant passages to a language model, and the model writes an answer that cites them. The model never has to memorize your handbook; it reads the right page at the moment of the question.

Most custom AI agents need this layer underneath them. An agent that drafts a reply to a warranty claim has to read the warranty terms for that product line, and an agent that helps a field technician has to read the right service bulletin. Tepia usually builds the RAG system first, proves its answers are accurate, then gives the agent tools to act on what it finds. If you are still choosing an approach, our guide to fine tuning vs RAG explains why retrieval wins whenever your facts change every week.

The steps below are the order Tepia follows. Skipping one, especially evaluation, is the usual reason a demo impresses in week two and disappoints in month three.

Step by step: building a RAG system on your data

1. Collect real questions. Pull 100 to 200 questions from support tickets, team chat, call notes and search logs, and record the correct answer and its source for each. This becomes your evaluation set.
2. Inventory the sources. List every system that holds answers: SharePoint, Google Drive, Confluence, Zendesk, product manuals as PDFs, and structured data in your ERP or database. Note the owner, how often it changes and who may see it.
3. Parse and clean. Extract text with its structure intact, including headings, tables and page numbers. Scanned PDFs need OCR, and superseded versions should be excluded rather than indexed next to the current one.
4. Chunk by structure. Split documents along headings and sections instead of fixed character counts, keep a small overlap, and attach metadata such as title, version date, product line and access group to every chunk.
5. Embed and index. Generate embeddings and store them beside a keyword index so both kinds of search run against the same chunks.
6. Retrieve and rerank. Run keyword and vector search together, merge the results, rerank the top candidates and filter by the user’s permissions before anything reaches the model.
7. Generate with citations. Instruct the model to answer only from the retrieved passages, cite each one, and say plainly when the sources do not contain the answer.
8. Evaluate, launch and monitor. Score against the evaluation set, release to a pilot group, collect feedback on each answer and add every failed question back into the set.

Tepia runs the first two steps during Discovery, because together they decide whether RAG is the right tool for your problem at all.

Key RAG decisions and how to make them

A handful of design choices shape accuracy more than the choice of model does. Here is how Tepia frames them with clients.

There is no universally best embedding model or vector database. Tepia picks the simplest option your team can run, then lets evaluation scores on your own questions settle disagreements instead of vendor benchmarks.

Permissions, freshness and failure modes to plan for

The most damaging RAG failure is not a hallucination. It is a correct answer drawn from a document the user should never have seen. Tepia stores each source document’s access list with its chunks and filters results by the signed in user’s groups, using single sign on through Microsoft Entra ID or Okta so permissions stay managed in one place.

Stale content is the second failure. If a return policy changed on Tuesday and the index rebuilds monthly, the system will confidently repeat the old rule. Tepia builds incremental sync that watches SharePoint or Drive for changes, reindexes only what changed, and retires superseded versions so two conflicting policies never both rank.

Plan for these as well:

• Tables flattened into plain text, so the model reads a spec sheet’s columns in the wrong order.
• Acronyms and internal jargon that embeddings do not understand, handled with a glossary and keyword search.
• Questions that need live data, such as order status, which belong in an agent tool call rather than the index.
• Answers without citations, which staff stop trusting after the first mistake.

How Tepia approaches RAG system builds

Tepia follows the same six phases on every engagement, with a US based project manager owning the plan.

1. Discovery: question collection, source inventory, permission mapping and a sample ingestion of your messiest documents. Deliverables are an Investigation Summary, an Interview Summary and User Stories such as a support rep asking which warranty applies to a serial number.
2. Design: a design questionnaire, moodboards and wireframes showing where answers appear, how citations display and how users flag a wrong answer, plus a style guide for the assistant’s tone.
3. Development and Testing: an Alpha on one source and one team, then a Beta across sources, with test plans for retrieval accuracy, permission leakage and load.
4. Training: hands on sessions showing staff how to phrase questions, read citations and report gaps.
5. Launch: production indexing, monitoring dashboards and a rollout by department, with support reps reviewing the first weeks of questions.
6. Support: evaluation reruns after model updates, new source connectors and feature requests.

Discovery typically runs 1 to 3 months, development and QA 2 to 6 months, and launch about a month, with a pilot on one well kept source at the short end. See our process for detail on each phase.

Who should build your RAG system, and when to buy instead

If your documents already live in Microsoft 365 or Google Workspace and the main need is staff asking general questions about them, try the assistant built into that suite first. Tepia will say so in Discovery rather than build something you do not need.

A custom build earns its place when:

• Answers must appear inside your own app, portal or CRM rather than a separate chat window.
• Sources span systems no packaged assistant connects to, such as a field service database or an older document archive.
• Permissions are granular, for example per customer, per region or per contract.
• An AI agent must act on what it retrieves, such as creating a case or drafting a reply.
• You need audit logs of what was retrieved and shown, for HIPAA or contractual reasons.

Tepia builds these through custom AI agent development and standalone RAG development. To measure accuracy before you commit, see LLM evaluation and testing, or talk to Tepia about your sources.

Frequently asked questions