Data Preparation for AI Projects

Why AI projects stall on data, not on the model

When a company asks Tepia why its AI pilot gives bad answers, the model is rarely the cause. The usual story is that the assistant found three versions of the employee handbook in a shared drive and cited the oldest one, or that a forecasting model learned from sales history where returns were booked as new orders. A stronger model reads the same bad data faster.

Data preparation is the unglamorous work that decides whether an AI feature is trusted. It means knowing which sources the AI should read, which version of each is current, who is allowed to see what, and how the data will stay current after launch. It also means admitting when the data you want to use does not exist yet, which is better learned in week two than in month five.

Tepia scopes this work explicitly instead of hiding it inside a model build. That keeps expectations honest and gives your team a cleaner data estate that pays off for reporting and integrations too.

A data readiness inventory: where problems usually hide

Tepia starts with an inventory of every source the AI feature might touch. These are the problems we find most often and how we handle them:

The inventory produces a readiness score for each source and a short list of fixes ranked by the AI use cases they unblock. If you want this as a standalone engagement, see our AI readiness assessment.

Preparing documents and knowledge for retrieval

Assistants that answer from company knowledge use retrieval: the system finds the most relevant passages and hands them to the model with the question. Retrieval quality depends on preparation, and Tepia handles it in a fixed order.

1. Parse each file into clean text while keeping headings, tables and page numbers, so answers can cite a section.
2. Chunk by meaning rather than by character count, splitting at headings so a policy clause is not cut in half.
3. Enrich every chunk with metadata: source, owner, effective date, product line, region and access group.
4. Resolve versions so only the current document is searchable, while older versions stay available for audit.
5. Embed and index into a vector database with keyword search alongside, since part numbers and policy codes match poorly on meaning alone.

Tepia then tests retrieval on its own with real questions before any model sees the results. More detail is on our RAG development page and in fine tuning vs RAG.

Preparing structured data for prediction and classification

Forecasting, scoring and classification need a different kind of preparation. The model learns from history, so the history has to be consistent. Tepia checks that identifiers mean the same thing across years, that cancelled and returned transactions are marked as such, and that there is enough history to capture seasonal patterns before promising a forecast.

Classification needs labels. If you want incoming requests routed by type, someone has to label a few hundred examples per type, and the labels have to agree with each other. Tepia builds a simple labeling screen, has two people label a sample independently, and measures how often they agree. Low agreement means the categories need redefining before any model is trained.

Evaluation splits are made by time, not at random, so the test reflects how the model will perform on next month’s data rather than last month’s. The pipelines that build these tables are part of our data integration and ETL work.

How Tepia approaches AI data preparation

Tepia runs its six phase process with data as the main deliverable. A US based project manager coordinates the system owners whose access and knowledge the work depends on.

1. Discovery: source inventory, access review, sample extracts and interviews with the people who maintain each system. Deliverables are an Investigation Summary, an Interview Summary and User Stories such as a manager asking which policy applies to a contractor.
2. Design: the target data model, metadata schema, chunking rules, redaction rules and a data dictionary, plus wireframes for any labeling or review screens.
3. Development and Testing: extraction and cleaning pipelines, the index, labeled datasets and retrieval tests. Alpha runs on one department’s data; Beta covers the full scope.
4. Training: sessions for document owners on tagging, retiring old versions and adding new sources without breaking the index.
5. Launch: full data migration into the new stores, scheduled syncs switched on and monitoring of pipeline failures.
6. Support: new sources added, schema changes handled and data quality reports reviewed with your team.

Typically Discovery takes 1 to 3 months and development 2 to 6 months, often running alongside the AI feature build. See our process.

Privacy, permissions and keeping data current

Preparing data for AI also means deciding what the AI must never see. Tepia applies redaction for names, account numbers and health details where the use case does not need them, and follows HIPAA aware handling for PHI and GDPR and CCPA practices for personal data. Model providers are configured so your content is not used for training.

Permissions matter as much as privacy. An assistant that indexes the HR folder without its access rules will happily summarize salary reviews for anyone who asks. Tepia syncs access groups from Microsoft Entra ID, Google Workspace or your own system and filters every retrieval by the asking user’s rights.

Finally, data goes stale. Pipelines detect changed and deleted files, reindex on a schedule, and report sources that have not been updated in months. Read more about security and compliance, or talk to Tepia about the data behind your AI plans.

Frequently asked questions