LLM Evaluation and AI Quality Testing

Why AI features need a different kind of testing

Traditional QA assumes the same input produces the same output. A language model breaks that assumption. Ask the same question twice and the wording changes; ask a slightly different question and the answer may draw on a different document. There is often no single correct string to assert against, only a range of acceptable answers and a set of clearly wrong ones.

Three other things make it harder. Prompts behave like code but are edited like copy, so small wording changes ship without review. Model providers release new versions on their own schedule, and a version that is better on average can still be worse on your cases. Retrieval adds its own failure mode, where the model answers fluently from the wrong source.

Tepia treats evaluation as part of the build, not a step at the end. The test set is defined in Discovery, before the first prompt is written, so everyone agrees on what good looks like and the team can measure progress against it every day.

What to measure: an evaluation matrix

Each AI feature gets its own matrix. Tepia agrees the dimensions and thresholds with your product owner, then automates as many as possible:

Thresholds differ by risk. A suggestion a user edits before sending can tolerate more misses than an answer shown directly to a patient or a customer.

Building a golden dataset from your own data

Generic benchmarks tell you little about your feature. Tepia builds the test set from your own material: real support tickets, real documents, real search queries, and the awkward cases your team remembers. A first set of a few hundred examples is usually enough to separate a good prompt from a bad one, as long as it covers each category of request and includes the hard cases.

• Representative cases sampled in proportion to how often each request type appears in production.
• Edge cases such as ambiguous questions, missing data, conflicting documents and very long inputs.
• Adversarial cases written to provoke leakage, invented facts or instructions the feature should ignore.
• Expected behavior written by your domain experts, including when the right answer is to say it does not know.

The set is versioned alongside the code. Every production failure that reaches a user becomes a new test case, so the same mistake cannot return unnoticed. If your data still needs work before this is possible, see data preparation for AI projects.

How Tepia approaches LLM evaluation and testing

Tepia runs its six phase process whether we are testing a feature we built or one another team built. A US based project manager coordinates your domain experts, who do the labeling that matters most.

1. Discovery: review of the feature, its prompts, retrieval setup and logs, plus interviews with the people who judge answer quality today. Deliverables are an Investigation Summary, an Interview Summary and User Stories such as a reviewer flagging an ungrounded answer.
2. Design: the evaluation matrix, thresholds per dimension, labeling guidelines, and wireframes for the review screen your experts will use.
3. Development and Testing: the golden dataset, automated graders, a model graded judge calibrated against human scores, CI integration and a red team suite. Alpha scoring runs on the current version to set a baseline.
4. Training: hands on sessions for reviewers on labeling, and for engineers on reading evaluation reports and adding cases.
5. Launch: evaluation gates switched on in the release pipeline and production tracing turned on with alerting.
6. Support: quarterly test set refreshes, new model comparisons and regression triage when scores move.

An evaluation project on an existing feature usually sits at the short end of the typical ranges: 1 to 3 months of Discovery and 2 to 6 months of development and QA. See our process.

Automated graders, human review and red teaming

Automation makes evaluation fast enough to run on every change, but it has to be trusted. Tepia uses deterministic checks wherever possible: schema validation, required citations, banned phrases, exact field matches. For open ended answers we use a separate model as a grader with a written rubric, and we measure its agreement with your human reviewers before anyone relies on its scores.

Human review stays in the loop. A sample of production conversations goes to a review queue each week, scored with the same rubric, so drift shows up in numbers rather than anecdotes.

Red teaming covers the risks listed by OWASP for LLM applications, including prompt injection through uploaded files, disclosure of system prompts or other users’ data, and tool calls an agent should never make. Findings feed both the test set and your AI governance and risk documentation. Our QA and automated testing team runs the conventional test suite alongside it.

Keeping quality stable after launch and across model upgrades

Most quality regressions Tepia sees in production come from changes nobody thought of as risky: a new model version, an edited prompt, a reindexed document store. Each one gets the same treatment before release.

• Pinned versions: the feature calls a specific model version, never a moving alias.
• Side by side runs: a candidate model or prompt runs against the full test set and the results are compared dimension by dimension.
• Shadow traffic: where risk is high, the candidate answers live requests silently before it serves users.
• Alerts: spikes in negative feedback, refusals or slow responses page the on call engineer.

This is also how teams make an informed choice between providers, a topic covered in OpenAI vs Claude for business apps. If you have an AI feature live and no test set behind it, talk to Tepia.

Frequently asked questions