LLM Regression Testing: Build a Suite That Gates Every Change

A prompt edit that fixes one complaint, a model upgrade the provider recommends, a new chunking setting on the retriever: each looks small, and each can quietly break answers that worked last week. In ordinary software a regression suite catches that. In an LLM product the idea is the same, but the tests look different, because outputs vary from run to run and most answers have no single correct string.

This guide is about the regression suite itself: what goes in it, how it scores, and how it decides whether a change ships. If you have not built an evaluation harness at all yet, start with why we build the eval harness before the model. If your system is RAG, RAG evaluation metrics covers how to score retrieval and answers separately. Here we assume you have some evals and want them to act as a gate.

What should trigger a regression run?

Anything that can change what the system says.

• Prompt and system instruction edits, including small wording changes and new examples.
• Model changes: a new version, a different provider, or an alias the provider moves to a newer model. Pin dated model versions where the provider offers them, so the model only changes when you change it.
• Retriever changes: chunking, embedding model, index rebuilds, reranker, filters, and the number of passages passed to the model.
• Tool and schema changes for agents. A renamed function, a new parameter or a reworded description can change which tool the model picks.
• Configuration: output length limits, reasoning settings, API version upgrades and SDK upgrades.
• Content: a large batch of new documents in the knowledge base can change answers as much as code does.

Treat each of these as a change with a pull request, and make the suite a required check on that pull request. If a change cannot go through review, such as a provider moving an alias, schedule regular runs so you find out before your users do.

How do you regression test an LLM application?

Build a versioned golden set from real production traffic, tagged by user segment and severity, with expected answers or rubrics approved by domain experts. Run it on every prompt, model, retriever, tool or configuration change. Score with deterministic checks first (schema, tool calls, citations, refusals, token and latency budgets), then with an LLM judge calibrated against expert labels. Run each case several times to absorb variance, report results per segment against a same-day baseline, and block any release that fails a must-pass case or drops a segment below its threshold.

How do you build a golden set from real traffic?

The golden set is the list of inputs, with expected outputs or grading rules, that every change is tested against. Its value depends on how closely it resembles what users actually send.

• Sample from production logs, not from the team's imagination. Stratify the sample so it covers each user type, language, channel and task, not only the most common request.
• Add every confirmed production failure as a case, with a note linking it to the incident. These are the cases most likely to come back.
• Include deliberately hard inputs: ambiguous requests, very long inputs, out-of-scope questions the system should decline, and attempts to override instructions. The prompt injection testing checklist is a good source of adversarial cases.
• Remove or replace personal data before a log entry becomes a test case. Test sets get copied into CI systems and shared with more people than production logs ever are.
• Have a domain expert write or approve the expected answer or rubric for each case. Engineers can judge format; they often cannot judge correctness in law, medicine or finance.

Tag every case with its segment (customer tier, language, product area, task type) and its severity. Those tags drive the reporting and the gate later on.

Deterministic checks first, a judge second

Many failures do not need a model to detect them. Deterministic checks are fast, cheap, repeatable and hard to argue with, so run them first and use them wherever they apply.

• Structure: the output parses, matches the JSON schema and has every required field.
• Tool calls: the agent called the expected tool with valid arguments, and did not call a forbidden one.
• Content rules: required disclaimers are present, banned phrases are absent, and no internal identifiers or secrets leak.
• Citations: every cited source exists in the retrieved set, and answers that need a citation have one.
• Refusals: out-of-scope questions are declined, and in-scope questions are not.
• Budgets: output length, token use and latency stay inside agreed limits.

Then use an LLM judge for what rules cannot capture: whether an answer is correct, complete, grounded in its sources and written in the right tone. Judges have known weaknesses. The MT-Bench paper by Zheng and colleagues (NeurIPS 2023 Datasets and Benchmarks track) examined position bias, verbosity bias and self-enhancement bias, where a judge favours an answer because of where it appears, how long it is, or which model wrote it.

Calibrate the judge before you trust it

• Ask narrow questions on a fixed scale, such as "Does the answer state the refund window from the policy? Yes or no", rather than "Rate this answer from 1 to 10".
• Have experts label a sample of cases, run the judge on the same sample, and measure how often they agree. Read the disagreements and fix the rubric before the judge goes into the gate.
• In pairwise comparisons, run each pair twice with the order swapped, and count a win only when both orders agree.
• Pin the judge's model version and prompt, and version them with the suite. A judge model is a dependency like any other: when it changes or is retired, recalibrate before comparing new scores with old ones.
• Keep human spot checks on a rotating sample, weighted towards the cases where the judge was least certain.

How do you handle non-deterministic outputs?

The same input can produce different outputs on different runs, so a single pass can flag a regression that is only noise, or hide one that is real. You cannot always remove the variation at the source either. Anthropic's model deprecations page, for example, lists temperature, top_p and top_k as removed in its newest API version, so the old habit of setting temperature to zero for tests is not available there. Design the suite for variance instead.

• Run each case several times and score its pass rate, rather than a single pass or fail.
• Compare against a baseline run of the current production version on the same day, not an old stored score, so provider-side drift does not look like your regression.
• Set thresholds with an agreed tolerance, so ordinary run-to-run variation does not block every release.
• Mark cases that flip between runs on the unchanged system as unstable. Tighten the rubric or the expected behaviour, rather than letting them make the gate unreliable.

Why must results be reported per segment?

A single average hides the regressions that matter. A change can lift the overall score while making answers worse for one language, one customer tier or one product area, and the average will look like progress. Report every metric by the segment tags on each case, and show the cases that moved.

• Pass rate per segment, candidate against baseline, with the difference highlighted.
• Cases that passed before and fail now, with both outputs side by side.
• Changes in token use and latency, since a model or prompt change can raise both without changing a single answer.

Store each run with the suite version, prompt version, model identifier and retriever configuration, so any number can be traced back to exactly what produced it. Our post on what to log from an LLM app without storing personal data covers how to keep those records useful without keeping user content.

What should the release gate say?

A gate is only useful if it is written down before the change arrives. Agree the rules with product and engineering leads, then let the pipeline enforce them. In our SI-driven QA work this is the step teams most often skip, and it is the one that turns a dashboard into a decision.

1. Must-pass cases: a small set of critical behaviours (safety refusals, regulated wording, core tasks) where any failure blocks the release.
2. No segment drops below its threshold, and no segment falls more than an agreed margin below baseline.
3. Deterministic checks pass at their required rate, such as every output parsing correctly.
4. Token use and latency stay inside budget.
5. Any exception is signed off by a named owner, with the reason recorded next to the run.

Run a fast subset on every pull request, and the full suite before each release and on every model change. The fast subset catches obvious breakage early; the full suite makes the release decision.

How do you keep the suite versioned and honest?

• Store cases, rubrics, judge prompts and thresholds in version control, and give the suite a version number that appears in every report.
• Hold back a set of cases that nobody tunes prompts against. If scores rise on the main set but not on the holdout, the prompt is overfitting to the tests.
• Review the suite on a regular cycle: retire cases whose expected answer is no longer true, and add new ones from recent traffic and incidents.
• When you change a rubric or the judge, rerun the baseline so old and new scores stay comparable.
• Give the suite an owner. Without one, it decays into a list of tests everybody skips.

A regression suite is a promise that what worked yesterday still works today, checked by a machine instead of by hope.

Where Octalcode fits

Octalcode builds LLM applications, SI agents and RAG systems, and we treat the regression suite as part of the build rather than an afterthought. Our SI-driven QA work covers golden sets, judge calibration and release gates.

If you already have evals but no gate, a sensible first step is to check what your current suite catches and misses against your own traffic. Whoever does that work, the suite and the findings should stay with your team.