LLM Regression Testing: How to Catch Quality Drops Before They Ship
Written By:
Matt Tanner
Published on
You rewrote your support assistant's system prompt to sound warmer, spot-checked a dozen conversations, and shipped it. Two days later, someone notices that "I'm locked out, and someone changed my email" now gets password-reset tips instead of an escalation for a likely account takeover. Nothing errored, and most answers read better than before. That's the failure LLM regression testing exists to catch: a change that improves what you were looking at and quietly breaks something you weren't.
This guide covers the whole loop, from golden dataset to a CI gate, with a pytest file and GitHub Actions workflow you can copy.
What is LLM regression testing?
LLM regression testing is the practice of re-running a fixed set of inputs through your LLM application after every change (prompt, model, retrieval, tools, or parameters) and confirming that behavior that used to work still works. Because outputs vary between runs, it scores outputs with checks and compares pass rates against a baseline instead of matching exact text.
In traditional software, regression tests are unit tests with fixed expected values, so one failed assertion means something broke. LLM testing doesn't get that guarantee. When researchers ran five LLMs configured to be deterministic on eight tasks, 10 runs each, they saw accuracy vary by up to 15% between runs. The same prompt and model can pass a case on Tuesday and fail it on Wednesday with nothing changed on your side.
That non-determinism changes how the tests are built:
Assertions become checks with thresholds. You test properties (valid JSON, the escalation flag is set, the answer cites the 30-day window) rather than exact strings.
One run is weak evidence. Important cases get sampled several times.
The baseline gets measured each time, on the same cases and in the same run as the candidate.
Why LLM apps regress
Most regressions in LLM applications come from a change somebody made on purpose for a different reason. The account-takeover miss above came from a tone fix. These are the usual starting points:
Prompt edits. New instructions compete with old ones. A warmer prompt that says "try to resolve the issue yourself" pulls directly against "escalate suspected account takeover."
Model swaps and version updates. A cheaper model, or the replacement for a retired one, can read the same instructions differently, wrapping JSON in a friendly sentence or refusing requests it used to answer.
Retrieval and chunking changes. A new chunk size can separate a policy exception from the rule it modifies, so the model retrieves "refunds within 30 days" without "except annual plans." A lower top-k can drop the one passage that held the answer.
Sampling parameters. Raising temperature for livelier copy also loosens every structured output sharing that config.
Tool changes. Renaming an argument or adding a second tool with an overlapping description changes which tool the model picks and what it passes.
Some regressions arrive with no commit at all: a refreshed knowledge base or a provider-side model update shifts results too. Edge cases show all of these first, like the long multi-part question or the request that's 90% routine and 10% policy exception.
Build a golden dataset
A golden dataset is the fixed set of test cases every version runs against. "Golden" is about stability. If you edit the dataset in the same pull request as the prompt, you can't tell whether the score moved because of the app or the test.
Where the test cases come from:
Real traffic. Sample production inputs across your main intents, with personal data stripped.
Known failures. Every bug report and incident becomes a case. The locked-account conversation goes in the day you find it.
Edge cases. Long inputs, ambiguous ones, adversarial ones, and the policy exceptions your retrieval tends to miss.
Store expected properties, not only expected answers. Input/output pairs with a reference answer feed similarity and judge checks. Most of what you care about fits a property, though: the response parses as JSON, action equals escalate, the answer mentions "30 days." Properties survive harmless rewording. Here's the format the test suite below reads:
The critical flag marks cases where a single failure should block a release, whatever the overall score says.
Size it with the math in mind. With 50 cases, one case is worth 2 percentage points of pass rate, so a "2% drop" is one conversation. If a version passes all 50, the most you can claim at 95% confidence is that its true failure rate sits below about 5.8%.
Grow toward a few hundred cases for the flows that matter, and tag each case by slice (intent, language, customer tier) so you can read results per slice. Add production cases on a schedule and version the file.
Choose checks and metrics
Use the cheapest check that catches the failure. Our guide to LLM evaluation metrics covers the full catalog. For regression testing, four kinds do most of the work:
Check | What it catches | Watch out for |
|---|---|---|
Deterministic (JSON validity, schema, length, required phrase, tool name) | Format breaks, missing fields, dropped escalations, wrong numbers | Only catches what you thought to write down |
Semantic similarity to a reference answer | Meaning drift after a rewrite or model swap | A close paraphrase can still get a key number wrong |
LLM-as-a-judge with a rubric | Tone, helpfulness, policy adherence, completeness | Cost, run-to-run variance, judge bias |
Groundedness (RAG) | Claims the retrieved context doesn't support | Needs the retrieved passages logged with each output |
Deterministic checks go first. They're free and give the same verdict every time, so a failure means the output really changed.
Semantic similarity handles open-ended answers. Embed the output and the reference and compare cosine similarity against a threshold, which is how promptfoo's similar assertion works. Close in meaning isn't the same as correct, so keep a deterministic check on the numbers that matter.
Judges cover what code can't express. An LLM-as-a-judge scores output against a written rubric, and judges carry measurable biases, including position, verbosity and self-enhancement bias, as the MT-Bench study documents. In a regression suite, verbosity bias means a friendlier, longer candidate can win on "helpfulness" for reasons unrelated to quality. Pin the judge model, prompt, and sampling settings, and compare its verdicts with a hundred or two human-labeled outputs before you let it block a merge.
Groundedness is the RAG-specific check. It confirms the retrieved passages support every claim in the answer.
Set pass/fail thresholds
Scores block nothing until you decide what failing means, and most suites need two kinds of thresholds.
Absolute thresholds apply to the candidate alone. Every critical case passes on every repeat, JSON validity is 100%, p95 latency stays under your SLO. If production also fails a critical case, that's a bug to fix, and matching it doesn't earn the candidate a pass.
Relative thresholds compare the candidate to the baseline on the same cases. The suite pass rate (passing case runs divided by total case runs) can't drop by more than a set tolerance, and each slice gets its own tolerance.
Noise is the hard part. A candidate can lose two points of pass rate with no real change, and three habits keep that in check:
Sample each case more than once. Run every case three to five times and score a per-case pass rate. A case that passes 2 of 3 times is flaky, which is worth knowing too.
Compare paired results, case by case. Both versions see identical cases, so compare them per case rather than as two independent averages. Evan Miller's paper on eval statistics recommends paired standard errors for exactly this setup. Then count the flips in each direction: pass to fail and fail to pass.
Measure your noise floor. Run the baseline against itself twice. The pass-rate gap between two identical runs tells you how wide your tolerance has to be before a drop means anything.
Flips also keep small samples honest. Say a candidate newly fails 9 cases and newly passes 3. That reads like a clear loss, but an exact McNemar test on those flips gives a p-value of about 0.15, so the net drop could plausibly be noise. Read the 9 anyway: if one is critical, the absolute threshold already blocked the change, and if they cluster in one slice, you've found the regression the average was hiding.
Run candidate against baseline
A few details decide whether you can trust the comparison:
Run both versions in the same job. A provider-side update, judge drift or a refreshed index then hits both sides equally, which a baseline score stored last month can't account for.
Hold everything else fixed. Same dataset version, checks, judge and repeat count. Change one thing per candidate where you can, so a regression points at its cause.
Keep outputs, not only scores. Save every output with its case ID and check results, because the diff you'll actually read is the flipped cases side by side.
You rarely test one candidate, though. Three prompt variants, a cheaper model and two chunk sizes make twelve configurations to score against the baseline, and running them one after another is where the afternoon goes.
That's the workflow we built RapidFire for. You define the configurations as a group, for example, a List of RFAPIModelConfig entries (each a model with an optional prompt manager) inside an RFGridSearch. Our open-source engine's run_evals() splits the eval data into shards (four by default) and runs the versions side by side on one machine. Your metric functions score every configuration, so baseline and candidates are judged by identical code.
If you add an accumulate_metrics_fn and declare your metric types, online aggregation shows running estimates with confidence intervals that narrow as each shard completes. For API-based generators, the dashboard also logs average latency per query and projected token spend, which matters when the regression is a cost jump.
You can then stop a clearly worse candidate early, or clone and modify a promising one, from the dashboard or a notebook panel. In the open-source engine, those steps are manual. Converge, our automation layer, stops weak runs and clones strong ones on its own within the time and spend limits you set, and logs the reason for each decision.
Shards are contiguous slices in the order you pass the data, so shuffle a test set sorted by intent first, or early estimates will be biased.
Wire it into CI/CD
[Image placeholder: Left-to-right flow diagram with five stages joined by arrows. Stage 1, a stacked-file icon labeled "Golden dataset" with the caption "golden.jsonl, versioned". From it, the arrow splits into two parallel boxes for stage 2: "Baseline run (config on main)" on top and "Candidate run (config in PR)" below, each with the caption "same cases, same checks, 3 repeats". Both boxes point into stage 3, a box labeled "Metric scores per case" with two lines inside: "critical cases" and "suite pass rate". Stage 3 points to stage 4, a diamond labeled "Pass/fail gate" with two rules beside it: "Every critical case passes every repeat" and "Pass rate within tolerance of baseline". Stage 5 has two exits from the diamond: a green arrow labeled "Pass" to a box "Merge", and a red arrow labeled "Fail or error" to a box "Block PR". Alt text: LLM regression testing flow from a golden dataset through baseline and candidate runs and metric scores to a pass/fail gate that merges or blocks the pull request.]
The gate only works if it runs on every change and nobody can merge around it. Plain pytest is enough to start, and the checks read like ordinary unit tests. This test file reads the dataset above, runs both versions three times per case, and encodes both kinds of threshold:
generate() stands in for your app, with "baseline" meaning the config on main and "candidate" the one in the pull request. We checked the gate logic against a stubbed generate() that makes no model calls. When the stubbed candidate stopped escalating the locked-account case, the critical test failed, the suite test reported 67% vs baseline 100%, newly failing: ['locked-account'], and pytest exited 1.
When the stub raised a provider timeout, pytest reported errors and also exited 1, so a flaky API produces a red check instead of a silent pass.
The workflow runs it on every pull request:
A few choices in there are deliberate:
permissions: contents: readnarrows what the job's GitHub token can do.timeout-minutes: 20caps a hung provider call. The default is 360 minutes.if: ${{ !cancelled() }}uploads the JUnit report even when tests fail.Pull requests from forks won't receive
OPENAI_API_KEY, because GitHub doesn't pass secrets to workflows triggered from a fork, so expect those runs to fail.
Make the job a required status check in branch protection so a red run blocks the merge. That's also why the workflow has no paths filter: when a required check's workflow is skipped by path filtering, GitHub leaves the check pending and the pull request can't merge. If you want to skip model calls on docs-only changes, test the changed files inside the job instead.
Control cost and runtime. In every case, repeat and version are model calls, plus judge calls. Run the critical cases and a stratified sample on pull requests, and the full set with more repeats nightly or before a release.
If you'd rather not hand-roll it, several evaluation tools package the same loop.
DeepEval, Confident AI's open-source framework, runs pytest-style assert_test checks with metric thresholds through deepeval test run. promptfoo's eval command exits with code 100 when any test case fails. Langfuse runs experiments over a dataset with item-level and run-level evaluators, and Evidently offers a declarative testing API. The threshold and baseline logic above applies whichever runner you pick.
Regression testing agents and RAG
Agents can reach the right answer by the wrong path: skipping a confirmation before a write, calling a tool they shouldn't, or looping five times before finishing. Log every tool call with its arguments and assert at three levels, the same split our guide to agent evaluation uses:
Outcome: the end state is right (the refund record exists, the ticket has the right priority), whatever the wording.
Trajectory: required steps happened, forbidden ones didn't, and the run stayed under its step budget.
Steps: each call picked the right tool, with arguments that match its schema.
Every step in an agent run is another sampled output, so give agent cases more repeats and gate critical tasks on passing every repeat, not most of them.
RAG needs retrieval and generation tested separately so a failure points at the right layer. For retrieval, label the passages each golden case needs and check recall@k: did the right chunk still come back after the chunking or embedding change? For generation, check groundedness against whatever was retrieved. Our guide to RAG evaluation metrics covers both families. A chunk-size change that drops recall on your policy-exception cases while answers stay fluent is the classic silent regression, and only a retrieval-level check will show it.
Common mistakes
A dataset too small to see the change. With 20 cases, each flip moves the pass rate five points, so the gate either fires on noise or gets loosened until it never fires.
Judge drift. When the judge model or rubric changes, every score moves and old baselines stop being comparable. Test a new judge against human-labeled cases, then re-baseline.
Overfitting to the test set. Tune a prompt until every golden case passes and you get a prompt that's good at your golden cases. Keep a held-out slice you don't look at while iterating.
Skipping production monitoring. Your test suite only covers failures you already know about. Run the same checks on sampled live traffic and turn new failures into test cases, so the next locked-account case lands in the suite before a prompt change instead of after.
Start with the cases that have already burned you, a pytest file and a required check. When the question becomes which of a dozen prompt, model, and retrieval variants to ship, open the RapidFire RAG walkthrough. It needs Python 3.12+, and no GPU if you only call OpenAI models. The FiQA and SciFact notebooks give you a working eval setup to swap your own test cases and checks into.
FAQ
How are LLM models tested?
At two levels. Benchmarks measure general capability on public datasets. Application testing, which matters more for a product, runs your own golden dataset through the full system, scores outputs with deterministic checks, semantic similarity, and LLM judges, and compares results with the production version.
Can you automate regression testing?
Yes. Put the test cases, checks, and thresholds in a test file, run it in your CI/CD pipeline on every pull request, and make the job a required check so failures block the merge. Automate the run and the blocking, but keep a person reading the flipped cases, because an aggregate score hides which behavior changed.
Which AI is best for regression analysis?
Regression analysis is a statistics task (modeling how one variable depends on others), separate from regression testing. An LLM can write and explain the code, but the coefficients should come from a statistics library such as statsmodels or scikit-learn, not from a model's text output. For regression testing, the best model is the one that passes your test cases.
How to test LLMs as a QA?
LLM testing for QA starts by treating each prompt, model, and retrieval setting as a release artifact. Build test cases from real user inputs and past bugs, write properties you can check (format, required facts, escalation rules), run each case several times because outputs vary, and compare every candidate against the current baseline. File flipped cases as bugs.