LLM Evaluation Dataset: How to Build One That Works
Written By:
Matt Tanner
Published on

Alt text (deployment only): An evaluation-record excerpt pairs conversation messages with expected facts and guidelines; a diagram assigns entire conversations to either development or holdout splits.
Your team swaps the model behind a billing assistant, reruns the 30 conversations someone pasted into a spreadsheet last spring, and 29 pass. A week later, the new model is promising refunds on charges that were never duplicated. The spreadsheet had no case like that, so the score couldn't have warned you. An LLM evaluation dataset is only as good as the cases in it, and most teams build theirs by accident.
This guide covers building one on purpose. You'll see what each test case needs, where cases come from, how to split and version the set so scores stay honest, and which public datasets are worth pulling in. It includes a tested Python script that splits a dataset and stores it in MLflow.
What is an LLM evaluation dataset?
An LLM evaluation dataset is a fixed set of test cases you rerun against every version of your LLM application. Each case holds an input, an expected output or the criteria a good answer must meet, and metadata such as where the case came from and which slice of traffic it represents. Once it's reviewed and frozen, many teams call it a golden dataset.
The expected part rarely needs to be one exact answer. For "I was charged twice for my annual plan," the reference might be two facts the reply must contain: the duplicate gets refunded, and the money goes back to the original card. Add one rule it must follow (never ask for the full card number), and you have a case. Open-ended tasks almost always need criteria like these, because two correct replies seldom share wording.
Frameworks name the parts differently. In DeepEval, a reviewed example is a golden; running your app on its input creates the test case you score. MLflow splits each record into inputs and expectations. The idea is the same, and in this guide a test case means one row of the dataset.
Public benchmarks vs. your own dataset
Public benchmarks answer a different question. MMLU spans 57 tasks, from elementary math to law, and TruthfulQA checks whether a model repeats popular misconceptions. Both say something about a base model, but neither knows your refund policy, your retrieval index or how your customers phrase things. A model that leads on MMLU can still apply the wrong rule to an annual plan.
Public test sets can also leak into training data. Researchers built GSM1k to mirror the GSM8K math benchmark, and some models showed lower accuracy on GSM1k than on GSM8K. That gap is a sign of test-set contamination, where data closely resembling benchmark questions ends up in training, though many frontier models in the same study showed little sign of it.
Keeping newly collected cases private reduces exposure to public training crawls, which is one more reason your own set should carry the decision. Privacy alone does not prove novelty: copied public material or cases previously shared with a provider can still overlap training data.
| Public benchmark | Your own dataset |
|---|---|---|
Question it answers | How capable is this model in general? | Does my application do this job? |
Best use | Shortlisting models | Choosing the prompt, model and retrieval settings you ship |
Contamination risk | Real for popular open sets | Low while it stays private |
Cost | Free download | Hours of collection and review |
Use open source benchmarks to narrow the shortlist, make the final call on your own evaluation dataset, and report the two scores separately so one can't hide the other.
What goes into a good dataset
A useful test case has five parts. Skip one, and you'll find out later, usually when a score moves, and nobody can say why.
Part | What it holds | Billing assistant example |
|---|---|---|
Input | The user message, or the conversation up to the turn under test | "I was charged twice for my annual plan." |
Expected output or criteria | A reference answer, facts that must appear, must and must-not rules | Refund the duplicate to the original card; never ask for the full card number |
Context (RAG) | The documents a correct answer depends on | |
Tags | Slice, origin, conversation ID, why the case is hard |
|
Provenance | Who wrote or reviewed the expectation, and when | Billing lead, Nov 3 |
Here's one record as a line of JSONL, pretty-printed:
The expectations decide which evaluation metrics you can compute. A reference answer supports similarity scores, a facts list supports a correctness check, and a must-not rule can often be checked in code. expected_facts and guidelines are reserved keys that MLflow's built-in judges read, and DeepEval goldens take must and must_not lists for the same job.
Tags make an evaluation dataset debuggable. "Pass rate dropped from 88% to 82%" gives you nothing to act on. "Duplicate-charge cases fell from 95% to 70% while everything else held" tells you where to look. Write difficulty as a reason ("charge date arrives late") instead of a bare "hard," because a reason tells the next person what the case tests.
RAG cases need the source documents. Store the IDs of the passages a correct answer depends on, so you can score retrieval and generation separately, as our guide to RAG evaluation explains.
Multi-turn cases need a cut point. The input is the history up to the turn you're testing, and the expectation applies only to the next reply. One support conversation can yield three cases cut at different turns, which matters when you split the data. Agents add a trajectory, meaning the tools called and their order, which our guide to agent evaluation covers in depth.
Where to get examples
The best test cases come from real-world traffic your LLM application already handles. Mix sources, and tag each case with its origin so you can still tell the piles apart after you merge them.
Production logs and traces. Sample by slice, or a random pull will hand you 200 password resets and two duplicate charges. Redact personal data before a trace becomes a test case. MLflow can add traces to a dataset directly and mark each one with a TRACE source type.
Support tickets and escalations. Conversations that reached a human are a ready-made list of what the assistant got wrong or couldn't handle.
Domain experts. A billing lead can write, in an hour, the edge cases that rarely show up in logs: partial refunds, a chargeback already in progress, a plan changed mid-cycle.
User feedback. Thumbs-down replies point at failures. A thumbs-up proves less than it seems, since a customer will happily like a reply that promised a refund your policy doesn't allow.
Synthetic data. Generators fill gaps fast. DeepEval's Synthesizer builds goldens from your documents and "uses a series of evolution techniques" to make them more complex and realistic, and Promptfoo's red-team plugins generate adversarial inputs aimed at your app.
Treat generated test cases as drafts. Review them, tag them as synthetic, and check whether real and generated slices score differently. A model can do well on text that reads like model output and still stumble on how real customers write. For a first version, we'd lean on real traffic, add a layer of expert-written edge cases, and use synthetic cases only where real ones are rare.
How to build one, step by step
Define the task and the slices. Write one sentence for what the application must do, then list the slices you care about, such as duplicate charges, annual-plan refunds, and locked accounts. Fix the sampling rule before you look at any outputs, or you'll keep the cases that obviously pass.
Collect candidates from the sources above, with tags attached at collection time.
Label. Someone who knows the policy writes expectations as facts and rules. If an LLM-as-a-judge will score the open-ended parts, compare its verdicts with human labels on a sample first, since judges show position, verbosity, and self-enhancement biases. When two reviewers disagree on a case, the policy is usually ambiguous, so fix the policy note before the label.
Split. Keep a dev set for iterating and a held-out set you check rarely. Split by conversation: cases cut from the same conversation share most of their text, so one in dev and one in holdout leaks answers.
Version. Freeze the file and name the version after its content hash, so every score traces back to exact bytes.
Store it where your team and your eval tooling can both read it.
Steps 4 to 6 are where you create a dataset in a tool, and they fit in one short script. Install the tested packages with pip install mlflow==3.18.0 scikit-learn==1.9.1. Save your cases as support_cases.jsonl, with one compact JSON object per line and at least two distinct conversation IDs.
This one reads a JSONL file of records shaped like the example above, splits by conversation with scikit-learn, and saves each split to MLflow. The split, dataset creation, and repeated merge were tested with MLflow 3.18.0 and scikit-learn 1.9.1 on fictional cases, without model or API calls.
Two details are easy to miss. GroupShuffleSplit's test_size counts groups, not rows: 0.25 sends 4 of 16 conversations to holdout, but the number of cases depends on how many cut points those conversations contain. And merge_records matches records on a hash of their inputs, so merging the same cases into that dataset updates them instead of adding duplicates. Run create_dataset once for each version and split; reuse the resulting dataset when merging again.
The SQLite URI is there because MLflow evaluation datasets need a SQL backend and won't work with file-based tracking. If your team works in DeepEval instead, its save_as() writes a dataset to JSON, CSV, or JSONL, and our roundup of evaluation tools compares the other options.
How big should it be?
Start small and grow with failures. A few dozen test cases across your main slices is enough to catch obvious breakage and to debug the scoring code, and every real failure you hit afterward becomes a new case.
Size starts to matter when you compare two configurations. Take the Wilson interval for a pass rate. A configuration that passes 40 of 50 cases (80%) has a 95% interval of roughly 67% to 89%; one that passes 43 of 50 (86%) has an interval of roughly 74% to 93%. At 400 cases, 80% and 86% give about 76% to 84% and 82% to 89%.
These intervals illustrate uncertainty in each pass rate, not a test of the difference between configurations. When both run on the same cases, use a paired comparison, such as McNemar's test for binary pass/fail results, and set the decision rule before the run. Correlated cases from one conversation also need group-aware analysis.
Coverage beats count. Four hundred cases that are mostly password resets give you a tight number for the wrong question. Check counts per slice: a slice with eight cases moves 12.5 points when one case flips, so treat small-slice scores as an alarm that tells you to look closer.
Keeping it from going stale
An evaluation dataset decays in two ways: the product changes under it, and your team gradually learns its answers.
Feed it from production. Every escalation, bad rating or incident is a candidate case. The loop runs from production traces and failures to curated cases, then a versioned dataset, an eval run, and the new failures that run surfaces.

Turn production failures into curated, versioned cases, then use evaluation results to identify the next failures to capture.
Alt text (deployment only): Dataset maintenance loop: production traces and failures become curated, versioned cases split by conversation; evaluation identifies new failure cases to curate.
Version every change. Bump the version whenever you add, edit or remove cases, and log that version with each eval run. A score without a dataset version can't be compared with anything.
Guard the holdout. Each time you read holdout failures and tweak the prompt to fix them, the holdout quietly turns into a dev set. Tune on dev and check the holdout before a release. Cases used to guide changes belong in dev afterward. Refresh the assessment set using a predefined sampling rule, not by selectively replacing failures while retaining successes.
Watch for overfitting to the eval. Few-shot examples copied from the dataset, prompts that quote test inputs, and judge rubrics tuned on the same cases all inflate scores. Keep your example pool separate from your eval cases, and search new prompt text for strings that also appear in the dataset.
Public datasets worth knowing
When you want a public baseline, these are good places to start. Most live on the Hugging Face Hub, so the Hugging Face Datasets library loads one in a line: load_dataset("cais/mmlu", "all", split="test") returns MMLU's 14,042 test questions. Read each dataset card for license and splits before you build on it; TruthfulQA's Hub copy, for example, has only a validation split.
Purpose | Dataset | What it tests |
|---|---|---|
General knowledge | Multiple-choice questions across 57 tasks | |
General knowledge | 817 questions where a common misconception invites a false answer | |
Reasoning | Commonsense sentence completion that's trivial for humans | |
Reasoning | 8.5K grade-school math word problems | |
Safety and bias | 274k toxic and benign statements about 13 minority groups | |
Safety and bias | Social bias in question answering across nine dimensions | |
RAG and retrieval | 18 retrieval datasets, including FiQA and SciFact | |
Code | Functional correctness of Python written from docstrings | |
Agents | Tool-using agents talking to a simulated user, graded on final database state |
For LLM safety work, run these alongside red-team cases written for your own product, since a public toxicity set can't know what your assistant must refuse.
Using the dataset to compare configurations
The dataset is the fixed yardstick. The value comes from running many prompt, model, and retrieval variants against it and keeping the one that wins on the slices you care about, which is the workflow we built RapidFire for.
Our open-source engine shards the evaluation data and runs multiple configurations side by side on the same hardware. You write the scoring, and the dataset's tags carry straight into it. compute_metrics_fn receives each batch with every field your preprocess_fn passed through. Copy the slice tag along, and your metric function can count passes per slice, so a duplicate-charge regression shows up as its own line instead of a dip in one average.
Add an accumulate_metrics_fn, declare each metric's type and value range, and RapidFire shows running estimates with confidence intervals as each shard finishes. A configuration that's clearly losing can then be stopped before it spends the rest of your token budget.
In the open-source engine, you stop, resume, or clone-modify runs yourself from the dashboard or a notebook panel. Converge, our automation layer, stops what isn't working and clones what is within the time and spend limits you set, and logs the reason for every stop and clone.
Shuffle before you run. Shards are contiguous slices in the order you pass the data. If your file is sorted by slice, the first shard may hold a single slice and the early estimates come out biased, although the final full-dataset numbers are exact either way. Shuffle the dev split once, run your sweeps on it, and save the holdout for confirming the winner.
Common mistakes
A high score is only useful if the dataset represents real failures and the evaluation stays independent of tuning. Before you trust the result, check for these common mistakes.
Only testing the happy path. If every test case is a clean, single-turn question, the evaluation dataset can't catch the failures that reach support.
Splitting by row. Cut points from one conversation land on both sides, and the holdout score looks better than it is.
Trusting labels nobody reviewed. Synthetic cases and thumbs-up replies make good candidates and poor expectations.
Reporting one blended score. A 2-point overall drop can hide a 25-point drop on one slice.
Comparing scores across dataset versions. If the cases changed, the scores don't line up.
Tuning on the holdout. It stops measuring anything the first time you fix a prompt to pass it.
Conclusion
Start with the cases that have already burned you, review their expected outcomes, and freeze a first version. Split by conversation, track results by slice, and keep the holdout separate from day-to-day tuning so improvements mean something beyond the cases you just fixed.
When you're ready to compare configurations on that dataset, get started with the RapidFire RAG walkthrough. Adapt its FiQA or SciFact notebook to your own cases, preprocessing and scoring functions. The setup requires Python 3.12+; you don't need a local GPU if both embedding and generation use OpenAI APIs rather than self-hosted models.
FAQ
What is a golden dataset for LLM evaluation?
A golden dataset is the reviewed, frozen version of your evaluation dataset: inputs paired with expected outputs or criteria that a domain expert has checked. Teams rerun it after every prompt, model or retrieval change. DeepEval calls each reviewed case a golden; MLflow stores the same idea as inputs plus expectations.
Can you use synthetic data for LLM evaluation?
Yes, as a supplement. Synthetic data fills gaps where real traffic is thin, such as rare edge cases or adversarial inputs. Review every generated case before it gets an expected output, tag it as synthetic, and compare its scores with real cases, since generated text often reads differently from how customers write.
Where can I find free LLM evaluation datasets?
Most public benchmarks are free on the Hugging Face Hub, including MMLU, TruthfulQA, GSM8K and HumanEval, and you can load them with the Hugging Face Datasets library. Check each dataset card for its license and splits, and treat popular sets as possibly contaminated, since parts may have appeared in training data.