DSPy Tutorial: From Typed Program to Held-Out Evaluation
Written By:
Matt Tanner
Published on

A customer writes in: their API token authenticates fine, but every upload crashes. Your support router files it under account because the message mentions a token, and the ticket sits in the wrong queue for a day. Someone tweaks the prompt, the next ticket routes correctly, and nobody can say whether the fix helped or just moved the failure somewhere else.
DSPy is an open-source Python framework where you write LLM programs as typed signatures and modules, then let an optimizer improve their prompts against a metric you define. In this DSPy tutorial, you'll build a three-label ticket classifier, measure it, let GEPA try to rewrite its instruction, and check the result on tickets the optimizer never saw.
Spoiler: when we ran it against a hosted model, none of GEPA's rewrites beat the original instruction, so the script kept the baseline. That's a real outcome, and the workflow is built to make that call for you.
Set up a pinned DSPy environment and define the task
Each ticket belongs to exactly one queue: billing, technical, or account. Twelve synthetic tickets carry the whole tutorial, split six for training, three for validation, and three for a sealed test. They teach the API and the evaluation discipline, but they're not a benchmark, and they leave out the mixed-intent and vague tickets a real support queue is full of.
Pin the version of DSPy you test with. This tutorial uses DSPy 3.4.1 and GEPA 0.1.4 on Python 3.13. DSPy needs Python 3.10 or later, and it ships bundled native engines for provider calls, so you don't need a separate LiteLLM setup for the OpenAI route used here. The DSPy installation guide lists the other providers and their credential variables.
pip install dspy at that pin pulls in GEPA 0.1.4 as a dependency. The pip check line should print No broken requirements found. The OpenAI API key comes from the environment: the native OpenAI route reads OPENAI_API_KEY, so the key never appears in your source. These commands assume macOS or Linux.
Create ticket_demo.py and paste each Python block below into it, in order. The first one imports DSPy and connects it to a language model:
task_lm is the language model that classifies tickets, and reflection_lm is what GEPA uses later to write new instructions. Each setting matters when you count calls:
model_type="responses"sends requests to OpenAI's Responses endpoint through the native engine.cache=Falseturns off DSPy's answer cache. Caching is on by default, so a rerun would replay stored answers, and your call count would look smaller than a fresh run.num_retries=0overrides the default of 3, so a failed request isn't retried behind your back.store=Falseasks OpenAI not to store the responses for later retrieval, which the API otherwise does by default.temperature=0andreasoning_effort="none"match our run. Swap in your own model if you like, then rerun the evaluation.
This is how you configure DSPy once: dspy.configure sets the default language model and adapter for every predictor in the script.
Express the contract as a signature and a module
A DSPy signature declares what goes in and what comes out. Here, the input fields and output fields are typed, and the docstring becomes the task instruction DSPy puts in the prompt:
Classifier wraps the signature in something you can call. DSPy modules follow one pattern: subclass dspy.Module, call super().__init__(), create sub-modules in __init__, and write a forward that takes the signature's inputs as keyword arguments and returns a Prediction. Here, forward hands the ticket to dspy.Predict, which calls the model and adapter you configured and gives back a prediction with a category field. Predict is also a parameter optimizers can find, so GEPA needs no extra wiring.
The print line is a one-call smoke test before you spend anything on evaluation. For that duplicate-charge ticket, the expected result is:
The Literal type does real work. When we captured the request this script sends on the native OpenAI route, DSPy had included the three labels as a strict JSON schema enum. It also validates the parsed answer against the same set. When we fed a stub model the answer "refund", DSPy raised AdapterParseError instead of passing an invalid label downstream.
A well-formed answer can still be wrong, though: {"category": "account"} for a crashing upload parses perfectly. Types guarantee structure, and checking correctness is the metric's job in the next section.
You'll see dspy.ChainOfThought in most DSPy examples. The ChainOfThought module adds a chain-of-thought step before the output. In DSPy 3.4.1, it adds a reasoning field before the task output and returns it as result.reasoning. A three-label router doesn't need it.
Create examples, a metric, and isolated data splits
Each labeled ticket becomes a dspy.Example. The category is the ground truth for scoring, and with_inputs("ticket") tells DSPy that only the ticket goes to the model:
No ticket appears in two splits, and each split covers all three labels and has one job:
Training set (6): GEPA reads these tickets and the metric's feedback when it writes new instructions.
Validation set (3): decides whether a rewritten program beats the baseline.
Test set (3): opened once, after the choice is frozen, to check it on tickets nothing was tuned against.

Next comes the metric. Evaluation needs a score, and GEPA wants a short explanation as well:
exact_category compares the prediction with the ground truth label and returns 1.0 or 0.0. category_feedback wraps it in the five-argument form GEPA accepts and returns a dspy.Prediction with both a score and a feedback string. That string goes into the reflection prompt word for word, so it explains how you graded each answer alongside the inputs and outputs GEPA supplies. Ours only restates the label, which matters later.
Check the scorer before any model touches it. If it gets it wrong, GEPA will happily optimize toward the wrong labels while reporting progress. These four lines cost nothing to run:
You can't call anything an improvement without a number to beat. dspy.Evaluate scores a program over a devset with your metric:
In DSPy 3.4.1, calling an evaluator returns an EvaluationResult object. Its score is a percentage on a 0 to 100 scale, and results holds one (example, prediction, score) tuple per row, which is where you look for failures. The old return_outputs argument is gone and raises an error if you pass it.
When you evaluate the program, look for Average Metric: 3.0 / 3 (100.0%) in the log. That's what our run printed, so baseline_val.score was 100.0.
Keep runtime errors separate from wrong answers. If a provider call fails, Evaluate scores that row with failure_score, which defaults to 0.0, so an outage and a wrong label can look the same in the aggregate. Once errors reach max_errors (inherited from dspy.settings, 10 by default), it cancels the whole evaluation with an exception. Check the evaluation results in baseline_val.results for missing predictions before you read anything into a low score.
To see the exact prompt DSPy built, run dspy.inspect_history(n=1) after any call. It prints the system instructions, the user message, and the raw response.
Write down what produced the number, too: package versions, model, instruction text, dataset, and the settings above. A score without that context can't be compared with next week's.
Three validation tickets move the score in 33.33-point steps. One flipped answer takes you from 100.0 to 66.67, and no smaller gain can show up. Even larger sets stay noisy: RapidFire's guide to LLM evaluation metrics uses an 80% pass rate on 50 test cases, with a 95% Wilson interval of about 67% to 89%. Treat every score in this tutorial as a plumbing check.
Optimize the instruction with GEPA on separate training and validation sets
GEPA is DSPy's reflective instruction optimizer: it shows a reflection model a few training examples with their outputs and feedback, asks for a better instruction, then scores the rewrite. The model weights never change: compile returns a program with the best instruction written into each predictor's signature.instructions.
If your prompt is really missing good examples, BootstrapFewShot is the simpler tool: it keeps passing training traces as demonstrations, though its extra rounds force temperature=1.0. For the wider landscape of DSPy optimizers, see RapidFire's overview of prompt optimization.
This run skips GEPA's auto presets and sets a small budget by hand:
GEPA takes exactly one budget setting: auto, max_full_evals, or max_metric_calls. The GEPA optimization guide uses auto="light", but for this one-predictor program, GEPA 0.1.4 turns light into a target of 392 metric calls, a lot of paid requests for a twelve-ticket demo. The log prints that number at the start ("Running GEPA for approx N metric calls"), so check it before letting a preset run.
The other settings each matter here:
trainset=train, valset=valkeeps the roles separate. The training set feeds reflective updates, and the validation set picks which programcompilereturns. Leave outvalset,and GEPA reuses the training set for selection.skip_perfect_score=Falseis needed because of how easy these tickets are. With the defaultTrue, GEPA skips any minibatch where the current program already scores perfectly. When we replayed our run with the default, every iteration logged โAll subsample scores perfect for parent 0. Skipping.โand GEPA never proposed anything.track_stats=Trueattachesdetailed_resultsto the returned program, which is wheretotal_metric_callscomes from.reflection_minibatch_size=2shows the reflection model two tickets per iteration (default 3), anduse_merge=Falseskips GEPA's merge step.
What GEPA proposed, and why it lost
Our run made three proposals. The first two are summarized below; the third is shown verbatim.
Seed: Classify a support ticket as billing, technical, or account.
Proposal 1 expanded the billing and technical definitions with invoice-tax and webhook examples. Proposal 2 added workspace-access and API-error examples.
Iteration 3 proposal: Classify each support ticket into exactly one category: billing, technical, or account. Use account for login, password, or other account-access and account-management issues; use billing for invoices, charges, payments, renewals, and other payment-related issues; use technical for product or service malfunctions and troubleshooting issues. Return only a JSON object with the field category and its selected category as the value.
All three tied the seed at 2/2. The final rejection was:
The minibatch for iteration 3 was "I need to reset the password for my login" and "My renewal invoice charged me twice this month." The seed already got both right, so the longer instruction could at best tie at 2/2. GEPA's default acceptance rule only keeps a proposal whose minibatch score is strictly higher than the parent's.
GEPA rejected all three rewrites before full validation, so candidate kept the seed instruction. The later 100.0/100.0 compares that unchanged instruction.
A returned baseline can mean two things. Either GEPA's proposals lost, as above, or it never produced one, which logs Reflective mutation did not propose a new candidate. The returned program looks identical either way, so run .venv/bin/python ticket_demo.py 2>&1 | tee run.log and read the iteration lines. A win looks like Accepted candidate (subsample score 1.0 -> 2.0), which we saw in a stubbed run where the seed misrouted billing tickets.
Where the requests come from
max_metric_calls=12 is a target GEPA can overshoot. It checks the budget before each iteration, so an iteration that starts under the limit runs to completion. Our run used 15 metric calls: 3 to score the seed on validation, then 4 per iteration (2 tickets for the current program, 2 for the proposal). After two iterations, the count was 11, still under 12, so the third ran and finished at 15.
Your bill covers more than metric calls. On the three-rejected-proposal path shown here, the full ticket_demo.py, with the checks in the next two sections, sends 34 requests:
1 smoke test
3 baseline validations
3 for GEPA scoring the seed on validation
3 iterations ร (2 current + 1 reflection + 2 proposal) = 15
3 candidate validation
6 held-out (baseline and selected)
3 to check the restored program
Our live run made 30 requests: the list above without the smoke test or restored-program check. It used 8,498 input and 763 output tokens, an estimated $0.00123 at gpt-6-luna's published rates of $0.10 and $0.50 per million input and output tokens. This is a token-cost estimate, not a bill. Check current rates before scaling up.
Freeze the choice and evaluate it on held-out cases
Now score the candidate on the same validation set, apply the selection rule, and write it down before the test set is touched:
The baseline wins ties. A rewritten instruction that scores the same on validation hasn't earned the switch, and a longer prompt costs tokens on every call. Writing selection.json before the held-out evaluation means the test results can't influence which program you keep. If the selected program does worse on test, report it and fix the dataset, because tuning against the test set turns it into a second validation set.
We ran the classifier and GEPA loop against gpt-6-luna and saved all 30 requests and responses, including the strict JSON schema. Replaying the expanded script offline against those records reproduced the scores below: 33 requests matched saved request bodies exactly, including three repeated held-out calls for the restore check. The added smoke test used an illustrative response in that replay. It was not a new live run; your results can differ.
Split | Baseline | Selected (baseline) | Errors |
|---|---|---|---|
Validation (3 tickets) | 3/3 (100.0) | 3/3 (100.0), candidate tied | 0 |
Held-out test (3 tickets) | 3/3 (100.0) | 3/3 (100.0) | 0 |
On the held-out tickets, both programs returned billing for the receipt mismatch, technical for the database error, and account for the ownership transfer, all matching the expected labels.
What this run shows: the splits, the tie rule, the frozen selection, and the held-out check all work end to end, and GEPA's proposal loop ran and reported honestly.
What it doesn't show: that GEPA improves routing. Every proposal tied the seed at 2/2, because the seed was already right on every ticket it was shown. To give GEPA something to fix, the training and validation sets need tickets the baseline gets wrong: an expired card blocking a renewal, an auth error after a role change, a ticket that asks for two things at once. Those need a labeling policy first: does a mixed-intent ticket go to its first request, a fourth label, or a human?
Save learned state and explain the next experiment
Saving to a .json path stores the program's learned state, which here is the instruction text and an empty demo list. To restore it, create the same Classifier and load the file on top:
Replaying our run, that printed Restored held-out: 100.0, and the file's "lm" entry was null. This JSON doesn't hold your API key, provider choice, or temperature, so the restored program uses whatever model you configured with dspy.configure. It also doesn't hold the Classifier class, so the Python definition has to ship alongside it. DSPy writes its own version metadata into the file, so add the rest yourself: keep your dataset version, model name, and selection.json next to it.
Whole-program saving with save_program=True is a different format: it pickles the entire module, which means the saved directory contains executable Python. Only load those from sources you trust. Both Module.load() and dspy.load() refuse pickles unless you pass allow_pickle=True, so the JSON path above is the safer default. The saving and loading guide covers both formats. If you switch models, rerun the evaluation, because the old score doesn't carry over.
Scale the comparison past three tickets
This tutorial's weakest point is its own test set. With three tickets, scores move in 33-point steps, and a tie cannot establish equal accuracy on real tickets. On a few hundred real tickets, you'd want the seed instruction and GEPA's best rewrite scored side by side, with a way to tell when the gap between them is real.
That's the comparison RapidFire runs. It doesn't import a DSPy program. You paste each instruction's text into your RapidFire setup, and it runs the configurations in parallel with your own user-provided evaluation functions:
The
exact_categorylogic moves into acompute_metrics_fn. It's mandatory, takes a batch and returns a dictionary keyed by metric name, each with avaluefor that batch. Return correct and total counts per batch, then useaccumulate_metrics_fnto compute overall accuracy.Each instruction goes in through
preprocess_fn, which is also mandatory. The docs' example builds the system message fromprompt_manager.get_instructions().Online aggregation gives a running accuracy estimate with confidence intervals that narrow as each configuration sees more shards of the data. Declare accuracy with
is_algebraic=Trueandvalue_range=(0, 1)to enable those estimates; untyped metrics appear without confidence intervals. Shuffle the ticket file once first if it's grouped by queue, or the early estimates will be biased.IC Ops lets you stop the weaker configuration or clone-modify the stronger one mid-run. In the open-source engine, you trigger them yourself from the dashboard or notebook panel. Converge automates that loop: it stops versions that aren't working, clones the ones that are, and logs the reason for each decision, so you can compare configuration tradeoffs without running every sweep by hand.
The RAG and context engineering walkthrough is the quickest way to try it, and with OpenAI models only, it runs without a GPU. Retrieval and multi-step chains are natural next experiments in DSPy too. The habits from this tutorial carry straight over: change one thing at a time, score it the same way every run, and keep the baseline until something actually beats it.