Prompt Chaining: What It Is, Examples and How to Test It
Written By:
Matt Tanner
Published on
Alt text (deployment only): Three prompts extract incidents, assign cause labels, and summarize for a manager. Three checks validate parsing and required fields, allowed labels, and the absence of unsupported numbers.
Hand one prompt a week of on-call notes and ask it to pull out every incident, work out how long each lasted, tag the cause, and summarize it all for your manager. You'll get a confident paragraph back, and when one of its numbers is wrong, you can't tell which of those four jobs the model fumbled.
Prompt chaining splits that request into separate calls, so you can read, check, and score each step's output before the next step uses it. Below: how chains work, two worked examples with illustrative outputs, the difference from chain of thought, a Python runner with checks between steps, and how to test a chain.
What is prompt chaining?
Prompt chaining is a prompt engineering technique that splits a complex task into a sequence of smaller prompts, where the output of one prompt becomes the input of the next. Each prompt handles one subtask, such as extracting, labeling, or rewriting, and your code passes the result along, ideally after checking it.
Anthropic's guide to building effective agents adds the part many tutorials skip: you can put programmatic checks, which it calls gates, on any intermediate step. A single prompt does all the work in one generation, so you can only inspect the final answer, while a prompt chain gives you an inspection point after every step.
Why use a chain instead of one big prompt
One prompt is quicker to write and faster to run, so a chain has to earn its extra calls:
Accuracy per step. Anthropic frames the trade as giving up latency for accuracy "by making each LLM call an easier task."
Easier debugging. When the result is wrong, the intermediate outputs show which step broke, instead of leaving you to guess which line of a long prompt the model ignored.
Control over format. Each step can return JSON that your code validates before anything downstream runs.
Mixing models. Each call names its own model, so a cheap one can handle extraction.
Cost. Small models on easy steps save money, though every extra call pays for its own input tokens.
The best controlled comparison we found is narrow. Sun et al. (2024) compared a three-prompt draft, critique, and refine chain with one "stepwise" prompt doing all three, on 100 summarization tasks judged by GPT-4 against a one-step GPT-4 summary. The chained GPT-4 refinements won 77 of 100 comparisons and the single-prompt refinements won 53. Two caveats: the chained summaries were about 54% longer than the baseline on average, and the paper only tested summarization.
How prompt chaining works
Whatever framework you use, prompt chaining comes down to four decisions.
Subtasks. Split the complex task where the work changes kind: pulling facts out, labeling them, reasoning over them, writing for a reader. Anthropic's own example is an outline, a check that the outline meets your criteria, then the document.
Handoff format. Decide exactly what the next step receives. JSON with named fields is easier to check than prose, and passing only what the next prompt needs is context engineering applied to a single link. Structured outputs help: OpenAI's docs note that JSON mode only guarantees valid JSON, while Structured Outputs also enforce your schema.
Validation between steps. Code checks each output before it moves on: does it parse, are required fields present, are labels from the allowed set, are numbers plausible? A schema doesn't remove this job. OpenAI says your application still needs to handle refusals, incomplete responses, and business rules the schema can't express. Anthropic's structured outputs docs add that a refusal or a max_tokens stop can return output that doesn't match the schema, and that enum values can come back with different capitalization.
Error handling. Decide what a failed check triggers: a retry with the error message, a fallback model, or stopping the chain. Quietly passing bad output forward is the one choice that's never right.
Validate each handoff in the chain; retry a failed step once, then stop rather than passing a bad result downstream.
Alt text (deployment only): Three-step incident-analysis chain: extract incidents as JSON, add cause labels, and summarize for a manager. Checks validate fields and durations, allowed labels, and numeric grounding; failed steps retry once before stopping.
Prompt chaining examples
The two examples below use illustrative outputs to show what a chain can catch and what can slip through. Follow the handoffs as well as the final answer: a valid JSON object or an allowed label can still contain the wrong fact.
Example 1: Extract, transform, analyze and summarize incident data
The input is a short set of on-call notes:
On-callnotes,week41(payments team)Mon02:10checkout-apireturning5xxafterdeploy4812. Rolledback,healthyat02:24.
Tue14:05ledger-workerqueuebackingup,Postgresconnectionsmaxedout. Raisedpoolsize,queuedrainedby14:50.
Wed09:30Callstothecardprocessortimingoutforabout 20 minutes. Their status page showed a degraded API. Nothing to do on our side.
Thu23:40checkout-api5xxagain. Featureflagnew_tax_ruleswasswitchedoninprodbymistake. Flagoffat23:52.
Fri11:15ledger-workerOOM-killedtwiceafterthebatchsizechange;512Milimittoolow. Bumpedto1Gi,stablesince11:35
On-callnotes,week41(payments team)Mon02:10checkout-apireturning5xxafterdeploy4812. Rolledback,healthyat02:24.
Tue14:05ledger-workerqueuebackingup,Postgresconnectionsmaxedout. Raisedpoolsize,queuedrainedby14:50.
Wed09:30Callstothecardprocessortimingoutforabout 20 minutes. Their status page showed a degraded API. Nothing to do on our side.
Thu23:40checkout-api5xxagain. Featureflagnew_tax_ruleswasswitchedoninprodbymistake. Flagoffat23:52.
Fri11:15ledger-workerOOM-killedtwiceafterthebatchsizechange;512Milimittoolow. Bumpedto1Gi,stablesince11:35
On-callnotes,week41(payments team)Mon02:10checkout-apireturning5xxafterdeploy4812. Rolledback,healthyat02:24.
Tue14:05ledger-workerqueuebackingup,Postgresconnectionsmaxedout. Raisedpoolsize,queuedrainedby14:50.
Wed09:30Callstothecardprocessortimingoutforabout 20 minutes. Their status page showed a degraded API. Nothing to do on our side.
Thu23:40checkout-api5xxagain. Featureflagnew_tax_ruleswasswitchedoninprodbymistake. Flagoffat23:52.
Fri11:15ledger-workerOOM-killedtwiceafterthebatchsizechange;512Milimittoolow. Bumpedto1Gi,stablesince11:35
Prompt 1 (extract) asks for raw values only, since code calculates durations later:
Extracteveryincidentfromthe on-callnotes.
Return only a JSON array,no other text. One object per incident withkeys:
"day","service","start"and"end"(clock times exactly as written,ornull),"stated_minutes"(integerifthe notes state a duration,elsenull),"what_happened"(one short sentence). Copy values;don't calculate anything.[{"day": "Mon","service": "checkout-api","start": "02:10","end": "02:24","stated_minutes": null,"what_happened": "Rollback of deployment 4812 completed successfully."},{"day": "Tue","service": "ledger-worker","start": "14:05","end": "14:50","stated_minutes": null,"what_happened": "Queue backed up due to Postgres connection issues. Queue drained by 14:50."},{"day": "Wed","service": "card processor","start": "09:30","end": null,"stated_minutes": null,"what_happened": "Calls to the card processor timed out for approximately 20 minutes. Status page indicated a degraded API."},...twomoreincidents...]
Extracteveryincidentfromthe on-callnotes.
Return only a JSON array,no other text. One object per incident withkeys:
"day","service","start"and"end"(clock times exactly as written,ornull),"stated_minutes"(integerifthe notes state a duration,elsenull),"what_happened"(one short sentence). Copy values;don't calculate anything.[{"day": "Mon","service": "checkout-api","start": "02:10","end": "02:24","stated_minutes": null,"what_happened": "Rollback of deployment 4812 completed successfully."},{"day": "Tue","service": "ledger-worker","start": "14:05","end": "14:50","stated_minutes": null,"what_happened": "Queue backed up due to Postgres connection issues. Queue drained by 14:50."},{"day": "Wed","service": "card processor","start": "09:30","end": null,"stated_minutes": null,"what_happened": "Calls to the card processor timed out for approximately 20 minutes. Status page indicated a degraded API."},...twomoreincidents...]
Extracteveryincidentfromthe on-callnotes.
Return only a JSON array,no other text. One object per incident withkeys:
"day","service","start"and"end"(clock times exactly as written,ornull),"stated_minutes"(integerifthe notes state a duration,elsenull),"what_happened"(one short sentence). Copy values;don't calculate anything.[{"day": "Mon","service": "checkout-api","start": "02:10","end": "02:24","stated_minutes": null,"what_happened": "Rollback of deployment 4812 completed successfully."},{"day": "Tue","service": "ledger-worker","start": "14:05","end": "14:50","stated_minutes": null,"what_happened": "Queue backed up due to Postgres connection issues. Queue drained by 14:50."},{"day": "Wed","service": "card processor","start": "09:30","end": null,"stated_minutes": null,"what_happened": "Calls to the card processor timed out for approximately 20 minutes. Status page indicated a degraded API."},...twomoreincidents...]
A model may wrap this array in a markdown code fence, which the parser strips. The illustrative extraction copies the clock times correctly. It missed Wednesday's stated duration, so the code records that one as unknown and computes 14, 45, 12, and 20 minutes for the rest.
Prompt 2 (transform) adds one label per incident from a fixed list:
Adda"cause"key to each incident in the JSON array.
Use exactly one of: "deploy", "config", "capacity", "dependency".
Return only the updated JSON array.
Adda"cause"key to each incident in the JSON array.
Use exactly one of: "deploy", "config", "capacity", "dependency".
Return only the updated JSON array.
Adda"cause"key to each incident in the JSON array.
Use exactly one of: "deploy", "config", "capacity", "dependency".
Return only the updated JSON array.
Its output, condensed to one line per incident (durations from code):
All five labels passed the allowed-values check, but we'd call Tuesday capacity (the connection pool maxed out) and Wednesday a dependency failure. Code then sums minutes per cause: config 57, capacity 20, deploy 14.
Prompt 3 (analyze) gets the labeled incidents plus those totals:
In this illustrative sequence, the first answer claims checkout-api has "a total downtime of 126 minutes (or 2 hours and 6 minutes)," and the summary check rejects it because 126, 2, and 6 aren't in the input. The retry passes the check but still gives ledger-worker "a cumulative downtime of 57 minutes" (the mislabeled config total) and moves the card processor outage to Thursday.
The parser handled the code fence, and a check caught the invented number, but nothing caught the rest: the wrong label from step 2 flowed into the totals and then into the summary.
Example 2: Draft, critique and rewrite a support reply
Claude's prompting docs call this self-correction loop (draft, review against criteria, refine) the most common prompt chaining pattern. The input:
Draft: Writeareplytothecustomerusingonlyfactsfromthecasenotes.
Keepitunder120words. Returnonlythereply.
Critique: Checkthedraftreplyagainstthecaseandthischecklist:
1. Answers every question the customer asked.
2. States only facts that appear inthe case notes.
3. Gives the refund timeline.
4. Under120 words.
List each problem on its own line. If there are none,reply exactly: OK
Rewrite: Rewrite the draft reply so it fixes every problem listed inthe
critique. Use only facts fromthecasenotes. Returnonlythereply
Draft: Writeareplytothecustomerusingonlyfactsfromthecasenotes.
Keepitunder120words. Returnonlythereply.
Critique: Checkthedraftreplyagainstthecaseandthischecklist:
1. Answers every question the customer asked.
2. States only facts that appear inthe case notes.
3. Gives the refund timeline.
4. Under120 words.
List each problem on its own line. If there are none,reply exactly: OK
Rewrite: Rewrite the draft reply so it fixes every problem listed inthe
critique. Use only facts fromthecasenotes. Returnonlythereply
Draft: Writeareplytothecustomerusingonlyfactsfromthecasenotes.
Keepitunder120words. Returnonlythereply.
Critique: Checkthedraftreplyagainstthecaseandthischecklist:
1. Answers every question the customer asked.
2. States only facts that appear inthe case notes.
3. Gives the refund timeline.
4. Under120 words.
List each problem on its own line. If there are none,reply exactly: OK
Rewrite: Rewrite the draft reply so it fixes every problem listed inthe
critique. Use only facts fromthecasenotes. Returnonlythereply
Consider a draft that gets the cause and refund timeline right, then adds "Please note that similar issues were encountered last month as well." The notes say January. If the critique step answers OK, that error can go straight to the customer.
A stronger critique prompt demands evidence: for each factual sentence in the draft, quote the supporting case-note words or write UNSUPPORTED. That still needs checking: the critic could quote the customer's message instead of evaluating the draft. Even a rewrite that drops the unsupported sentence can leave another gap:
It never acknowledges this is the customer's second duplicate, which they raised directly. A critic needs its own test set, just like the writer.
Prompt chaining vs. chain of thought
Chain-of-thought prompting, introduced by Wei et al. in 2022, gets the model to write "a series of intermediate reasoning steps" before its answer, originally by putting a few worked examples in the prompt. That all happens inside one response, while a chain spreads the work across calls your code connects.
Prompt chaining
Chain of thought
Where the steps live
Separate model calls
One response
Who defines the steps
You, in code
The model, as it writes
Where your code can step in
After every call
Only after the whole response
Cost shape
One call per step
One call with a longer output
Best fit
Pipelines with distinct jobs (extract, label, write)
One reasoning problem (math, logic)
The two combine well, since any step in a chain can use chain of thought. Self-consistency builds on chain of thought differently: it samples several reasoning paths and keeps the most consistent answer. Meta prompting, in Suzgun and Kalai's sense, has one model act as a conductor that splits the task and hands subtasks to "expert" instances of itself. The split happens at runtime there, while a chain fixes it in code.
How to build a prompt chain
Define the goal. Write down what a correct final output looks like, ideally something code can check, such as "three sentences, every number traceable to the notes."
Split the work into subtasks. Give each prompt one job, and keep arithmetic, sorting, and counting in code. For these notes, the four computable durations are 14, 45, 12, and 20 minutes. Copy the start and end times and subtract them in Python rather than asking the model to do the arithmetic.
Plan the handoff. Name the fields each step returns and decide what "unknown" looks like (we used null). If a step keeps misreading its input, add worked examples to its prompt; our guide to few-shot prompting covers how to pick them.
Add checks. Parse, then check required keys, allowed values, plausible ranges and, for generated text, cheap rules like "no numbers that weren't in the input."
Iterate one link at a time. Change one prompt, rerun the same inputs, and compare.
Here is a runner for example 1, with validation before each handoff. It uses the OpenAI Python SDK's Responses API. For a local endpoint, set OPENAI_BASE_URL only if the server implements that API, not just Chat Completions.
Use Python 3.10 or later and install the SDK with python -m pip install openai. Save the runner as chain.py, the example's on-call notes as oncall_notes.txt, and the three prompt texts as triple-quoted strings named EXTRACT, TRANSFORM, and ANALYZE in prompts.py in the same directory.
Set OPENAI_API_KEY to your provider credential and CHAIN_MODEL to a Responses-capable model available to your account, then run python chain.py. A hosted provider may charge for the calls. This example handles same-day clock times in one time zone; cross-midnight incidents need dates as well as times.
importjson,os,refromcollectionsimportCounterfromopenaiimportOpenAIfrompromptsimportEXTRACT,TRANSFORM,ANALYZE# the prompt text shown aboveclient = OpenAI()# reads OPENAI_API_KEY and, if set, OPENAI_BASE_URLMODEL = os.environ["CHAIN_MODEL"]CAUSES = {"deploy","config","capacity","dependency"}defcall(instructions,text):
returnclient.responses.create(model=MODEL,instructions=instructions,input=text).output_textdefparse_json(raw):
raw = raw.strip()ifraw.startswith("`"): # small models often wrap JSON in a markdown fenceraw = raw.strip("`").removeprefix("json")returnjson.loads(raw)# json.JSONDecodeError is a ValueErrordefclock_minutes(value):
ifnotisinstance(value,str)ornotre.fullmatch(r"(?:[01][0-9]|2[0-3]):[0-5][0-9]",value):
raiseValueError("expected HH:MM in 24-hour time")hours,mins = map(int,value.split(":"))returnhours * 60 + minsdefminutes(start,end):
# This example assumes same-day incidents in one time zone.duration = clock_minutes(end) - clock_minutes(start)ifduration <= 0:
raiseValueError("end must follow start on the same day")returndurationdefcheck_extract(raw):
items = parse_json(raw)ifnotisinstance(items,list)ornotitems:
raiseValueError("expected a non-empty JSON array")foritinitems:
ifnotisinstance(it,dict):
raiseValueError("each incident must be an object")missing = {"day","service","start","end","stated_minutes"} - set(it)ifmissing:
raiseValueError(f"missing keys {sorted(missing)}")forkeyin("day","service"):
ifnotisinstance(it[key],str)ornotit[key].strip():
raiseValueError(f"expected non-empty {key}")forkeyin("start","end"):
ifit[key]isnotNone:
clock_minutes(it[key])stated = it["stated_minutes"]ifstatedisnotNoneandnot(type(stated)isintandstated > 0):
raiseValueError("stated_minutes must be a positive integer or null")# Durations are computed here, never by the model. None means unknown.both = it["start"]isnotNoneandit["end"]isnotNoneit["minutes"] = minutes(it["start"],it["end"])ifbothelseit["stated_minutes"]m = it["minutes"]ifmisnotNoneandnot(type(m)isintandm > 0):
raiseValueError(f"bad duration on {it['day']}")returnitemsdefcheck_transform(raw):
items = check_extract(raw)foritinitems:
it["cause"] = str(it.get("cause")).strip().lower()# tolerate casing driftifit["cause"]notinCAUSES:
raiseValueError(f"unknown cause {it['cause']!r}")returnitemsdefcheck_summary(raw,facts):
invented = set(re.findall(r"\d+",raw)) - set(re.findall(r"\d+",facts))ifinvented:
raiseValueError(f"numbers not in the input: {sorted(invented)}")returnraw.strip()defrun_step(instructions,text,check,retries=1):
"""Call the model, check the output, retry once with the error, then stop."""prompt = textfor_inrange(retries + 1):
raw = call(instructions,prompt)try:
returncheck(raw)exceptValueErroraserr:
error = errprompt = f"{text}\n\nYour last answer was rejected: {err}. Fix it."raiseRuntimeError(f"step failed after {retries + 1} tries: {error}")notes = open("oncall_notes.txt").read()incidents = run_step(EXTRACT,notes,check_extract)labeled = run_step(TRANSFORM,json.dumps(incidents),check_transform)totals = Counter()foritinlabeled:
totals[it["cause"]] += it["minutes"]or0# arithmetic stays in codefacts = json.dumps({"incidents": labeled,"minutes_by_cause": totals})print(run_step(ANALYZE,facts,lambdaraw: check_summary(raw,facts)))
importjson,os,refromcollectionsimportCounterfromopenaiimportOpenAIfrompromptsimportEXTRACT,TRANSFORM,ANALYZE# the prompt text shown aboveclient = OpenAI()# reads OPENAI_API_KEY and, if set, OPENAI_BASE_URLMODEL = os.environ["CHAIN_MODEL"]CAUSES = {"deploy","config","capacity","dependency"}defcall(instructions,text):
returnclient.responses.create(model=MODEL,instructions=instructions,input=text).output_textdefparse_json(raw):
raw = raw.strip()ifraw.startswith("`"): # small models often wrap JSON in a markdown fenceraw = raw.strip("`").removeprefix("json")returnjson.loads(raw)# json.JSONDecodeError is a ValueErrordefclock_minutes(value):
ifnotisinstance(value,str)ornotre.fullmatch(r"(?:[01][0-9]|2[0-3]):[0-5][0-9]",value):
raiseValueError("expected HH:MM in 24-hour time")hours,mins = map(int,value.split(":"))returnhours * 60 + minsdefminutes(start,end):
# This example assumes same-day incidents in one time zone.duration = clock_minutes(end) - clock_minutes(start)ifduration <= 0:
raiseValueError("end must follow start on the same day")returndurationdefcheck_extract(raw):
items = parse_json(raw)ifnotisinstance(items,list)ornotitems:
raiseValueError("expected a non-empty JSON array")foritinitems:
ifnotisinstance(it,dict):
raiseValueError("each incident must be an object")missing = {"day","service","start","end","stated_minutes"} - set(it)ifmissing:
raiseValueError(f"missing keys {sorted(missing)}")forkeyin("day","service"):
ifnotisinstance(it[key],str)ornotit[key].strip():
raiseValueError(f"expected non-empty {key}")forkeyin("start","end"):
ifit[key]isnotNone:
clock_minutes(it[key])stated = it["stated_minutes"]ifstatedisnotNoneandnot(type(stated)isintandstated > 0):
raiseValueError("stated_minutes must be a positive integer or null")# Durations are computed here, never by the model. None means unknown.both = it["start"]isnotNoneandit["end"]isnotNoneit["minutes"] = minutes(it["start"],it["end"])ifbothelseit["stated_minutes"]m = it["minutes"]ifmisnotNoneandnot(type(m)isintandm > 0):
raiseValueError(f"bad duration on {it['day']}")returnitemsdefcheck_transform(raw):
items = check_extract(raw)foritinitems:
it["cause"] = str(it.get("cause")).strip().lower()# tolerate casing driftifit["cause"]notinCAUSES:
raiseValueError(f"unknown cause {it['cause']!r}")returnitemsdefcheck_summary(raw,facts):
invented = set(re.findall(r"\d+",raw)) - set(re.findall(r"\d+",facts))ifinvented:
raiseValueError(f"numbers not in the input: {sorted(invented)}")returnraw.strip()defrun_step(instructions,text,check,retries=1):
"""Call the model, check the output, retry once with the error, then stop."""prompt = textfor_inrange(retries + 1):
raw = call(instructions,prompt)try:
returncheck(raw)exceptValueErroraserr:
error = errprompt = f"{text}\n\nYour last answer was rejected: {err}. Fix it."raiseRuntimeError(f"step failed after {retries + 1} tries: {error}")notes = open("oncall_notes.txt").read()incidents = run_step(EXTRACT,notes,check_extract)labeled = run_step(TRANSFORM,json.dumps(incidents),check_transform)totals = Counter()foritinlabeled:
totals[it["cause"]] += it["minutes"]or0# arithmetic stays in codefacts = json.dumps({"incidents": labeled,"minutes_by_cause": totals})print(run_step(ANALYZE,facts,lambdaraw: check_summary(raw,facts)))
importjson,os,refromcollectionsimportCounterfromopenaiimportOpenAIfrompromptsimportEXTRACT,TRANSFORM,ANALYZE# the prompt text shown aboveclient = OpenAI()# reads OPENAI_API_KEY and, if set, OPENAI_BASE_URLMODEL = os.environ["CHAIN_MODEL"]CAUSES = {"deploy","config","capacity","dependency"}defcall(instructions,text):
returnclient.responses.create(model=MODEL,instructions=instructions,input=text).output_textdefparse_json(raw):
raw = raw.strip()ifraw.startswith("`"): # small models often wrap JSON in a markdown fenceraw = raw.strip("`").removeprefix("json")returnjson.loads(raw)# json.JSONDecodeError is a ValueErrordefclock_minutes(value):
ifnotisinstance(value,str)ornotre.fullmatch(r"(?:[01][0-9]|2[0-3]):[0-5][0-9]",value):
raiseValueError("expected HH:MM in 24-hour time")hours,mins = map(int,value.split(":"))returnhours * 60 + minsdefminutes(start,end):
# This example assumes same-day incidents in one time zone.duration = clock_minutes(end) - clock_minutes(start)ifduration <= 0:
raiseValueError("end must follow start on the same day")returndurationdefcheck_extract(raw):
items = parse_json(raw)ifnotisinstance(items,list)ornotitems:
raiseValueError("expected a non-empty JSON array")foritinitems:
ifnotisinstance(it,dict):
raiseValueError("each incident must be an object")missing = {"day","service","start","end","stated_minutes"} - set(it)ifmissing:
raiseValueError(f"missing keys {sorted(missing)}")forkeyin("day","service"):
ifnotisinstance(it[key],str)ornotit[key].strip():
raiseValueError(f"expected non-empty {key}")forkeyin("start","end"):
ifit[key]isnotNone:
clock_minutes(it[key])stated = it["stated_minutes"]ifstatedisnotNoneandnot(type(stated)isintandstated > 0):
raiseValueError("stated_minutes must be a positive integer or null")# Durations are computed here, never by the model. None means unknown.both = it["start"]isnotNoneandit["end"]isnotNoneit["minutes"] = minutes(it["start"],it["end"])ifbothelseit["stated_minutes"]m = it["minutes"]ifmisnotNoneandnot(type(m)isintandm > 0):
raiseValueError(f"bad duration on {it['day']}")returnitemsdefcheck_transform(raw):
items = check_extract(raw)foritinitems:
it["cause"] = str(it.get("cause")).strip().lower()# tolerate casing driftifit["cause"]notinCAUSES:
raiseValueError(f"unknown cause {it['cause']!r}")returnitemsdefcheck_summary(raw,facts):
invented = set(re.findall(r"\d+",raw)) - set(re.findall(r"\d+",facts))ifinvented:
raiseValueError(f"numbers not in the input: {sorted(invented)}")returnraw.strip()defrun_step(instructions,text,check,retries=1):
"""Call the model, check the output, retry once with the error, then stop."""prompt = textfor_inrange(retries + 1):
raw = call(instructions,prompt)try:
returncheck(raw)exceptValueErroraserr:
error = errprompt = f"{text}\n\nYour last answer was rejected: {err}. Fix it."raiseRuntimeError(f"step failed after {retries + 1} tries: {error}")notes = open("oncall_notes.txt").read()incidents = run_step(EXTRACT,notes,check_extract)labeled = run_step(TRANSFORM,json.dumps(incidents),check_transform)totals = Counter()foritinlabeled:
totals[it["cause"]] += it["minutes"]or0# arithmetic stays in codefacts = json.dumps({"incidents": labeled,"minutes_by_cause": totals})print(run_step(ANALYZE,facts,lambdaraw: check_summary(raw,facts)))
If your provider supports structured outputs, pass the schema too and keep the checks. To skip the hand-rolled plumbing, LangChain and LangGraph both support this pattern, and LangGraph's docs list prompt chaining as a workflow. Our LangGraph tutorial covers the building blocks: nodes do the work, edges decide what runs next, and shared state carries data between steps.
Drawbacks and failure modes
Latency. Calls run in sequence, so latencies add up. Three steps plus one retry require four model calls; the elapsed time depends on the model and deployment.
Cost. Each link pays for its own input tokens, handoffs often repeat context, and retries add calls you didn't budget for.
Error propagation. A wrong output that passes its check becomes the next step's ground truth. Example 1's Tuesday label turned into a wrong total in code and then a wrong claim in the summary.
Brittle formats. Parsers break on code fences, extra prose, renamed keys or changed casing, and a model or prompt change can introduce any of them.
Retries that make things worse. A stricter extraction check could reject an incident with no usable duration, only for the retry to "fix" it by returning that incident and dropping the other four. Check that a retry didn't shrink the output.
Critics that rubber-stamp. Example 2's critique approved a draft with an unsupported claim in it.
When not to chain
Short tasks. Before you create a chain, check whether one prompt already scores well on your evals. If it does, prompt chaining only adds calls, code and failure points. Anthropic's advice is to add complexity "only when it demonstrably improves outcomes."
Tasks a newer model handles in one call. Claude's prompting docs say that with adaptive thinking and subagent orchestration, Claude handles most multistep reasoning internally. They keep explicit chains for inspecting intermediate outputs or enforcing a pipeline. Run the single call and the chain on the same test set before committing.
Fully dynamic flows. If the next step depends on what the model finds, such as which tool to call or how many searches to run, you want an agent. Anthropic separates workflows, which follow "predefined code paths," from agents, where models "dynamically direct their own processes and tool usage." The loop, tools, and stopping rules around that model are its agent harness.
Routing sits in between: in Anthropic's terms, it classifies an input and directs it to a specialized follow-up task, which can be a chain of its own.
How to test and tune a chain
A chain needs a score per step and one for the end result. In example 1, every step passed its checks, and the summary was still wrong.
Score each step against labels. Build a small labeled set per link, such as notes with known incidents for extraction and incidents with agreed causes for the transform step. Measure field-level accuracy, not just validity: our transform step returned 5 valid labels, and we'd accept 3.
Score the end result with a rubric or deterministic checks, because decent steps can still add up to a bad answer.
Track check failures and retries per step, plus cost per successful run, to find the fragile link.
Compare variants on the same inputs. Freeze one step's inputs (saved upstream outputs or hand-labeled ones) and run its prompt or model variants on that set, then retest the downstream links, since they now see different inputs. The prompt optimization loop applies to each link.
Our open-source RapidFire engine is built for this comparison work. Its eval pipeline produces one generation per example: your preprocess_fn builds the prompts, an optional postprocess_fn parses the output (a natural home for the checks above), and your compute_metrics_fn scores each batch. So you test a chain one link at a time: make a link's saved inputs the eval set and put its prompt or model variants into one grid.
The engine shards the eval set and runs every config on one shard at a time, so variants post early scores side by side. Declare each metric's type and value range and you also get running estimates with confidence intervals. With MLflow tracing on (the local default), traces show the exact prompt each config sent and what came back.
In the open-source engine, you stop weak configs or clone and modify strong ones yourself with IC Ops. Converge automates those calls: it stops weak versions, clones strong ones, logs the reason for each decision, and stays within the time and spend limits you set. To try the engine, start with the RapidFire RAG walkthrough, which has a Colab notebook and a local install path.
FAQ
What is the main difference between prompt chaining and chain of thought prompting?
Prompt chaining splits a task across several model calls, and your code passes each output to the next prompt, so you can check every intermediate result. Chain-of-thought prompting keeps the work in one call and has the model write out its reasoning first. Any step of a chain can use chain of thought.
What are the 5 P's of prompting?
There's no standard list. It's a mnemonic from prompt engineering courses, and the courses don't agree: one version is prime, persona, privacy, product, and polish, while another is persona, precision, parameters, process, and polish. Use whichever you learned as a checklist when you create a prompt for each step, then judge it by measured results.
Can you provide an example of a chain of prompts?
For weekly incident notes, prompt 1 extracts each incident as JSON with its service and start and end times. Prompt 2 labels each incident's cause from a fixed list, and prompt 3 writes a three-sentence summary from the labeled incidents plus totals calculated in code. Code checks the JSON and labels between steps. Example 1 above shows the prompts and illustrative outputs.
What is chaining in AI?
Chaining means connecting model calls, tools, or code so each step's output feeds the next. Prompt chaining is the simplest form, a fixed sequence of prompts. Frameworks such as LangChain and LangGraph provide building blocks for it, and agents go further by letting the model pick the next step.