rapidfire.

Prompt Chaining: What It Is, Examples and How to Test It

Written By:

Matt Tanner

Published on


Prompt Chaining: What It Is, Examples and How to Test It โ€” diagram 1

Alt text (deployment only): Three prompts extract incidents, assign cause labels, and summarize for a manager. Three checks validate parsing and required fields, allowed labels, and the absence of unsupported numbers.

Hand one prompt a week of on-call notes and ask it to pull out every incident, work out how long each lasted, tag the cause, and summarize it all for your manager. You'll get a confident paragraph back, and when one of its numbers is wrong, you can't tell which of those four jobs the model fumbled.

Prompt chaining splits that request into separate calls, so you can read, check, and score each step's output before the next step uses it. Below: how chains work, two worked examples with illustrative outputs, the difference from chain of thought, a Python runner with checks between steps, and how to test a chain.

What is prompt chaining?

Prompt chaining is a prompt engineering technique that splits a complex task into a sequence of smaller prompts, where the output of one prompt becomes the input of the next. Each prompt handles one subtask, such as extracting, labeling, or rewriting, and your code passes the result along, ideally after checking it.

Anthropic's guide to building effective agents adds the part many tutorials skip: you can put programmatic checks, which it calls gates, on any intermediate step. A single prompt does all the work in one generation, so you can only inspect the final answer, while a prompt chain gives you an inspection point after every step.

Why use a chain instead of one big prompt

One prompt is quicker to write and faster to run, so a chain has to earn its extra calls:

  • Accuracy per step. Anthropic frames the trade as giving up latency for accuracy "by making each LLM call an easier task."

  • Easier debugging. When the result is wrong, the intermediate outputs show which step broke, instead of leaving you to guess which line of a long prompt the model ignored.

  • Control over format. Each step can return JSON that your code validates before anything downstream runs.

  • Mixing models. Each call names its own model, so a cheap one can handle extraction.

  • Cost. Small models on easy steps save money, though every extra call pays for its own input tokens.

The best controlled comparison we found is narrow. Sun et al. (2024) compared a three-prompt draft, critique, and refine chain with one "stepwise" prompt doing all three, on 100 summarization tasks judged by GPT-4 against a one-step GPT-4 summary. The chained GPT-4 refinements won 77 of 100 comparisons and the single-prompt refinements won 53. Two caveats: the chained summaries were about 54% longer than the baseline on average, and the paper only tested summarization.

How prompt chaining works

Whatever framework you use, prompt chaining comes down to four decisions.

Subtasks. Split the complex task where the work changes kind: pulling facts out, labeling them, reasoning over them, writing for a reader. Anthropic's own example is an outline, a check that the outline meets your criteria, then the document.

Handoff format. Decide exactly what the next step receives. JSON with named fields is easier to check than prose, and passing only what the next prompt needs is context engineering applied to a single link. Structured outputs help: OpenAI's docs note that JSON mode only guarantees valid JSON, while Structured Outputs also enforce your schema.

Validation between steps. Code checks each output before it moves on: does it parse, are required fields present, are labels from the allowed set, are numbers plausible? A schema doesn't remove this job. OpenAI says your application still needs to handle refusals, incomplete responses, and business rules the schema can't express. Anthropic's structured outputs docs add that a refusal or a max_tokens stop can return output that doesn't match the schema, and that enum values can come back with different capitalization.

Error handling. Decide what a failed check triggers: a retry with the error message, a fallback model, or stopping the chain. Quietly passing bad output forward is the one choice that's never right.


Prompt Chaining: What It Is, Examples and How to Test It โ€” diagram 2

Validate each handoff in the chain; retry a failed step once, then stop rather than passing a bad result downstream.

Alt text (deployment only): Three-step incident-analysis chain: extract incidents as JSON, add cause labels, and summarize for a manager. Checks validate fields and durations, allowed labels, and numeric grounding; failed steps retry once before stopping.

Prompt chaining examples

The two examples below use illustrative outputs to show what a chain can catch and what can slip through. Follow the handoffs as well as the final answer: a valid JSON object or an allowed label can still contain the wrong fact.

Example 1: Extract, transform, analyze and summarize incident data

The input is a short set of on-call notes:

On-call notes, week 41 (payments team)

Mon 02:10 checkout-api returning 5xx after deploy 4812. Rolled back, healthy at 02:24.
Tue 14:05 ledger-worker queue backing up, Postgres connections maxed out. Raised pool size, queue drained by 14:50.
Wed 09:30 Calls to the card processor timing out for about 20 minutes. Their status page showed a degraded API. Nothing to do on our side.
Thu 23:40 checkout-api 5xx again. Feature flag new_tax_rules was switched on in prod by mistake. Flag off at 23:52.
Fri 11:15 ledger-worker OOM-killed twice after the batch size change; 512Mi limit too low. Bumped to 1Gi, stable since 11:35

On-call notes, week 41 (payments team)

Mon 02:10 checkout-api returning 5xx after deploy 4812. Rolled back, healthy at 02:24.
Tue 14:05 ledger-worker queue backing up, Postgres connections maxed out. Raised pool size, queue drained by 14:50.
Wed 09:30 Calls to the card processor timing out for about 20 minutes. Their status page showed a degraded API. Nothing to do on our side.
Thu 23:40 checkout-api 5xx again. Feature flag new_tax_rules was switched on in prod by mistake. Flag off at 23:52.
Fri 11:15 ledger-worker OOM-killed twice after the batch size change; 512Mi limit too low. Bumped to 1Gi, stable since 11:35

On-call notes, week 41 (payments team)

Mon 02:10 checkout-api returning 5xx after deploy 4812. Rolled back, healthy at 02:24.
Tue 14:05 ledger-worker queue backing up, Postgres connections maxed out. Raised pool size, queue drained by 14:50.
Wed 09:30 Calls to the card processor timing out for about 20 minutes. Their status page showed a degraded API. Nothing to do on our side.
Thu 23:40 checkout-api 5xx again. Feature flag new_tax_rules was switched on in prod by mistake. Flag off at 23:52.
Fri 11:15 ledger-worker OOM-killed twice after the batch size change; 512Mi limit too low. Bumped to 1Gi, stable since 11:35

Prompt 1 (extract) asks for raw values only, since code calculates durations later:

Extract every incident from the on-call notes.
Return only a JSON array, no other text. One object per incident with keys:
"day", "service", "start" and "end" (clock times exactly as written, or null),
"stated_minutes" (integer if the notes state a duration, else null),
"what_happened" (one short sentence). Copy values; don't calculate anything.

[{"day": "Mon", "service": "checkout-api", "start": "02:10", "end": "02:24", "stated_minutes": null, "what_happened": "Rollback of deployment 4812 completed successfully."},
 {"day": "Tue", "service": "ledger-worker", "start": "14:05", "end": "14:50", "stated_minutes": null, "what_happened": "Queue backed up due to Postgres connection issues. Queue drained by 14:50."},
 {"day": "Wed", "service": "card processor", "start": "09:30", "end": null, "stated_minutes": null, "what_happened": "Calls to the card processor timed out for approximately 20 minutes. Status page indicated a degraded API."},
 ...two more incidents...]
Extract every incident from the on-call notes.
Return only a JSON array, no other text. One object per incident with keys:
"day", "service", "start" and "end" (clock times exactly as written, or null),
"stated_minutes" (integer if the notes state a duration, else null),
"what_happened" (one short sentence). Copy values; don't calculate anything.

[{"day": "Mon", "service": "checkout-api", "start": "02:10", "end": "02:24", "stated_minutes": null, "what_happened": "Rollback of deployment 4812 completed successfully."},
 {"day": "Tue", "service": "ledger-worker", "start": "14:05", "end": "14:50", "stated_minutes": null, "what_happened": "Queue backed up due to Postgres connection issues. Queue drained by 14:50."},
 {"day": "Wed", "service": "card processor", "start": "09:30", "end": null, "stated_minutes": null, "what_happened": "Calls to the card processor timed out for approximately 20 minutes. Status page indicated a degraded API."},
 ...two more incidents...]
Extract every incident from the on-call notes.
Return only a JSON array, no other text. One object per incident with keys:
"day", "service", "start" and "end" (clock times exactly as written, or null),
"stated_minutes" (integer if the notes state a duration, else null),
"what_happened" (one short sentence). Copy values; don't calculate anything.

[{"day": "Mon", "service": "checkout-api", "start": "02:10", "end": "02:24", "stated_minutes": null, "what_happened": "Rollback of deployment 4812 completed successfully."},
 {"day": "Tue", "service": "ledger-worker", "start": "14:05", "end": "14:50", "stated_minutes": null, "what_happened": "Queue backed up due to Postgres connection issues. Queue drained by 14:50."},
 {"day": "Wed", "service": "card processor", "start": "09:30", "end": null, "stated_minutes": null, "what_happened": "Calls to the card processor timed out for approximately 20 minutes. Status page indicated a degraded API."},
 ...two more incidents...]

A model may wrap this array in a markdown code fence, which the parser strips. The illustrative extraction copies the clock times correctly. It missed Wednesday's stated duration, so the code records that one as unknown and computes 14, 45, 12, and 20 minutes for the rest.

Prompt 2 (transform) adds one label per incident from a fixed list:

Add a "cause" key to each incident in the JSON array.
Use exactly one of: "deploy", "config", "capacity", "dependency".
Return only the updated JSON array.
Add a "cause" key to each incident in the JSON array.
Use exactly one of: "deploy", "config", "capacity", "dependency".
Return only the updated JSON array.
Add a "cause" key to each incident in the JSON array.
Use exactly one of: "deploy", "config", "capacity", "dependency".
Return only the updated JSON array.

Its output, condensed to one line per incident (durations from code):

Mon checkout-api    14 min    deploy
Tue ledger-worker   45 min    config
Wed card processor  unknown   capacity
Thu checkout-api    12 min    config
Fri ledger-worker   20 min    capacity
Mon checkout-api    14 min    deploy
Tue ledger-worker   45 min    config
Wed card processor  unknown   capacity
Thu checkout-api    12 min    config
Fri ledger-worker   20 min    capacity
Mon checkout-api    14 min    deploy
Tue ledger-worker   45 min    config
Wed card processor  unknown   capacity
Thu checkout-api    12 min    config
Fri ledger-worker   20 min    capacity

All five labels passed the allowed-values check, but we'd call Tuesday capacity (the connection pool maxed out) and Wednesday a dependency failure. Code then sums minutes per cause: config 57, capacity 20, deploy 14.

Prompt 3 (analyze) gets the labeled incidents plus those totals:

You get incidents and downtime totals per cause.
Write a three-sentence summary for the engineering manager: the biggest
source of downtime, any repeat offender, and one follow-up to schedule

You get incidents and downtime totals per cause.
Write a three-sentence summary for the engineering manager: the biggest
source of downtime, any repeat offender, and one follow-up to schedule

You get incidents and downtime totals per cause.
Write a three-sentence summary for the engineering manager: the biggest
source of downtime, any repeat offender, and one follow-up to schedule

In this illustrative sequence, the first answer claims checkout-api has "a total downtime of 126 minutes (or 2 hours and 6 minutes)," and the summary check rejects it because 126, 2, and 6 aren't in the input. The retry passes the check but still gives ledger-worker "a cumulative downtime of 57 minutes" (the mislabeled config total) and moves the card processor outage to Thursday.

The parser handled the code fence, and a check caught the invented number, but nothing caught the rest: the wrong label from step 2 flowed into the totals and then into the summary.

Example 2: Draft, critique and rewrite a support reply

Claude's prompting docs call this self-correction loop (draft, review against criteria, refine) the most common prompt chaining pattern. The input:

Customer message: I was charged twice for my March invoice (INV-2291). I want the
duplicate refunded and I want to know why it happened. This is the second time this year.

Case notes: Billing confirmed a duplicate charge on INV-2291. Cause: our payment retry fired
during a processor timeout on March 3. Refund issued today; banks usually post it in 5 to 10
business days. The earlier duplicate (January) had the same cause. Engineering shipped a fix
for retries on March 5

Customer message: I was charged twice for my March invoice (INV-2291). I want the
duplicate refunded and I want to know why it happened. This is the second time this year.

Case notes: Billing confirmed a duplicate charge on INV-2291. Cause: our payment retry fired
during a processor timeout on March 3. Refund issued today; banks usually post it in 5 to 10
business days. The earlier duplicate (January) had the same cause. Engineering shipped a fix
for retries on March 5

Customer message: I was charged twice for my March invoice (INV-2291). I want the
duplicate refunded and I want to know why it happened. This is the second time this year.

Case notes: Billing confirmed a duplicate charge on INV-2291. Cause: our payment retry fired
during a processor timeout on March 3. Refund issued today; banks usually post it in 5 to 10
business days. The earlier duplicate (January) had the same cause. Engineering shipped a fix
for retries on March 5

The three prompts:

Draft:    Write a reply to the customer using only facts from the case notes.
          Keep it under 120 words. Return only the reply.
Critique: Check the draft reply against the case and this checklist:
          1. Answers every question the customer asked.
          2. States only facts that appear in the case notes.
          3. Gives the refund timeline.
          4. Under 120 words.
          List each problem on its own line. If there are none, reply exactly: OK
Rewrite:  Rewrite the draft reply so it fixes every problem listed in the
          critique. Use only facts from the case notes. Return only the reply

Draft:    Write a reply to the customer using only facts from the case notes.
          Keep it under 120 words. Return only the reply.
Critique: Check the draft reply against the case and this checklist:
          1. Answers every question the customer asked.
          2. States only facts that appear in the case notes.
          3. Gives the refund timeline.
          4. Under 120 words.
          List each problem on its own line. If there are none, reply exactly: OK
Rewrite:  Rewrite the draft reply so it fixes every problem listed in the
          critique. Use only facts from the case notes. Return only the reply

Draft:    Write a reply to the customer using only facts from the case notes.
          Keep it under 120 words. Return only the reply.
Critique: Check the draft reply against the case and this checklist:
          1. Answers every question the customer asked.
          2. States only facts that appear in the case notes.
          3. Gives the refund timeline.
          4. Under 120 words.
          List each problem on its own line. If there are none, reply exactly: OK
Rewrite:  Rewrite the draft reply so it fixes every problem listed in the
          critique. Use only facts from the case notes. Return only the reply

Consider a draft that gets the cause and refund timeline right, then adds "Please note that similar issues were encountered last month as well." The notes say January. If the critique step answers OK, that error can go straight to the customer.

A stronger critique prompt demands evidence: for each factual sentence in the draft, quote the supporting case-note words or write UNSUPPORTED. That still needs checking: the critic could quote the customer's message instead of evaluating the draft. Even a rewrite that drops the unsupported sentence can leave another gap:

Thank you for bringing this matter to our attention. A duplicate charge on INV-2291 was
identified due to a payment retry during a processor timeout on March 3. As a result, we have
initiated a refund process, which will be reflected in your account within 5 to 10 business
days. Additionally, an engineering team has implemented a fix to prevent future occurrences of
retries, addressing the underlying cause of the duplicate charge. If you require any further
information or assistance, please do not hesitate to reach out

Thank you for bringing this matter to our attention. A duplicate charge on INV-2291 was
identified due to a payment retry during a processor timeout on March 3. As a result, we have
initiated a refund process, which will be reflected in your account within 5 to 10 business
days. Additionally, an engineering team has implemented a fix to prevent future occurrences of
retries, addressing the underlying cause of the duplicate charge. If you require any further
information or assistance, please do not hesitate to reach out

Thank you for bringing this matter to our attention. A duplicate charge on INV-2291 was
identified due to a payment retry during a processor timeout on March 3. As a result, we have
initiated a refund process, which will be reflected in your account within 5 to 10 business
days. Additionally, an engineering team has implemented a fix to prevent future occurrences of
retries, addressing the underlying cause of the duplicate charge. If you require any further
information or assistance, please do not hesitate to reach out

It never acknowledges this is the customer's second duplicate, which they raised directly. A critic needs its own test set, just like the writer.

Prompt chaining vs. chain of thought

Chain-of-thought prompting, introduced by Wei et al. in 2022, gets the model to write "a series of intermediate reasoning steps" before its answer, originally by putting a few worked examples in the prompt. That all happens inside one response, while a chain spreads the work across calls your code connects.

Prompt chaining

Chain of thought

Where the steps live

Separate model calls

One response

Who defines the steps

You, in code

The model, as it writes

Where your code can step in

After every call

Only after the whole response

Cost shape

One call per step

One call with a longer output

Best fit

Pipelines with distinct jobs (extract, label, write)

One reasoning problem (math, logic)

The two combine well, since any step in a chain can use chain of thought. Self-consistency builds on chain of thought differently: it samples several reasoning paths and keeps the most consistent answer. Meta prompting, in Suzgun and Kalai's sense, has one model act as a conductor that splits the task and hands subtasks to "expert" instances of itself. The split happens at runtime there, while a chain fixes it in code.

How to build a prompt chain

  1. Define the goal. Write down what a correct final output looks like, ideally something code can check, such as "three sentences, every number traceable to the notes."

  2. Split the work into subtasks. Give each prompt one job, and keep arithmetic, sorting, and counting in code. For these notes, the four computable durations are 14, 45, 12, and 20 minutes. Copy the start and end times and subtract them in Python rather than asking the model to do the arithmetic.

  3. Plan the handoff. Name the fields each step returns and decide what "unknown" looks like (we used null). If a step keeps misreading its input, add worked examples to its prompt; our guide to few-shot prompting covers how to pick them.

  4. Add checks. Parse, then check required keys, allowed values, plausible ranges and, for generated text, cheap rules like "no numbers that weren't in the input."

  5. Iterate one link at a time. Change one prompt, rerun the same inputs, and compare.

Here is a runner for example 1, with validation before each handoff. It uses the OpenAI Python SDK's Responses API. For a local endpoint, set OPENAI_BASE_URL only if the server implements that API, not just Chat Completions.

Use Python 3.10 or later and install the SDK with python -m pip install openai. Save the runner as chain.py, the example's on-call notes as oncall_notes.txt, and the three prompt texts as triple-quoted strings named EXTRACT, TRANSFORM, and ANALYZE in prompts.py in the same directory.

Set OPENAI_API_KEY to your provider credential and CHAIN_MODEL to a Responses-capable model available to your account, then run python chain.py. A hosted provider may charge for the calls. This example handles same-day clock times in one time zone; cross-midnight incidents need dates as well as times.

import json, os, re
from collections import Counter
from openai import OpenAI
from prompts import EXTRACT, TRANSFORM, ANALYZE  # the prompt text shown above

client = OpenAI()  # reads OPENAI_API_KEY and, if set, OPENAI_BASE_URL
MODEL = os.environ["CHAIN_MODEL"]
CAUSES = {"deploy", "config", "capacity", "dependency"}


def call(instructions, text):
    return client.responses.create(model=MODEL, instructions=instructions, input=text).output_text


def parse_json(raw):
    raw = raw.strip()
    if raw.startswith("`"):  # small models often wrap JSON in a markdown fence
        raw = raw.strip("`").removeprefix("json")
    return json.loads(raw)  # json.JSONDecodeError is a ValueError


def clock_minutes(value):
    if not isinstance(value, str) or not re.fullmatch(r"(?:[01][0-9]|2[0-3]):[0-5][0-9]", value):
        raise ValueError("expected HH:MM in 24-hour time")
    hours, mins = map(int, value.split(":"))
    return hours * 60 + mins


def minutes(start, end):
    # This example assumes same-day incidents in one time zone.
    duration = clock_minutes(end) - clock_minutes(start)
    if duration <= 0:
        raise ValueError("end must follow start on the same day")
    return duration


def check_extract(raw):
    items = parse_json(raw)
    if not isinstance(items, list) or not items:
        raise ValueError("expected a non-empty JSON array")
    for it in items:
        if not isinstance(it, dict):
            raise ValueError("each incident must be an object")
        missing = {"day", "service", "start", "end", "stated_minutes"} - set(it)
        if missing:
            raise ValueError(f"missing keys {sorted(missing)}")
        for key in ("day", "service"):
            if not isinstance(it[key], str) or not it[key].strip():
                raise ValueError(f"expected non-empty {key}")
        for key in ("start", "end"):
            if it[key] is not None:
                clock_minutes(it[key])
        stated = it["stated_minutes"]
        if stated is not None and not (type(stated) is int and stated > 0):
            raise ValueError("stated_minutes must be a positive integer or null")
        # Durations are computed here, never by the model. None means unknown.
        both = it["start"] is not None and it["end"] is not None
        it["minutes"] = minutes(it["start"], it["end"]) if both else it["stated_minutes"]
        m = it["minutes"]
        if m is not None and not (type(m) is int and m > 0):
            raise ValueError(f"bad duration on {it['day']}")
    return items


def check_transform(raw):
    items = check_extract(raw)
    for it in items:
        it["cause"] = str(it.get("cause")).strip().lower()  # tolerate casing drift
        if it["cause"] not in CAUSES:
            raise ValueError(f"unknown cause {it['cause']!r}")
    return items


def check_summary(raw, facts):
    invented = set(re.findall(r"\d+", raw)) - set(re.findall(r"\d+", facts))
    if invented:
        raise ValueError(f"numbers not in the input: {sorted(invented)}")
    return raw.strip()


def run_step(instructions, text, check, retries=1):
    """Call the model, check the output, retry once with the error, then stop."""
    prompt = text
    for _ in range(retries + 1):
        raw = call(instructions, prompt)
        try:
            return check(raw)
        except ValueError as err:
            error = err
            prompt = f"{text}\n\nYour last answer was rejected: {err}. Fix it."
    raise RuntimeError(f"step failed after {retries + 1} tries: {error}")


notes = open("oncall_notes.txt").read()
incidents = run_step(EXTRACT, notes, check_extract)
labeled = run_step(TRANSFORM, json.dumps(incidents), check_transform)

totals = Counter()
for it in labeled:
    totals[it["cause"]] += it["minutes"] or 0  # arithmetic stays in code

facts = json.dumps({"incidents": labeled, "minutes_by_cause": totals})
print(run_step(ANALYZE, facts, lambda raw: check_summary(raw, facts)))
import json, os, re
from collections import Counter
from openai import OpenAI
from prompts import EXTRACT, TRANSFORM, ANALYZE  # the prompt text shown above

client = OpenAI()  # reads OPENAI_API_KEY and, if set, OPENAI_BASE_URL
MODEL = os.environ["CHAIN_MODEL"]
CAUSES = {"deploy", "config", "capacity", "dependency"}


def call(instructions, text):
    return client.responses.create(model=MODEL, instructions=instructions, input=text).output_text


def parse_json(raw):
    raw = raw.strip()
    if raw.startswith("`"):  # small models often wrap JSON in a markdown fence
        raw = raw.strip("`").removeprefix("json")
    return json.loads(raw)  # json.JSONDecodeError is a ValueError


def clock_minutes(value):
    if not isinstance(value, str) or not re.fullmatch(r"(?:[01][0-9]|2[0-3]):[0-5][0-9]", value):
        raise ValueError("expected HH:MM in 24-hour time")
    hours, mins = map(int, value.split(":"))
    return hours * 60 + mins


def minutes(start, end):
    # This example assumes same-day incidents in one time zone.
    duration = clock_minutes(end) - clock_minutes(start)
    if duration <= 0:
        raise ValueError("end must follow start on the same day")
    return duration


def check_extract(raw):
    items = parse_json(raw)
    if not isinstance(items, list) or not items:
        raise ValueError("expected a non-empty JSON array")
    for it in items:
        if not isinstance(it, dict):
            raise ValueError("each incident must be an object")
        missing = {"day", "service", "start", "end", "stated_minutes"} - set(it)
        if missing:
            raise ValueError(f"missing keys {sorted(missing)}")
        for key in ("day", "service"):
            if not isinstance(it[key], str) or not it[key].strip():
                raise ValueError(f"expected non-empty {key}")
        for key in ("start", "end"):
            if it[key] is not None:
                clock_minutes(it[key])
        stated = it["stated_minutes"]
        if stated is not None and not (type(stated) is int and stated > 0):
            raise ValueError("stated_minutes must be a positive integer or null")
        # Durations are computed here, never by the model. None means unknown.
        both = it["start"] is not None and it["end"] is not None
        it["minutes"] = minutes(it["start"], it["end"]) if both else it["stated_minutes"]
        m = it["minutes"]
        if m is not None and not (type(m) is int and m > 0):
            raise ValueError(f"bad duration on {it['day']}")
    return items


def check_transform(raw):
    items = check_extract(raw)
    for it in items:
        it["cause"] = str(it.get("cause")).strip().lower()  # tolerate casing drift
        if it["cause"] not in CAUSES:
            raise ValueError(f"unknown cause {it['cause']!r}")
    return items


def check_summary(raw, facts):
    invented = set(re.findall(r"\d+", raw)) - set(re.findall(r"\d+", facts))
    if invented:
        raise ValueError(f"numbers not in the input: {sorted(invented)}")
    return raw.strip()


def run_step(instructions, text, check, retries=1):
    """Call the model, check the output, retry once with the error, then stop."""
    prompt = text
    for _ in range(retries + 1):
        raw = call(instructions, prompt)
        try:
            return check(raw)
        except ValueError as err:
            error = err
            prompt = f"{text}\n\nYour last answer was rejected: {err}. Fix it."
    raise RuntimeError(f"step failed after {retries + 1} tries: {error}")


notes = open("oncall_notes.txt").read()
incidents = run_step(EXTRACT, notes, check_extract)
labeled = run_step(TRANSFORM, json.dumps(incidents), check_transform)

totals = Counter()
for it in labeled:
    totals[it["cause"]] += it["minutes"] or 0  # arithmetic stays in code

facts = json.dumps({"incidents": labeled, "minutes_by_cause": totals})
print(run_step(ANALYZE, facts, lambda raw: check_summary(raw, facts)))
import json, os, re
from collections import Counter
from openai import OpenAI
from prompts import EXTRACT, TRANSFORM, ANALYZE  # the prompt text shown above

client = OpenAI()  # reads OPENAI_API_KEY and, if set, OPENAI_BASE_URL
MODEL = os.environ["CHAIN_MODEL"]
CAUSES = {"deploy", "config", "capacity", "dependency"}


def call(instructions, text):
    return client.responses.create(model=MODEL, instructions=instructions, input=text).output_text


def parse_json(raw):
    raw = raw.strip()
    if raw.startswith("`"):  # small models often wrap JSON in a markdown fence
        raw = raw.strip("`").removeprefix("json")
    return json.loads(raw)  # json.JSONDecodeError is a ValueError


def clock_minutes(value):
    if not isinstance(value, str) or not re.fullmatch(r"(?:[01][0-9]|2[0-3]):[0-5][0-9]", value):
        raise ValueError("expected HH:MM in 24-hour time")
    hours, mins = map(int, value.split(":"))
    return hours * 60 + mins


def minutes(start, end):
    # This example assumes same-day incidents in one time zone.
    duration = clock_minutes(end) - clock_minutes(start)
    if duration <= 0:
        raise ValueError("end must follow start on the same day")
    return duration


def check_extract(raw):
    items = parse_json(raw)
    if not isinstance(items, list) or not items:
        raise ValueError("expected a non-empty JSON array")
    for it in items:
        if not isinstance(it, dict):
            raise ValueError("each incident must be an object")
        missing = {"day", "service", "start", "end", "stated_minutes"} - set(it)
        if missing:
            raise ValueError(f"missing keys {sorted(missing)}")
        for key in ("day", "service"):
            if not isinstance(it[key], str) or not it[key].strip():
                raise ValueError(f"expected non-empty {key}")
        for key in ("start", "end"):
            if it[key] is not None:
                clock_minutes(it[key])
        stated = it["stated_minutes"]
        if stated is not None and not (type(stated) is int and stated > 0):
            raise ValueError("stated_minutes must be a positive integer or null")
        # Durations are computed here, never by the model. None means unknown.
        both = it["start"] is not None and it["end"] is not None
        it["minutes"] = minutes(it["start"], it["end"]) if both else it["stated_minutes"]
        m = it["minutes"]
        if m is not None and not (type(m) is int and m > 0):
            raise ValueError(f"bad duration on {it['day']}")
    return items


def check_transform(raw):
    items = check_extract(raw)
    for it in items:
        it["cause"] = str(it.get("cause")).strip().lower()  # tolerate casing drift
        if it["cause"] not in CAUSES:
            raise ValueError(f"unknown cause {it['cause']!r}")
    return items


def check_summary(raw, facts):
    invented = set(re.findall(r"\d+", raw)) - set(re.findall(r"\d+", facts))
    if invented:
        raise ValueError(f"numbers not in the input: {sorted(invented)}")
    return raw.strip()


def run_step(instructions, text, check, retries=1):
    """Call the model, check the output, retry once with the error, then stop."""
    prompt = text
    for _ in range(retries + 1):
        raw = call(instructions, prompt)
        try:
            return check(raw)
        except ValueError as err:
            error = err
            prompt = f"{text}\n\nYour last answer was rejected: {err}. Fix it."
    raise RuntimeError(f"step failed after {retries + 1} tries: {error}")


notes = open("oncall_notes.txt").read()
incidents = run_step(EXTRACT, notes, check_extract)
labeled = run_step(TRANSFORM, json.dumps(incidents), check_transform)

totals = Counter()
for it in labeled:
    totals[it["cause"]] += it["minutes"] or 0  # arithmetic stays in code

facts = json.dumps({"incidents": labeled, "minutes_by_cause": totals})
print(run_step(ANALYZE, facts, lambda raw: check_summary(raw, facts)))

If your provider supports structured outputs, pass the schema too and keep the checks. To skip the hand-rolled plumbing, LangChain and LangGraph both support this pattern, and LangGraph's docs list prompt chaining as a workflow. Our LangGraph tutorial covers the building blocks: nodes do the work, edges decide what runs next, and shared state carries data between steps.

Drawbacks and failure modes

  • Latency. Calls run in sequence, so latencies add up. Three steps plus one retry require four model calls; the elapsed time depends on the model and deployment.

  • Cost. Each link pays for its own input tokens, handoffs often repeat context, and retries add calls you didn't budget for.

  • Error propagation. A wrong output that passes its check becomes the next step's ground truth. Example 1's Tuesday label turned into a wrong total in code and then a wrong claim in the summary.

  • Brittle formats. Parsers break on code fences, extra prose, renamed keys or changed casing, and a model or prompt change can introduce any of them.

  • Retries that make things worse. A stricter extraction check could reject an incident with no usable duration, only for the retry to "fix" it by returning that incident and dropping the other four. Check that a retry didn't shrink the output.

  • Critics that rubber-stamp. Example 2's critique approved a draft with an unsupported claim in it.

When not to chain

Short tasks. Before you create a chain, check whether one prompt already scores well on your evals. If it does, prompt chaining only adds calls, code and failure points. Anthropic's advice is to add complexity "only when it demonstrably improves outcomes."

Tasks a newer model handles in one call. Claude's prompting docs say that with adaptive thinking and subagent orchestration, Claude handles most multistep reasoning internally. They keep explicit chains for inspecting intermediate outputs or enforcing a pipeline. Run the single call and the chain on the same test set before committing.

Fully dynamic flows. If the next step depends on what the model finds, such as which tool to call or how many searches to run, you want an agent. Anthropic separates workflows, which follow "predefined code paths," from agents, where models "dynamically direct their own processes and tool usage." The loop, tools, and stopping rules around that model are its agent harness.

Routing sits in between: in Anthropic's terms, it classifies an input and directs it to a specialized follow-up task, which can be a chain of its own.

How to test and tune a chain

A chain needs a score per step and one for the end result. In example 1, every step passed its checks, and the summary was still wrong.

  • Score each step against labels. Build a small labeled set per link, such as notes with known incidents for extraction and incidents with agreed causes for the transform step. Measure field-level accuracy, not just validity: our transform step returned 5 valid labels, and we'd accept 3.

  • Score the end result with a rubric or deterministic checks, because decent steps can still add up to a bad answer.

  • Track check failures and retries per step, plus cost per successful run, to find the fragile link.

  • Compare variants on the same inputs. Freeze one step's inputs (saved upstream outputs or hand-labeled ones) and run its prompt or model variants on that set, then retest the downstream links, since they now see different inputs. The prompt optimization loop applies to each link.

Our open-source RapidFire engine is built for this comparison work. Its eval pipeline produces one generation per example: your preprocess_fn builds the prompts, an optional postprocess_fn parses the output (a natural home for the checks above), and your compute_metrics_fn scores each batch. So you test a chain one link at a time: make a link's saved inputs the eval set and put its prompt or model variants into one grid.

The engine shards the eval set and runs every config on one shard at a time, so variants post early scores side by side. Declare each metric's type and value range and you also get running estimates with confidence intervals. With MLflow tracing on (the local default), traces show the exact prompt each config sent and what came back.

In the open-source engine, you stop weak configs or clone and modify strong ones yourself with IC Ops. Converge automates those calls: it stops weak versions, clones strong ones, logs the reason for each decision, and stays within the time and spend limits you set. To try the engine, start with the RapidFire RAG walkthrough, which has a Colab notebook and a local install path.

FAQ

What is the main difference between prompt chaining and chain of thought prompting?

Prompt chaining splits a task across several model calls, and your code passes each output to the next prompt, so you can check every intermediate result. Chain-of-thought prompting keeps the work in one call and has the model write out its reasoning first. Any step of a chain can use chain of thought.

What are the 5 P's of prompting?

There's no standard list. It's a mnemonic from prompt engineering courses, and the courses don't agree: one version is prime, persona, privacy, product, and polish, while another is persona, precision, parameters, process, and polish. Use whichever you learned as a checklist when you create a prompt for each step, then judge it by measured results.

Can you provide an example of a chain of prompts?

For weekly incident notes, prompt 1 extracts each incident as JSON with its service and start and end times. Prompt 2 labels each incident's cause from a fixed list, and prompt 3 writes a three-sentence summary from the labeled incidents plus totals calculated in code. Code checks the JSON and labels between steps. Example 1 above shows the prompts and illustrative outputs.

What is chaining in AI?

Chaining means connecting model calls, tools, or code so each step's output feeds the next. Prompt chaining is the simplest form, a fixed sequence of prompts. Frameworks such as LangChain and LangGraph provide building blocks for it, and agents go further by letting the model pick the next step.