rapidfire.

Multi-Agent Evaluation: Metrics, Methods, and Tools

Written By:

Matt Tanner

Published on

Your support workflow has a supervisor and three workers: an orders agent, a billing agent, and a refunds agent. A customer asks for a refund on a damaged item. In an example test run, the basic checks pass: the supervisor routes the request to the expected workers, the orders agent looks up order A-1042, and the refunds agent calls an allowed refund tool.

Yet the recorded refund is $98 instead of the expected $89. The trace also shows that the handoff omitted the order ID and amount. That is a separate defect to investigate, not enough evidence to explain the wrong refund.

That gap between "the basic checks passed" and "the task worked" is what multi-agent evaluation is for. This guide builds on the single-agent basics in our AI agent evaluation guide. It covers what changes when several agents share a task: what to measure, how to find the agent at fault, and how to compare configurations before you ship.

What is multi-agent evaluation?

Multi-agent evaluation measures whether a system of cooperating AI agents completes its task, and whether each agent and each handoff did its part. It scores three levels together: the end-to-end outcome, each agent's behavior in its role, and the agent interactions in between, such as handoffs, shared context, and duplicated work.

Scoring agents one at a time misses the middle level. Google Cloud's October 2025 Agent Factory recap uses a support system where Agent A gathers refund details and hands them to Agent B. Scored alone, Agent A's task completion "might be zero because it didn't actually issue the refund," even though handing off was its whole job.

And if Agent A passes the wrong details, the system fails even when Agent B's logic is perfect. The recap's conclusion is the right starting rule: in multi-agent systems, "what really matters is the end-to-end evaluation."

An end-to-end score tells you a run failed, though, and says nothing about which agent or handoff to fix. That's the job of the other two levels. Standards groups are working on the same problem: IEEE has an Industry Connections activity on evaluation techniques for multi-agent systems aimed at "transparent, auditable, and trusted multi-agent systems."

Why evaluating several agents is harder than evaluating one

A single agent can fail in plenty of ways, but at least every failure lives in one prompt, one model, and one tool set. Multi-agent systems add failure modes that only appear when agents work together:

  • Errors compound across handoffs. If the supervisor, orders agent, and refunds agent each pass 95% of their own checks, multiplying gives 0.95 ร— 0.95 ร— 0.95 โ‰ˆ 86% for the chain. That calculation assumes independent failures and that every stage must succeed. Real handoffs can make failures dependent, and narrow per-agent checks can miss incorrect arguments, so score complete runs as well.

  • Non-determinism stacks up. Anthropic's guide to agent evals warns that "a task that passed on one eval run might fail on the next." With three agents sampling on every run, repeat each task enough times to tell a regression from noise.

  • Behavior emerges from agent interactions. Describing its Research system, Anthropic notes that "small changes to the lead agent can unpredictably change how subagents behave." A one-line edit to the supervisor prompt is also a change to every worker's inputs.

  • Credit assignment is hard, even for automated tools. On the Who&When benchmark of failure logs from 127 LLM multi-agent systems, the best automated method identified the responsible agent 53.5% of the time and the failing step only 14.2% of the time.

  • Cost scales with agent count. In Anthropic's multi-agent research write-up, multi-agent systems "use about 15ร— more tokens than chats."

The MAST failure taxonomy sorts 14 failure modes into three categories: system design issues, inter-agent misalignment, and task verification. Its authors also released 1,600+ annotated traces from seven multi-agent frameworks. Inter-agent misalignment can't show up at all when you test agents one at a time.

Common multi-agent architectures and what each breaks

Where failures hide depends on how control moves between agents, so each of these multi-agent architectures points your evaluation at a different weak spot.

Architecture

How control moves

What tends to break

Check first

Supervisor/worker

A supervisor calls workers and gets results back

Wrong routing, thin handoff payloads, the supervisor summarizing away a worker's error

Routing accuracy, required handoff fields

Hierarchical

Supervisors delegate to sub-supervisors, which manage workers

Context lost at each layer, delegation loops, token use that grows with depth

Routing per layer, turns and tokens per task

Peer/debate

Agents exchange messages and converge on an answer

Agreement without verification, one agent dominating, rounds that never settle

Final answer against a reference, rounds to converge

Pipeline

Fixed order, each stage feeds the next

An early error flows downstream unchecked

Output checks at each stage, end-to-end outcome

Frameworks expose these patterns differently. LangChain supports subagents called as tools, handoffs, routers, and custom LangGraph workflows. CrewAI's hierarchical process gives a manager agent responsibility for planning, delegation, and validation. Google ADK organizes agents under a root agent, while the AgentCore example uses a supervisor that calls specialized agents as tools. AutoGen, a framework for multi-agent applications, is now archived, and Microsoft points new users to Microsoft Agent Framework.

Supervisor setups also raise a scoring question: which agent owns the final response? When a worker runs as a tool, control returns to the supervisor, which decides what the user sees. When a worker takes over the conversation, its output is the answer.

What to measure

Every metric from single-agent evaluation still applies to each agent: tool selection, argument correctness, trajectory and pass rates across trials, all covered in our agent evaluation guide. Multi-agent systems add an interaction level and change how you read cost. Arize's Phoenix docs on multi-agent evaluation use a similar split, with agent, interaction, system, and user levels.

Level

Metric

What it catches

How to score

System

Task completion (end state)

A wrong refund after clean-looking steps

Code check on the final state

System

Cost, tokens, and latency per task

A topology that works but costs too much

Sum across all agents, including failed runs

Agent

Tool use per role

Wrong tool or bad arguments inside one agent

Compare calls to expected tools and args

Agent

Role adherence

A worker calling tools outside its role, or a supervisor doing a worker's job

Per-agent tool allowlist

Interaction

Routing accuracy

The supervisor sent the task to the wrong worker

Labeled routing cases

Interaction

Handoff completeness

Required fields missing from a transfer

Schema check on transfer arguments

Interaction

Agent communication quality

A handoff message the receiver can't act on

LLM judge with a rubric

Interaction

Redundant work

Two agents repeating the same lookup

Count duplicate tool calls per run

Report each metric per agent and per task type, so a strong orders agent doesn't hide a weak refunds agent in the average. For definitions of the general-purpose measures (latency, token use, judge-based scores), see our guide to LLM evaluation metrics.

[Image placeholder: Diagram of one supervisor agent delegating to three worker agents, left to right: "Supervisor" at the top center with three arrows down to "Orders agent", "Refunds agent" and "Billing agent". Each arrow is labeled "handoff". A horizontal trace timeline runs beneath the agents showing spans in order: "transfer_to_orders", "lookup_order", "transfer_to_refunds", "issue_refund". Three colored brackets mark the evaluation levels: a bracket around the whole timeline labeled "System level: task completion, cost, latency"; a bracket around each agent's spans labeled "Agent level: tool use, role adherence"; and markers on the two handoff arrows labeled "Interaction level: routing, handoff completeness, redundant work". Alt text: Multi-agent evaluation diagram showing a supervisor agent, three worker agents and system-level, agent-level and interaction-level checks on the trace]

How to evaluate: methods

The usual single-agent grader types carry over, aimed at new targets.

Code-based checks handle most interaction-level metrics: the final state, per-agent tool allowlists, required handoff fields, and duplicate calls. They're cheap and deterministic, so run them on every trial.

LLM-as-a-judge covers what code can't, like whether a handoff message gave the receiver enough context or whether the final reply was helpful. Anthropic's research team found that "a single LLM call with a single prompt outputting scores from 0.0-1.0 and a pass-fail grade" was the most consistent judge setup for them.

Validate any judge against human labels before you trust it, as covered in our guide to LLM-as-a-judge. The judge can be a multi-agent system too: ChatEval uses a team of LLM referees that debate the quality of a response.

Human evaluation catches what nobody wrote a check for. Anthropic's human testers noticed early research agents "consistently chose SEO-optimized content farms" over more authoritative sources. Review a sample of traces regularly and turn what reviewers find into new checks and rubric items.

Trace-level vs span-level evals. A trace-level eval grades the whole run: outcome, total cost, number of turns, and for multi-turn conversations, the full exchange. A span-level eval grades one agent invocation or one tool call. OpenTelemetry's GenAI semantic conventions, still at Development status, give you shared names for both. They use invoke_agent spans when an LLM drives control flow and invoke_workflow when the application defines it.

In the execute-tool conventions, tool calls that transfer work to another agent carry gen_ai.transfer.target.name and gen_ai.transfer.mode, which is return_to_caller or pass_control. Capturing tool call arguments is opt-in, so turn it on in your eval environment if you want to check handoff payloads.

Here's a scorer for an example trace like the one in the intro, with tool calls represented as dicts. The trace and final state are fixtures: this code grades them rather than running agents or issuing refunds. It uses only the Python standard library and checks the outcome, allowed tools, and handoffs.

from collections import Counter

# One run of a refund workflow, flattened from execute_tool spans.
trace = [
    {"agent": "supervisor", "tool": "transfer_to_orders",
     "transfer_target": "orders", "args": {"order_id": "A-1042"}},
    {"agent": "orders", "tool": "lookup_order", "args": {"order_id": "A-1042"}},
    {"agent": "supervisor", "tool": "transfer_to_refunds",
     "transfer_target": "refunds", "args": {"reason": "damaged item"}},
    {"agent": "refunds", "tool": "lookup_order", "args": {"order_id": "A-1042"}},
    {"agent": "refunds", "tool": "issue_refund",
     "args": {"order_id": "A-1042", "amount": 98.00}},
]
final_state = {"refunds": [{"order_id": "A-1042", "amount": 98.00}]}

ALLOWED_TOOLS = {"supervisor": {"transfer_to_orders", "transfer_to_refunds"},
                 "orders": {"lookup_order"},
                 "refunds": {"lookup_order", "issue_refund"}}
HANDOFF_FIELDS = {"refunds": {"order_id", "amount", "reason"}}

def score(trace, final_state, expected_amount, expected_order_id):
    refunds = final_state["refunds"]
    missing = {}
    for s in trace:
        if "transfer_target" in s:
            target = s["transfer_target"]
            missing.setdefault(target, set()).update(HANDOFF_FIELDS.get(target, set()) - s["args"].keys())
    calls = Counter((s["tool"], tuple(sorted(s["args"].items()))) for s in trace)
    return {
        "system.task_completed": (len(refunds) == 1 and refunds[0]["order_id"] == expected_order_id
                                  and refunds[0]["amount"] == expected_amount),
        "agent.role_adherence": all(s["tool"] in ALLOWED_TOOLS[s["agent"]] for s in trace),
        "interaction.handoff_missing": {k: sorted(v) for k, v in missing.items() if v},
        "interaction.repeated_calls": sum(n - 1 for n in calls.values()),
    }

print(score(trace, final_state, expected_amount=89.00, expected_order_id="A-1042"))
from collections import Counter

# One run of a refund workflow, flattened from execute_tool spans.
trace = [
    {"agent": "supervisor", "tool": "transfer_to_orders",
     "transfer_target": "orders", "args": {"order_id": "A-1042"}},
    {"agent": "orders", "tool": "lookup_order", "args": {"order_id": "A-1042"}},
    {"agent": "supervisor", "tool": "transfer_to_refunds",
     "transfer_target": "refunds", "args": {"reason": "damaged item"}},
    {"agent": "refunds", "tool": "lookup_order", "args": {"order_id": "A-1042"}},
    {"agent": "refunds", "tool": "issue_refund",
     "args": {"order_id": "A-1042", "amount": 98.00}},
]
final_state = {"refunds": [{"order_id": "A-1042", "amount": 98.00}]}

ALLOWED_TOOLS = {"supervisor": {"transfer_to_orders", "transfer_to_refunds"},
                 "orders": {"lookup_order"},
                 "refunds": {"lookup_order", "issue_refund"}}
HANDOFF_FIELDS = {"refunds": {"order_id", "amount", "reason"}}

def score(trace, final_state, expected_amount, expected_order_id):
    refunds = final_state["refunds"]
    missing = {}
    for s in trace:
        if "transfer_target" in s:
            target = s["transfer_target"]
            missing.setdefault(target, set()).update(HANDOFF_FIELDS.get(target, set()) - s["args"].keys())
    calls = Counter((s["tool"], tuple(sorted(s["args"].items()))) for s in trace)
    return {
        "system.task_completed": (len(refunds) == 1 and refunds[0]["order_id"] == expected_order_id
                                  and refunds[0]["amount"] == expected_amount),
        "agent.role_adherence": all(s["tool"] in ALLOWED_TOOLS[s["agent"]] for s in trace),
        "interaction.handoff_missing": {k: sorted(v) for k, v in missing.items() if v},
        "interaction.repeated_calls": sum(n - 1 for n in calls.values()),
    }

print(score(trace, final_state, expected_amount=89.00, expected_order_id="A-1042"))
from collections import Counter

# One run of a refund workflow, flattened from execute_tool spans.
trace = [
    {"agent": "supervisor", "tool": "transfer_to_orders",
     "transfer_target": "orders", "args": {"order_id": "A-1042"}},
    {"agent": "orders", "tool": "lookup_order", "args": {"order_id": "A-1042"}},
    {"agent": "supervisor", "tool": "transfer_to_refunds",
     "transfer_target": "refunds", "args": {"reason": "damaged item"}},
    {"agent": "refunds", "tool": "lookup_order", "args": {"order_id": "A-1042"}},
    {"agent": "refunds", "tool": "issue_refund",
     "args": {"order_id": "A-1042", "amount": 98.00}},
]
final_state = {"refunds": [{"order_id": "A-1042", "amount": 98.00}]}

ALLOWED_TOOLS = {"supervisor": {"transfer_to_orders", "transfer_to_refunds"},
                 "orders": {"lookup_order"},
                 "refunds": {"lookup_order", "issue_refund"}}
HANDOFF_FIELDS = {"refunds": {"order_id", "amount", "reason"}}

def score(trace, final_state, expected_amount, expected_order_id):
    refunds = final_state["refunds"]
    missing = {}
    for s in trace:
        if "transfer_target" in s:
            target = s["transfer_target"]
            missing.setdefault(target, set()).update(HANDOFF_FIELDS.get(target, set()) - s["args"].keys())
    calls = Counter((s["tool"], tuple(sorted(s["args"].items()))) for s in trace)
    return {
        "system.task_completed": (len(refunds) == 1 and refunds[0]["order_id"] == expected_order_id
                                  and refunds[0]["amount"] == expected_amount),
        "agent.role_adherence": all(s["tool"] in ALLOWED_TOOLS[s["agent"]] for s in trace),
        "interaction.handoff_missing": {k: sorted(v) for k, v in missing.items() if v},
        "interaction.repeated_calls": sum(n - 1 for n in calls.values()),
    }

print(score(trace, final_state, expected_amount=89.00, expected_order_id="A-1042"))

Running it prints:

{'system.task_completed': False, 'agent.role_adherence': True, 'interaction.handoff_missing': {'refunds': ['amount', 'order_id']}, 'interaction.repeated_calls': 1}
{'system.task_completed': False, 'agent.role_adherence': True, 'interaction.handoff_missing': {'refunds': ['amount', 'order_id']}, 'interaction.repeated_calls': 1}
{'system.task_completed': False, 'agent.role_adherence': True, 'interaction.handoff_missing': {'refunds': ['amount', 'order_id']}, 'interaction.repeated_calls': 1}

Every recorded call used an allowed tool, but the refund amount was wrong. The scorer also finds missing handoff fields and a repeated lookup. It doesn't validate the refund tool's arguments against the order record or explain where $98 came from. The fixture omits lookup results and the agents' full inputs, so neither the repeated lookup nor the missing handoff proves which component caused the wrong refund.

Attributing failures to the right agent

When evaluating multi-agent runs, a scorer narrows the search, but it can't prove which agent caused the failure. Three techniques help you test possible causes.

Read the trace backward from the bad state. Start at the $98 refund, find the span that wrote it, then the input that agent received, then the handoff that produced that input. MAST's categories help you label what you find: the missing amount is inter-agent misalignment, and the fact that nobody compared the refund to the order total is a task verification gap.

Run per-agent ablations. Replace one agent at a time with an oracle that returns the labeled correct output, keep everything else fixed, and rerun the task set. If replacing the supervisor's handoffs with complete, correct ones improves completion, that supports a handoff-related explanation. It does not establish that the refunds agent handled missing information correctly; inspect and test that behavior separately.

Replay with a corrected handoff. For a single failing run:

  1. Save the exact input the refunds agent received, and write a corrected copy that includes order_id and amount.

  2. Reset the order fixture so no refund exists yet.

  3. Replay only the refunds agent with each input, several times each, with the same prompt, model and settings.

  4. Score every replay with the same checks.

  5. If corrected inputs consistently produce $89 refunds while the original inputs fail, that supports the handoff as a contributing cause. Still check why the refunds agent proceeded without the required information. If both inputs fail, inspect the agent's behavior and tool results rather than blaming the handoff alone.

A step-by-step evaluation loop

Evaluating multi-agent systems works best as a loop you rerun on every change.

  1. Build tasks. Anthropic suggests that "20-50 simple tasks drawn from real failures is a great start." Label each task with the expected end state and, where it applies, the correct routing. Add cases that stress handoffs: missing data, conflicting details from the user, and a worker that returns an error.

  2. Instrument traces. Record every agent invocation, transfer, and tool call with arguments and token counts. The agent harness that runs your agents is where you reset fixtures between trials and attach the trace to each result.

  3. Grade. Run code checks on every trial, judges where code can't decide, and human review on a sample. Repeat each task several times.

  4. Compare configurations. Every agent has its own prompt, model, and retrieval settings, and the topology itself is a knob. Change one role at a time when you're diagnosing, but compare complete configurations when you're choosing.

  5. Gate changes. Confirm the winning configuration on held-out tasks, add every new failure to the task set, and block a release when end-to-end task completion drops or cost per completed task rises past your threshold.

Step 4 is the step we built RapidFire for. Our execution engine shards your eval data and runs many configurations side by side on the same hardware, one shard at a time. For metrics you mark as distributive or algebraic, you see running estimates with confidence intervals before any run finishes.

In the open-source engine, you stop weak runs, resume them, or clone-modify promising ones from the dashboard or the notebook panel. Converge, our automation layer, makes those calls for you within the time and spend limits you set and logs the reason for every stop and clone.

Each eval run builds a prompt for each example, calls the generator, and scores the output with your own metric functions, so the natural multi-agent use is comparing one role at a time.

Take the supervisor's routing prompt and model: build variants as a list or with grid, random, or Optuna search, run them on API models or self-hosted vLLM, and score routing accuracy on your labeled cases. For API models, the dashboard also reports average query latency, total tokens, and projected token spend for each configuration.

Tools and frameworks

Tracing and evaluation platforms tell you what happened inside a run, while experiment tools help you decide what to change, so it's worth pairing one of each. For a broader comparison, see our roundup of LLM evaluation tools.

Tool

What it offers for multi-agent work

Notes

Arize Phoenix

Tracing built on OpenTelemetry and OpenInference, evals, datasets, and experiments

Elastic License 2.0

LangSmith

Experiments on datasets; online evaluators can score whole multi-turn threads

From LangChain

Langfuse

Tracing, LLM-as-a-judge, annotation queues, datasets and experiments

Core is MIT-licensed

Braintrust

Traces, datasets and experiments for agents

Positions itself as an observability platform for agents

Galileo

Observability, evaluation and guardrails

Acquired by Cisco in May 2026; joining Splunk Observability Cloud

Google ADK evaluation

Tool trajectory and response criteria; eval cases can include sub-agent responses

Runs from the web UI, pytest, or the adk eval CLI

AWS AgentCore Evaluations

Built-in evaluators at session, trace, and tool levels, online or on demand

Strands and LangGraph via OpenTelemetry or OpenInference

DeepEval

Component-level evals attached to spans; each sub-agent invocation gets its own span

Apache-2.0; component-level evals are single-turn

RapidFire

Many configurations side by side, with running estimates and stop or clone mid-run

Open-source single-machine engine; Converge automates the loop

Before you commit, check what each tool records at a handoff. If it can't show the input each agent actually received, the attribution steps above won't work.

Mistakes to avoid

  • Scoring agents only in isolation. Google's Agent A example shows how a perfect handoff can score zero and a broken one can sink a downstream agent that did everything right.

  • Trusting the final message. "Your refund is on its way" proves nothing. Check the refund record.

  • Skipping the single-agent baseline. LangChain's own multi-agent docs say "not every complex task requires this approach." If one agent with the same tools and budget does as well, the extra agents are only adding cost.

  • Changing everything at once. A new supervisor prompt, a new worker model, and an extra worker in one change make attribution impossible.

  • Counting only successful runs in cost. Failed runs and retries burn tokens too.

Multi-agent evaluation comes down to scoring the outcome, each agent and each handoff on the same runs, then comparing configurations instead of guessing. To try configuration comparison yourself, start with the RapidFire getting-started walkthrough, which runs on Colab or a local install. If you need to compare multi-step agent workflows, talk to our team.

FAQ

How to evaluate a multi-agent system?

Build 20 to 50 tasks from real failures, each with an expected end state. Run each task several times, record a trace for every run, and score three levels: the system outcome, each agent's tool use and role adherence, and the handoffs between agents. Lead with code checks and use LLM judges and human review for the rest.

What are the top 5 LLM evaluation tools?

There's no universal top five. For multi-agent work, a sensible shortlist combines a tracing and evaluation platform such as Arize Phoenix, LangSmith, Langfuse, or Braintrust, a code-first library such as DeepEval, and an experiment tool for comparing configurations. Choose based on your framework and the evaluation level you need to cover.

Can you give me an example of a multi-agent system?

Anthropic's Research feature uses an orchestrator-worker pattern: a lead agent coordinates the process and delegates to specialized subagents that work in parallel. The support system in this guide, with a supervisor routing requests to orders, billing, and refunds agents, is another common pattern.

Is ChatGPT a multi-agent system?

Not as OpenAI describes it. OpenAI calls ChatGPT agent a "unified agentic system" that works on its own virtual computer, using a visual browser, a text browser, and a terminal, and says it brought those capabilities together "within one model." That's one agent with many tools. A multi-agent system splits the work across separate agents with their own roles, which you'd build yourself with a framework like the ones above.