LLM Latency Optimization: Diagnose the Bottleneck Before Tuning

A fast first word and a fast finished answer are different targets. A chatbot can start responding promptly, then take too long to finish. A document extractor can generate quickly while the application spends most of its time retrieving evidence or validating the result.

LLM latency optimization starts by choosing which wait matters, measuring it, and changing the part responsible. That gives you a more useful experiment than switching models and hoping the whole application gets faster. The goal is a completed, acceptable answer within your time budget, including when other requests are competing for the same service.

Define the latency your users actually experience

Start with the consumer. A person reading a streamed answer cares when useful text appears and whether it keeps arriving smoothly. An agent waiting for a validated tool argument can't act on the first few characters. Measure both when your application serves both.

If your dashboard has one column called “latency,” you can't tell which part of the wait changed. Name each measurement by where its clock starts and stops:

For a client-observed request, TTFT can include network travel, queueing, and prompt processing. It is not a measurement of server prefill alone. If the server exposes its own timings, retain them as separate fields.

A streaming callback also isn't necessarily a token clock. APIs can emit lifecycle events, empty deltas, or content chunks. If you measure the first non-empty answer-text event, call it time to first answer text, not exact token-level TTFT. Don't count a role marker or tool event as visible answer text.

ITL and TPOT conventions differ between tools. For a completed request with more than one output token and compatible first/final-token boundaries, calculate average post-first-token time as:

(E2E − TTFT) / (output_tokens − 1)

(E2E − TTFT) / (output_tokens − 1)

(E2E − TTFT) / (output_tokens − 1)

For example, a request finishes in 4.2 seconds, returns its first token at 0.6 seconds, and produces 301 tokens. The remaining 300 token intervals average (4.2 − 0.6) / 300 = 0.012 seconds, or 12 milliseconds.

Streaming events can contain several tokens, so gaps between events don't recover each token's timing. Weighting also matters when combining requests. In Anyscale's example, one request returns 100 tokens at about 10 ms each and another returns 2 tokens at about 50 ms each. Giving each request equal weight yields 30 ms; weighting by tokens yields about 10.8 ms. Check which convention your dashboard uses.

Keep raw durations or aggregatable distributions, because precomputed quantiles can't be combined into a global percentile.

For a simple example, suppose replica A serves 900 requests that each take 0.5 seconds, and replica B serves 100 requests that each take 2 seconds. Their p95s are 0.5 and 2 seconds. Averaging those gives 1.25 seconds, but the pooled p95 is 2 seconds: the slowest 10% of all requests took 2 seconds. Calculate the percentile from the combined observations or histogram buckets.

Trace the slow path before changing the model

Consider a policy assistant that answers questions from company documents. Before returning an answer, it verifies access, searches the documents, reranks the passages, calls the model, and checks the citations.

Each of those steps can add to the user's wait. Tracing them separately shows which part needs attention.

Give each operation its own span, with a parent span covering the application request. Record when the model call starts, when useful answer text arrives, when generation ends, and when validation finishes. The RAG pipeline stages are useful boundaries for separating retrieval work from generation.

Measure distinct boundaries. Application completion includes preparation and validation; first answer text and final model output are separate milestones. Conceptual order only, not measured durations.

If citation checking happens after the model finishes, a faster model doesn't remove that check. If two independent searches overlap, don't add their full durations and call the sum application latency. Follow the path the request actually waits for.

For each result, keep enough context to reproduce the comparison:

• Configuration: model identifier or snapshot where available, endpoint, prompt version, generation settings, and streaming mode.

• Workload: task category, input and output lengths, retrieval size, and whether tools were required.

• Environment: client/server region where known, request rate or concurrency, competing work, and cache state.

• Outcome: completed, timed out, canceled, or failed; retry count; quality verdict; and the timing boundary used.

A hosted LLM API may not expose its queue or prefill spans. Leave those components unknown rather than subtracting guesses from the round-trip latency. Compare what you can observe, then use provider telemetry where it exists to narrow the explanation.

Compare like workloads before interpreting the latency chart:

• Separate cold starts, cache misses, and warm-cache traffic.

• Compare short answers with short answers, and long-document requests with similar inputs.

• Report p50 and p95 for each useful slice. Use p99 only when you have enough samples to characterize the tail, and state the sample count and collection window.

Keep failed requests in the outcome report even when they have no completed-response latency. Otherwise, a configuration that times out on the hardest inputs can appear faster simply because those inputs disappeared from its latency distribution.

Match each bottleneck to an intervention you control

An API consumer can change prompts, requested output, workflow structure, and supported model settings. A serving owner can also change scheduling, memory allocation, capacity, and deployment topology. Keep those scopes separate, so the next experiment is something your team can actually run.

Back to the policy assistant: suppose answer text arrives promptly, but users keep waiting while citations are checked afterward. That points to the table's final row. Test the citation checker, such as verifying independent citations in parallel, and require the same citation-quality checks to pass.

Shorten work before buying faster work

For a classifier, ask for the label and required evidence rather than an essay explaining every alternative. For an extractor, preserve required fields but remove commentary the consuming code ignores. A lower output cap is only useful if the response still completes correctly; a cut-off answer isn't latency reduction worth shipping.

Test instruction and format changes through the same prompt optimization loop as any other behavior change. Include cases that need an exception, an abstention, or a longer explanation, so the concise variant doesn't win by avoiding difficult work.

Apply the same discipline to context management. Removing irrelevant retrieved passages is a different experiment from compressing every passage. First check that the required evidence survives; then compare latency and answer quality. A shorter prompt that forces another retrieval or generation attempt may move work elsewhere rather than remove it.

For repeated instructions or reference material, investigate prompt caching. Keep reusable content stable, but verify the actual model's cache rules and returned cache usage. A shared prompt prefix doesn't guarantee a cache hit: matching boundaries, retention, and model-specific behavior matter. Cached input processing also doesn't mean the previous answer is returned instead of a new generation.

Remove avoidable dependencies

Draw the dependency graph before parallelizing API calls. Searching two independent, authorized sources can be a candidate for overlap. Steps that depend on each other must wait: generation that starts before retrieval returns has no retrieved evidence to use. Record where branches join, and measure when the final result is ready.

OpenAI's latency principles include fewer requests, parallel work, and avoiding an LLM where ordinary code is sufficient. A deterministic status lookup or format conversion deserves that test before another model call. Keep required validation ahead of any consequential action; changing the schedule must not silently change the application's acceptance rules.

Tune the serving system only when you own it

For self-hosted inference, inspect queue time, running requests, KV cache pressure, and preemption alongside GPU utilization. A busy GPU alone doesn't identify which knob will improve the response time your users care about.

One concrete example is vLLM V1 chunked prefill. Prefill processes the input prompt; decode generates the output tokens. vLLM V1 prioritizes pending decode requests, then uses the remaining max_num_batched_tokens budget for chunks of prefill work.

Reducing the batched-token budget can favor inter-token latency, while increasing it can favor time to first token by processing more prefill work per batch. Measure both TTFT and ITL before and after the change.

Continuous batching and load balancing also need workload-specific checks. Compare the request distribution and load you expect, not just a full GPU and a high tokens-per-second counter.

Quantization, speculative decoding, and tensor parallelism are operator experiments with compatibility and hardware constraints; none guarantees latency improvement. Model size, memory bandwidth, communication, and scheduling can all change which configuration wins. Keep a known-good serving baseline while testing one declared change.

Use streaming to shorten waiting without hiding slow completion

Streaming lets a client receive output while generation continues. Users can start reading sooner, but streaming does not by itself make the model generate tokens faster. Measure the display milestone separately from model completion.

OpenAI's streaming interface distinguishes text deltas from lifecycle and function-call events. Your UI should make a similarly deliberate distinction between “work started,” answer text, and a completed result. A status indicator is useful feedback, but it isn't evidence that the answer has begun.

Test the whole delivery path: server event emission, reverse proxy, client receipt, and rendering. If the client receives small chunks promptly but renders only after completion, the serving system isn't the place to fix the visible wait.

If a proxy buffers the response, changing the model may leave that symptom untouched. nginx enables proxy buffering by default. In the nginx location that serves the stream, test proxy_buffering off; and confirm that chunks reach the browser as they arrive.

An upstream X-Accel-Buffering: no header can also disable buffering, unless proxy_ignore_headers lists that header.

Structured output needs particular care. Streamed argument fragments aren't complete validated tool arguments. Don't execute an action because an early fragment looks plausible; wait for the required completion and validation checks. For a read-only assistant, show safe partial text where appropriate while still recording whether the full response completed.

Streaming responses are therefore a delivery decision as well as a performance experiment. Define which partial content is useful, which must be withheld, and what the user sees when a stream fails midway.

Compare configurations under the same quality and load conditions

Choose the acceptance rule before choosing the fastest row. For the policy assistant, that might mean a correct answer with citations supporting the material claims, or a correct abstention when the documents don't contain the answer. A shorter response fails if it drops the exception the user asked about.

Build a versioned evaluation set containing ordinary requests and known difficult cases. Freeze the quality checks while comparing configurations. Use exact checks for required labels or fields; where judgment is necessary, calibrate an LLM judge against human-reviewed examples rather than accepting whichever evaluator produces the highest score.

Then run the comparison in two parts:

1. Find quality-acceptable candidates. Compare declared model, prompt, retrieval, or workflow variants on the same tasks. Retain the baseline and inspect failures by category, not only aggregate scores.

2. Validate serving behavior. Run the finalists through the actual endpoint with the intended input/output distribution and a controlled load sweep. Record arrival rate or concurrency, cache warmup policy, repeated trials, and resource conditions. Keep timeout, error, and completion rates beside latency percentiles.

Use a blank record like this for each configuration and workload slice. Fill it with the observations from your own run.

For the policy assistant, compare configurations with our platform using a small grid: two API models, two answer prompts, and two retrieval depths give eight candidates. Our multi-config experiments run these against shards of the same evaluation set. Your custom metric functions score citation quality. Declare the metric's aggregation type and value range to get running estimates with confidence intervals, and shuffle the data first if it is grouped by difficulty or label.

With RFAPIModelConfig generators, MLflow enabled, and accumulate_metrics_fn configured, the dashboard also updates average query latency after each portion of the evaluation set finishes.

That number times the complete, non-streamed model call. It leaves out retrieval and reranking, so it is closest to the model-request end-to-end measurement in the table above. Keep measuring time to first token, gaps between tokens, and latency percentiles at your endpoint. Self-hosted vLLM generators do not produce this dashboard metric.

Use the running results to screen candidates while the experiment is active. The IC Ops panel lets you stop a weak run or Clone-Modify a promising one with a different prompt and compare the new variant. These operations take effect at shard boundaries. Converge, our automation layer, uses interim results to stop weak variants, clone promising ones, and launch new configurations within your time and spend limits.

If configurations share hardware or a provider quota, record that contention and rerun finalists under controlled conditions. Preserve individual timing observations or suitable distributions for percentile analysis; don't sum or average batch percentiles as if they were counts.

Choose the next experiment from evidence

Pick the slow span, a change you control, and the quality condition that must survive it. Keep the baseline until the candidate passes both task evaluation and realistic serving checks. That gives you a defensible latency improvement.

To compare configurations for your workload, talk to our team with representative tasks, your current measurement boundary, and the latency target you're trying to meet. Keep inference-provider usage and serving-test costs in the experiment budget.

LLM latency optimization FAQ

How do I optimize LLM latency?

Measure application completion and the model request separately, then identify the longest relevant wait. Test a change that targets it, such as less unnecessary output, fewer serial calls, or a serving adjustment. Keep quality and failure rates in the acceptance rule.

Which LLM has the lowest latency?

There isn't a useful universal winner. The result depends on the task, output length, endpoint, load, and quality requirement. Compare candidates on your own workload, using the same timing boundaries and serving conditions.

How can I make an LLM respond faster?

Separate starting sooner from finishing sooner. Streaming can expose useful text earlier, while reducing unnecessary work can shorten completion. If queueing dominates, another prompt edit may not address it. Confirm the cause before changing the implementation.

Why can throughput improve while latency worsens?

Throughput counts work completed across the system; latency describes a request's wait. Serving more requests together can increase aggregate tokens per second while individual requests take longer. Check both, and report how much completed work also met your quality and deadline requirements.