LLM Cost Optimization That Reduces Cost per Successful Task

A lower-priced model or shorter prompt can reduce the API bill while increasing the cost of resolving a question. Consider a support assistant that answers questions about a company's policies. A routine lookup may be straightforward, but a question involving a policy exception needs the answer to preserve that exception.

If the assistant misses it, resolving the question may require another attempt or a human correction. The cheaper request still happened; it just didn't finish the job. Those failures belong in the comparison alongside the cheaper requests.

Use cost per successful task to decide whether a configuration is an improvement. This guide works through establishing a baseline, choosing a cost lever, and comparing configurations under the same quality and latency requirements.

What LLM cost optimization should optimize

LLM cost optimization reduces the cost of completing useful work while meeting explicit quality and latency requirements. Here, cost per successful task means total measured run cost, including failed attempts, retries, and tool charges within a stated accounting boundary, divided by the number of successfully completed tasks.

Write the success rule before comparing models. For a policy assistant, it might require a correct answer, supporting evidence, and no omitted exception. An appropriate refusal can count as success when the task calls for one; refusing every difficult question cannot.

Cost per successful task = total included run cost ÷ successful tasks

Cost per successful task = total included run cost ÷ successful tasks

Cost per successful task = total included run cost ÷ successful tasks

Count each task execution once, even if it contains several API calls or retries. If nothing succeeds, the ratio is undefined, not zero, and that configuration cannot win.

Keep four quantities separate:

• Token price: the provider's rate per token.

• Total application spend: the whole bill, including failed attempts.

• Experiment cost: what you spend comparing configurations.

• Production serving spend: what the selected configuration costs at real traffic volumes.

A lower cost per token does not establish lower LLM inference cost per completed task. Track the experiment budget separately from the production savings it may help you find.

Establish a cost, quality, and latency baseline

Start with a fixed set of tasks that reflects what your application receives. For the support assistant, separate ordinary policy lookups, questions involving exceptions, and requests that lack enough information to answer. Preserve the slice labels so a good overall score cannot hide a broken exception-handling path. This follows the task-specific testing and human-calibrated scoring approach in OpenAI's evaluation guidance, rather than relying on a generic benchmark.

Record enough information to explain the cost of each execution:

• Identity: task ID, slice, configuration ID, provider, model version, and prompt/retrieval versions.

• Usage: input tokens, output tokens (including billed reasoning tokens where applicable), applicable cached-input or cache-write categories, model calls, retrieval operations, tool charges, and retries.

• Outcome: success or failure under the frozen rubric, with a reason and any manual correction recorded separately.

• Timing: end-to-end duration, including routing, retrieval, retries, and the terminal outcome. Report p50 and p95, and keep failures and timeouts visible.

Reconcile token usage against the provider's actual accounting. Missing usage for a failed request is unknown cost, not evidence that it was free.

Don't charge cached input once at the ordinary rate and again at the cache rate. OpenAI, for example, documents mutually exclusive uncached-input, cached-input, and cache-write rates; the applicable rules depend on the model. The prompt-caching reference explains those categories.

For OpenAI Responses usage, ordinary input is input_tokens minus input_tokens_details.cached_tokens and, where applicable, input_tokens_details.cache_write_tokens; price each category separately.

The table compares two hypothetical configurations on the same tasks. The cost boundary includes generation, retrieval, tools, and all failed or repeated attempts, but excludes human labor and fixed infrastructure.

Per request, B looks better: USD 15 ÷ 100 = 15 cents, compared with A's 20 cents.

Per successful task, the result flips. B's 60 successes put it at USD 15 ÷ 60 = 25 cents. A's 90 successes put it at USD 20 ÷ 90, or about 22 cents.

B also leaves 40 failed tasks to resolve. The table excludes that follow-up labor, and neither row alone proves the configuration meets your quality or latency requirements.

If infrastructure or manual correction belongs in your decision, add it consistently to both configurations. Don't compare one fully loaded cost with another token-only estimate.

Choose the lever that matches the bottleneck

Use the baseline to choose what to test. A prompt-prefix cache won't remove redundant workflow steps, and model routing won't fix a retriever that never returns the policy exception.

Model choice and routing

Compare candidate models on the same tasks before building a router. A smaller model is a candidate, not an automatic winner. Keep model selection tied to explicit acceptance criteria, as OpenAI's model-selection guide recommends.

For the support assistant, model routing could send routine lookups to a lower-cost model and questions involving exceptions to a stronger one. Test the router's mistakes too: an exception question misclassified as routine still belongs in the routine route's bill and failure count. If fallback calls repair those mistakes, include their cost and time rather than scoring only the final model call.

Prompt and retrieved context reduction

Begin with duplicated instructions, repeated passages, and examples that no longer improve the score. Our prompt optimization guide explains how to compare these edits without mistaking a shorter prompt for a better one.

For RAG, distinguish the candidate set retrieved from the passages actually passed to the generator. Retrieving broadly and reranking to a smaller context is a different experiment from retrieving less in the first place. The retrieval pipeline makes those decisions separately. Include the reranker's work when comparing costs; a shorter generation prompt is only part of the calculation.

Treat conversation-history compaction the same way. Check whether the compacted version retains the user's constraints and unresolved questions. Measure the compaction step itself and any extra calls needed to recover discarded information.

Prompt prefix caching and response caching

Prompt caching reuses the processed state of an eligible shared prefix; the model still generates a new answer. Keep stable instructions together and measure actual cache reads rather than assuming repeated text always hits. Check the model's minimum cacheable prefix length before expecting a hit. Eligibility, rates, and cache lifetimes vary by provider and model.

Response caching instead returns an existing answer. Semantic caching broadens the lookup to similar requests, which can produce an incorrect or outdated match. Azure's semantic-cache documentation explicitly warns about this distinction. “Can I cancel my annual plan?” and “Can I cancel my annual plan after renewal?” are a useful hypothetical near-match test: similar wording may require different policy answers.

Partition protected response caches by the relevant user or tenant context, authorize access before returning an entry, and invalidate affected answers when their source material or permissions change. OWASP's cache-isolation guidance covers why a separated cache key alone is not authorization. Score cache-hit answers as outputs, not as automatically successful requests.

Asynchronous batch work

Bulk classification and scheduled evaluations may tolerate delayed results; an interactive support answer usually cannot. OpenAI's Batch API guide currently documents a 50% discount relative to synchronous APIs and a 24-hour completion window. Those are OpenAI's terms, not a universal property of batching.

Check model eligibility, job outcomes, and the deadline before counting a discount. Don't assume batch and cache discounts stack: use the actual pricing rules and billed categories for the selected provider and model.

Workflow calls, retries, and stopping criteria

Inspect the trace for repeated retrieval, unnecessary rewriting, and loops that keep asking the same unanswered question. Give each loop a tested stopping condition and record why it stopped. A cap that lowers spend by abandoning valid tasks is not a quality-preserving improvement.

Keep verification steps that catch meaningful errors. If a deterministic check can replace a model call, compare the resulting outcomes rather than deleting validation altogether. OpenAI's latency guide distinguishes reducing requests and generated tokens from making the user wait less. Streaming can show progress earlier; it does not, by itself, reduce inference work or the bill.

Constrain output tokens only where the result still fulfills its contract. For OpenAI reasoning models, max_output_tokens limits both reasoning and the visible answer. A tight cap can produce an incomplete response before any visible answer while still incurring input and reasoning charges; count that unsuccessful execution and its cost. A classifier may need a label, while a policy explanation needs enough detail to communicate its exception. Don't multiply separate savings estimates across these levers: changes to context, models, and caches affect overlapping parts of the same workload.

Run a controlled configuration experiment

Freeze the dataset, scoring rubric, and baseline before changing the application. Then define the quality floor and p95 latency ceiling that a candidate must meet before cost can decide anything. For example, require the exception slice to pass its own acceptance rule instead of letting routine lookups compensate for it.

Use a small, interpretable configuration grid: model alternatives, prompt variants, and retrieval settings chosen from the observed bottleneck. First compare a change against the baseline; then test promising combinations. That makes it easier to distinguish a useful individual change from an interaction that only works in one combination.

Run every candidate on comparable task slices. Repeat sampling where output variability matters, retain seeds where supported, and keep model settings fixed unless they are part of the experiment. Reserve held-out tasks for confirmation after selecting a candidate. A narrow lead on the development set needs more evidence, especially when a small number of difficult tasks decides the result.

Keep an experiment record that someone else can audit:

Our Converge platform helps you compare model, prompt and retrieval variants on the same tasks. For the policy assistant, a cheaper candidate must still preserve the exceptions that make an answer correct. Converge uses interim results to stop weak variants, clone promising ones and try new candidates within your time and spend limits.

Compare cost per successful task using total spend and total successes from the same executed tasks. Aggregate the totals before dividing. Consider two batches:

Batch 1 costs USD 5 and has 25 successes: 5 ÷ 25 = USD 0.20 per success.

Batch 2 costs USD 30 and has 75 successes: 30 ÷ 75 = USD 0.40 per success.

Pool the totals: (5 + 30) ÷ (25 + 75) = USD 0.35 per success.

Averaging the two batch ratios gives USD 0.30, incorrectly giving the smaller batch equal weight. Adding them gives USD 0.60. Neither is the cost per success across both batches.

Use actual charges for the final decision, including retrieval, repair or other costs you chose to count. Our dashboard's token and spend figures project the whole evaluation set while a run is unfinished. For example, a four-shard run that uses USD 2 on its first shard may show about USD 8 in projected spend. Do not divide that projection by successes from only the completed shard. Unknown model pricing leaves spend absent, and self-hosted vLLM runs do not produce these API-generator system metrics.

Validate the winner in production and keep rollback

Promote the confirmed candidate through a controlled rollout using your existing deployment and monitoring tools. Keep the previous configuration available, name the person who can revert it, and define the quality, latency, and cost conditions that trigger investigation or rollback.

Compare production traffic slices, not just the aggregate invoice. A rollout that receives more routine questions can look cheaper even if the configuration did not improve. Likewise, a warm cache during an offline replay may not represent new users or changing documents. Track the actual cache-hit mix and review samples of reused answers.

Watch request volume, retries per task, failed outcomes, cost per success, and latency together. Add budget ceilings, but decide what the application should do when it reaches one: queue eligible work, use a separately validated fallback, or stop with a clear explanation. Don't silently degrade the quality contract to stay under budget.

Re-run the comparison when provider behavior, model versions, retrieval data, or the traffic mix changes. Cost governance needs an owner for both spend and outcomes; otherwise, a lower invoice can conceal work transferred to support staff. Keep manual correction visible even when it sits outside the model-cost boundary.

Compare cost under your quality bar

Pick one expensive task slice, freeze its success criteria, and compare a candidate with the baseline.

Our RAG walkthrough shows how to compare retrieval variants on a small dataset. Treat it as a starting point for the experiment, then apply your own success criteria and cost accounting. API-backed runs require provider access and incur usage charges; the small Colab example itself does not report spend.

LLM cost optimization FAQ

What is the most cost-effective LLM model?

The one that completes your tasks at the lowest measured cost while meeting its quality and latency requirements. Evaluate the whole configuration, including context, routing, retries, and tools. A model that is economical for classification may be a poor choice for questions requiring policy exceptions. There is no workload-independent winner.

Why are LLMs so expensive?

For an application, the useful question is which work you repeatedly pay for. Long inputs, generated output, multiple calls, and failed attempts can all contribute. Retrieved context and tool execution add their own costs. Break the bill down by task and component before choosing an optimization; a token-price comparison alone misses that boundary.

How expensive are LLMs to run?

Estimate your own cost per task from representative executions, then project it against expected volume and workload mix. For API-based systems, use applicable provider charges; for self-hosted serving, include the infrastructure and operating costs you actually bear. Record which costs are excluded, and revisit the estimate when usage or the configuration changes.

Is prompt caching the same as context compression?

No. Prompt caching reuses processing for a matching prefix; compression changes the context supplied to the model. Evaluate them together, because changing that prefix can affect cache reuse, while removing information can change whether the task succeeds.