LLM Model Selection for Your Application

Suppose your support team wants to use an LLM to sort incoming tickets. The application needs two things from each response: the right queue for the ticket and the facts a support specialist needs to handle it.

One model returns valid JSON every time, but sends a billing dispute to the technical team. Another chooses the right queue but leaves out why the customer is disputing the charge. Neither is a good fit for this task, even if it scores well on a public benchmark.

LLM model selection means choosing a model and configuration for the work your application actually performs. Start with what must be true of the result, remove candidates that cannot meet those requirements, then compare the eligible ones on your own cases. The goal is a choice you can defend, with its limitations written down.

Define the task and the constraints a model must meet

For our support-triage example, the model receives a ticket and its conversation history. It selects an allowed category and extracts the evidence a support specialist needs. It must also handle missing information without inventing details and finish within the application’s response-time budget.

Write the acceptance criteria before collecting model scores. OpenAI's model-selection guidance recommends comparing representative tasks on the same inputs and deciding in advance what qualifies as a useful result. Apply that principle to your application.

Separate hard constraints from preferences. A model that cannot run through an approved endpoint is ineligible even if its outputs look good. A model that needs a little more review may remain eligible if review is part of the intended workflow. Don't let a blended score silently make that decision for you.

Before replacing the model, inspect one wrongly categorized ticket and ask:

• Was the category definition ambiguous?

• Was a relevant message missing from the input?

• Was the evidence present, but the model ignored it?

If the next test also supplies better context, you cannot tell which change helped. Fix the context first, then compare models on the same inputs.

Keep this diagnostic bounded. Save one failing input, its assembled prompt, available evidence, and expected result. Use that record to decide whether you need a model comparison, a prompt change, or a separate investigation of the surrounding application.

Shortlist eligible candidates using current evidence

Create a shortlist from candidates you can actually deploy. Include the current model as a baseline where possible, then choose alternatives that test a useful hypothesis: less cost for the same constrained task, better handling of long tickets, or a deployment arrangement your team can operate.

Public benchmarks are a starting point for finding candidates, not acceptance tests for your application. A general score does not show whether a model preserves your evidence fields or distinguishes your ticket categories. AWS's model-selection example makes this distinction explicitly, comparing task quality alongside latency and cost rather than relying on leaderboard position.

Use a matrix like this for each candidate. Fill in one copy per candidate, leaving a cell empty until you have checked its source.

Check the exact endpoint rather than assuming every version in a model family has the same features. A shortlist entry should be specific enough that another engineer can reproduce which system you evaluated. “The latest model” is not an adequate identifier for a decision record.

For downloadable weights, Hugging Face model cards describe intended use, limitations, training information, and evaluation results, with metadata that can display a license. Treat the card as documentation to inspect, not a certification. Follow the actual license and resolve missing information before marking the candidate eligible.

Open weights and open source are different checks. Model weight availability does not, by itself, establish the permissions or materials required by an open-source definition. The Open Source Initiative's Open Source AI Definition specifies more than access to weights. Keep the license review with the appropriate owner instead of inferring permission from a repository badge.

Also keep hosting separate from model identity. Running weights yourself and calling a provider that hosts those weights are different operating arrangements. Verify where data goes, who operates the endpoint, and what your team must maintain. Neither arrangement is an automatic answer to your organization's requirements.

Build a representative comparison without leaking the answer

The comparison should resemble the work you will ask the selected model to do. For the triage example, collect sanitized tickets covering routine requests, overlapping categories, missing facts, and long histories. Keep reference outputs and evaluation rules outside the inputs shown to the candidate.

Task-specific evaluation combines representative inputs with explicit criteria and checks calibrated against human judgment. Typical, edge, and adversarial cases serve different purposes; retain those distinctions in your report rather than hiding them inside one average.

Use two separate pools of cases. Development cases are available while you revise prompts and settings. Held-out cases confirm the chosen configuration after those changes are frozen.

Once you tune against held-out failures, that set becomes part of development. Record the change and collect fresh cases for confirmation.

For the triage example, cover these slices:

• A short ticket with one clear request, to establish basic routing behavior.

• A conversation containing multiple issues, to test the category priority rule.

• A long history with the relevant evidence buried among earlier messages.

• Missing or contradictory facts, where a review-needed result is acceptable.

• Relevant languages and formats, including the ones your team finds difficult to label.

• Ticket text that tries to override the task, so instruction-following failures remain visible.

Specify what passing means at the case level. A ticket about a disputed charge may require the billing category and the customer's stated reason, but not a confident explanation of what happened in the billing system. If the necessary evidence is absent, the accepted output may be a request for review rather than a guessed field.

Use code for exact conditions such as valid categories and required fields. Reserve a rubric for properties that need judgment, such as whether the summary preserves the material facts. When calibrating an LLM judge, compare its verdicts with expert labels and inspect disagreements, especially false passes on important failures. Also check for self-preference toward a candidate's outputs and, in pairwise judging, sensitivity to answer order. A fluent explanation from the judge is not independent proof that an answer is correct.

Don't select a universal sample size from a tutorial. Plan coverage around the decision you need to make, report how many cases each slice contains, and acknowledge what remains uncertain. A handful of ordinary tickets provides little evidence about an uncommon failure. Likewise, a small apparent difference between candidates may not justify switching. Before running more tests, decide what additional evidence would resolve that uncertainty.

Compare models and configurations under a fair experiment

Two useful questions in a model comparison are: which candidate works best with the existing application, and which can work best after a comparable amount of adaptation. Run those as separate phases so the result answers a recognizable question.

First, run a fixed baseline comparison. Use the same inputs, rubric, retrieved evidence, and tool fixtures. Preserve the application prompt where the interfaces allow it, and record unavoidable adaptations. If one endpoint uses a different tool schema or request format, document that conversion rather than pretending the requests were identical.

Save a manifest with every run:

• Model ID, version, provider, and endpoint.

• Prompt template and example selection.

• Dataset version and retrieval snapshot.

• Available tools and their fixed responses.

• Supported sampling or reasoning settings and output budget.

• Request timeout, retry policy, concurrency, and cache behavior.

• Scorer implementation, rubric and scorer model, if one is used.

Then allow a budget-matched tuning phase if it serves the decision. Choose a comparable search budget and stopping rule for each candidate. The budget changes the question you answer:

• Equal trials: which model responds best to the same number of tuning attempts?

• Equal dollars: what can each model achieve within the same spend, allowing cheaper trials to run more often?

• Equal engineering time: what can the team achieve within a fixed sprint?

Record the rule and actual inference spend separately.

Prompt optimization belongs in this second phase. A prompt tuned for one model need not transfer equally well to another, so identical prompts are useful controls but not proof of each candidate's best achievable result. In AWS's scoped 360-Eval example, the authors explicitly acknowledge that their prompts favored Anthropic models. That limitation is a reason to label the experiment's scope, not a reason to discard controlled comparisons.

Retain per-case outputs, errors, timeouts, and retries. When the scorer fails, mark the evaluation as unresolved rather than assigning the candidate a neutral score. When the application fails to complete a request, keep that operational failure visible even if a later retry succeeds. Otherwise, the report describes only the requests that survived its filters.

Repeat representative cases when output variability matters. Setting temperature to zero is not a cross-provider guarantee of identical results. For example, OpenAI's archived 2023 reproducibility example documents residual nondeterminism even with matching request settings and backend fingerprints. Check the current behavior of the endpoint you use, and never count a replayed cached answer as another independent generation.

For the support-triage example, compare a lower-cost model with a stronger candidate on the same tickets. Score whether each picks the right queue and preserves the customer's reason for contacting support. Inspect the difficult cases as well as the average: a cheaper model is only useful if it still handles the tickets your team cannot afford to misroute.

Our Converge platform helps you run that comparison. It tests model and prompt variants against your evaluation examples and metrics, then uses interim results to stop weak variants, clone promising ones, and try new candidates within your time and spend limits. You can inspect the results and the reasons behind those decisions instead of managing every trial yourself.

Use the comparison to choose a candidate that meets your quality requirements at an acceptable cost and latency, then confirm it on held-out tickets. Bring your own cases and scoring rules, and include provider charges in the experiment budget. Our experiment walkthrough is a starting point when you are ready to set up a comparison.

Compare quality with the cost and latency of completed work

A low price per token is not enough to choose a model. You need to know what it costs to obtain an outcome the application can use, including attempts that fail and work that needs repair. Keep those measurements next to quality rather than declaring a winner from any one column.

Use this table to record the comparison. Add a column for every eligible candidate and keep the per-case records behind each result.

Define the timing boundary. Time to first token measures when the first output reaches the caller; token-generation speed describes the output stream. Neither alone tells you when triage finishes. Include the application steps needed to return usable output, and report human review or repair time separately when it is part of the workflow. OpenAI's selection guidance includes that review-and-repair time in the practical comparison.

Keep the test conditions with the measurements: input and output lengths, concurrency, cold or warm behavior, timeout policy, and observation window. Don't compare one candidate's lightly loaded interactive run with another's saturated batch run and attribute the entire difference to the model.

• API candidates: record billed usage categories and provider prices at measurement time, including retries and failed work. Keep evaluator calls separate from application serving costs.

• Self-hosted candidates: include provisioned resources, utilization, serving overhead and engineering or operations effort. Removing per-token billing does not remove these costs.

Calculate the cost of getting a result your application can use:

Cost per accepted outcome = total spend within the declared boundary ÷ accepted outcomes

Cost per accepted outcome = total spend within the declared boundary ÷ accepted outcomes

Cost per accepted outcome = total spend within the declared boundary ÷ accepted outcomes

Include the cost of unsuccessful attempts in total spend. If no outcomes are accepted, the ratio is undefined. Report the failure count.

For a worked example, suppose candidate A costs USD 12 for 910 accepted outcomes, while B costs USD 8.40 for 560. A costs about USD 13.19 per 1,000 accepted outcomes; B costs USD 15.00. B has the cheaper run, but A costs less per usable result.

For the triage example, say whether an “accepted” result must be correct without intervention or can include a specialist's repair. Both are useful measurements, but they describe different products. If you include repaired results, record the repair workload so a cheaper model does not conceal extra work for the support team.

Make and document the selection decision

Apply mandatory constraints before choosing among tradeoffs. Remove an ineligible candidate even if it has the best average quality score. Then compare the remaining model choices on the outcomes and operating conditions you agreed to measure.

If candidate B ties or trails A on every relevant measure and loses on at least one, you need a specific reason to pick B. An operational requirement missing from the comparison might justify it, but that requirement belongs in the record and may need another check.

When the tradeoff is real, make it explicit. In the illustrative triage application, one candidate might require less manual repair while another might finish sooner. The decision owner must choose which difference matters, subject to the hard limits.

An inconclusive result is allowed. If the difference is unstable or a critical slice has too little evidence, name the next experiment and the question it will answer. For example, compare the two finalists on more ambiguous multi-issue tickets while holding the scorer fixed. Don't keep changing the evaluation until a favored candidate wins.

Make the follow-up proportional to the decision. A change that only reduces a minor operating cost may not justify a migration with unresolved quality risk. Conversely, you can't defend a baseline that fails a mandatory requirement by calling the alternatives inconclusive. Separate “no reason to switch yet” from “the current model is acceptable.”

You also don't need one model for every task. Ticket classification and drafting a customer reply can have separate acceptance criteria and separate model selections. Record those task-level choices independently; deciding how a live system dispatches requests between them is a different architecture decision.

A useful selection record should let another engineer explain why the model was chosen without rerunning the experiment or relying on the person who ran it.

Revisit the choice when the application changes

Keep the selected configuration as the reference point, along with the evidence that justified it. A later comparison should explain what changed and why the old decision might no longer hold.

Set concrete triggers for review:

• The provider deprecates the selected model or changes the version you use.

• Ticket languages, lengths, or issue types change materially.

• The application needs a new modality, output field, or tool.

• Required task outcomes regress, including on a previously acceptable slice.

• Actual cost, completion time, or manual-review workload shifts enough to affect the decision.

• A newly eligible candidate offers a specific hypothesis worth testing.

When replacing an endpoint, preserve the previous decision record, compare the replacement against it, and follow your team's regression and release process. A familiar model-family name does not establish that the replacement behaves the same.

The same discipline answers the common selection questions. You don't need the biggest model if an eligible smaller candidate meets the task's quality floor. A leaderboard cannot make the application-specific decision for you. And one successful choice for classification does not establish the right choice for summarization or tool use.

Start with a bounded experiment: one task, explicit acceptance criteria, a small eligible shortlist, and cases that expose the differences you care about. Our OSS walkthrough shows how to assemble a configuration experiment with your own preprocessing and scoring. Its API-backed setup can run on a CPU-only machine; you still need approved provider access and must account for API charges. Feed its results into the decision record above.