Building an enterprise AI assistant that can retrieve information and produce a plausible answer is relatively easy. Building one that is measurably accurate, consistently grounded, and optimized for its intended outcome is a much harder problem.

Application quality depends on a long list of interacting choices: how documents are chunked, tokenized, and embedded, how candidates are searched, how many are retrieved, whether a reranker re-scores them, and how many survive. The space is large, the knobs interact, and intuition is a poor guide.

This is a case study of two RapidFire AI partners and their AI-powered retrieval systems. One is a global leader in enterprise data and analytics, with a customer-facing chatbot over travel-insurance policy documents. The other is a global automotive manufacturer, with an internal knowledge assistant spanning four manufacturing domains.

For each customer, RapidFire AI evaluated numerous candidate configurations in a single  hyperparallel, evidence-driven sweep instead of a slow sequence of hand-picked guesses and found winning configurations neither of their teams had reached by hand:

• For the first, we enhanced retrieval quality by about 43 percent with the generator LLM held fixed. 

• For the second, we beat the better of the customer’s two prototype pipelines on 5 out of 7 metrics by searching across the whole pipeline, from document parsing to the reranker.

The Use Case

Both customers had the same question for their working retrieval + generation pipeline: Is this the best we can do?

The data-and-analytics leader had a chatbot, published by its solutions team as an illustrative example of the use case, that answers policyholder questions from a travel-insurance policy. It was built to show what the pattern can do, not tuned for retrieval quality–so there was real headroom to find. 

The automotive manufacturer had a knowledge assistant spanning four production domains (Welding, Assembly, Electronics, and Powertrain) and 28 expert-curated questions, plus two production stacks its own team had already built and benchmarked, which we refer to here as Vector DBMS 1 and Vector DBMS 2.

Different industries, different corpora, same problem shape: a configuration space too large to explore by hand, and no principled way to tell which knob was actually moving quality.

What Was Slowing Them Down

An AI retrieval system has many tunable knobs and each one interacts with the others. Chunk size changes what the embedder sees. The embedder changes what the search returns. The search type and the number of retrieved chunks change what the reranker has to work with. The reranker changes what the language model finally reads. Changing any one of them can require re-indexing the corpus and re-running the full evaluation.

Figure 1. Examples of the many tunable “knobs” in an enterprise AI retrieval-and-generation pipeline, such as prompt templates, embedding specifics, chunking specifics, retriever, reranker, and generator hyperparameters; tool-calling specifics, knowledge graph configurations,, and agentic workflow structure. Each of these choices can materially affect accuracy, grounding, latency, cost, reliability, and business outcomes.

The common practice today is to hand-pick a handful of “best practice” combinations, run them one at a time, eyeball the answers, and stop when something looks “good enough.” That approach has two costs:

• First, sequential runs limit the search space you can cover, so most of it goes unexplored, potentially leaving significant application impact on the table.

• Second, skimming a few outputs tells you which run looked better, but not which knob was responsible or whether a different combination would have won. The decision ends up resting on intuition rather than evidence.

How RapidFire AI Changes The Game

RapidFire AI is an adaptive execution engine for AI experimentation. Instead of running configurations one after another, it uses shard-based execution to run large numbers of them concurrently on the same hardware and evaluation set. We call it “hyperparallelism”--this is what turns “try a few configs” into a “big-picture sweep” (Figure 2).

Figure 2. Before and after RapidFire AI. On the left, configurations are tried one at a time, which is slow, manual, and ad hoc, so only tens of configurations get tested. On the right, RapidFire AI runs them as a hyperparallel sweep with interactive control (stop, clone, and modify), testing far more configurations on the same hardware and converging on the best by evidence.

For the two customers in this study, RapidFire AI ran 40 and 432 configurations, respectively, each in a single sweep, with every run scored the same way on the same evaluation set. So the winners were chosen on evidence, not on “vibes” or what outputs happened to read well.

The two studies in this post sit at different points on that same method. Here is how they line up before we walk through each:

Table 1. The two studies at a glance: same engine, two scopes.

Case study 1: A Global Leader in Enterprise Data and Analytics

The Use Case: The first customer’s application, an illustrative customer-facing chatbot for travel insurance, enables a policyholder to ask a question in plain English, such as how the policy defines an accident, whether lost baggage is covered and up to what limit, or how to extend cover when a return trip is delayed, and get a short answer drawn straight from the policy wording. 

The Data: The source document is long, dense legal text, full of definitions, benefit limits, conditions, and exclusions, which is exactly what most people do not want to read end to end. The chatbot also handles follow-up questions, rephrasing each one against the conversation so that “what is the limit per bag?” is understood in the context of the question before it.

Why Retrieval Matters: Everything the chatbot says depends on retrieval. The LLM can only answer from the passages the retriever hands it, so when the relevant section of the policy is not retrieved, the answer is wrong or evasive no matter how capable the model is. 

To measure where it stood, RapidFire AI built a controlled benchmark: 76 question and answer pairs synthesized from the policy itself, each labeled at synthesis time with the page or pages it was drawn from. This is a self-generated benchmark, not a set of human-curated user questions, but it is internally consistent and deterministic to score, so relative comparisons between configurations are trustworthy. 

On that benchmark, the reference chatbot was surfacing the labeled pages only about half the time, which is unsurprising for an illustrative build that was likely never fully tuned–this made retrieval the obvious place to add immediate value. So we set out to answer a specific question: How much can retrieval engineering alone improve this chatbot, holding everything else fixed? 

The Setup: To make the comparison fair, we pinned the two most obvious levers to match the customer’s original system: the generator LLM (gpt-3.5-turbo) and the embedder model (text-embedding-3-small). Any gain therefore had to come from retrieval-engineering choices (e.g., chunking scheme and size, search type, number of chunks retrieved, and reranking), not from upgrading to a costlier LLM. RapidFire AI then swept 40 retrieval configurations over that same set. We set up the data and ran RapidFire AI for this use case in just a handful of days.

The Results: The winning configuration used a retrieve-then-rerank design: split the policy into 512-token chunks, retrieve the top 10 candidates by similarity, then let a cross-encoder reranker re-score them and keep the best 2. That single change lifted the retrieval composite score from the reference build’s 0.439 to 0.627, a gain of about 43 percent on this benchmark. And it improved every retrieval metric, not just the average (Figure 3).

Figure 3. Case 1: Per-metric retrieval improvement of the winning config over the reference build. Every retrieval metric rose significantly. The generator LLM and embedder were pinned to the baseline; so the gain is attributable to retrieval engineering alone.

These retrieval metrics are directly comparable across runs because every config was evaluated against the same labeled benchmark using the same scoring procedure, even though the labels are LLM-generated rather than human-assigned. The flip side is that the absolute scores reflect agreement with a synthesized answer key, not validated real-user success–so read 0.439 to 0.627 as a quick controlled demonstration that retrieval engineering moves the needle, not as an immediately production-ready measurement. 

Answer quality, scored separately by an LLM judge, moved in the same direction, which is some corroboration that the ranking is not just an artifact of the page labels. We report that judge gain qualitatively rather than as a single percentage, because the baseline and the sweep used different judge implementations (a multi-call RAGAS judge for the baseline, a single-call rubric judge for the sweep, both on gpt-4o-mini), so their absolute judge scores are not directly comparable.

The Bigger Picture: Ranking all 40 configs shows the pattern clearly (Figure 4). Most clear the baseline, and the strongest are nearly all retrieve-then-rerank designs. A follow-up leave-one-out ablation confirmed that every part of the winning recipe is load-bearing: removing the reranker, for example, trades ranking precision for more grounding context, and the composite drops. 

The reusable takeaway is simple: retrieve a wide set of candidates, then rerank. The exact chunk size and k that won here are specific to this policy, so a different corpus would need its own sweep to find its best values.

Figure 4. Case 1: All 40 swept configs ranked by retrieval composite. The dashed line is the pinned baseline at 0.439. Most configs clear it, and retrieve-then-rerank designs (coral) dominate the top of the ranking.

Case Study 2: A Global Automotive Manufacturer

The Use Case: This is an internal assistant for the factory floor. Technicians and engineers at the manufacturer’s plants use it to ask questions in plain language and get answers pulled from the technical documentation behind four production domains: Welding, Assembly, Electronics, and Powertrain. Rather than paging through manuals and specifications while a line is running, they can ask the assistant directly and get a grounded answer. This puts a premium on consistency.

The Data: The documentation is broad and uneven across the four domains. So, a system that answers well in one area can fall down in another, and an assistant that is unreliable in even one domain is hard to trust on the floor. The base data files are also a complex mix of modalities, spanning PDFs with annotated images, documents, and even spreadsheets with images.

The Setup: This customer came to us with a serious head start. Over a couple of months, their own team had benchmarked their existing stacks, hand-curated the 28-question evaluation set across the four domains, and built up a clear sense of what a good answer looked like in each. RapidFire AI started from that labeling and domain understanding; what we added was the breadth of experimentation on top of it, and, as the results and the ablation below show, that is where most of the lift came from.

The question here was broader. Where Case 1 measured lift against a single baseline on one composite score, here we compared the three systems across seven evaluation metrics spanning retrieval quality, grounding, relevance, and answer correctness. Several metrics were implemented through the Ragas evaluation framework using LLM-based judges, to learn which performed best. So instead of pinning the model and embedder, RapidFire AI opened every axis: the generator LLM, the embedder, the document-preprocessing variant, the search type, the number of chunks retrieved, the reranker model, and the reranker depth – a full grid of 432 configurations evaluated. After spending a handful of days with the team to understand the data, we ran this entire RapidFire AI experiment on it in just two days.

The Results: The winning config paired claude-sonnet-4-6 with text-embedding-3-large, a score-threshold search returning 10 candidates, and a cross-encoder/ms-marco-TinyBERT-L6 reranker keeping the top 5. Crucially, it ran on Vector DBMS 1, the vector database the customer already used: the win came from tuning the retrieval stack on top of that infrastructure, not from swapping it out. Measured on the customer’s own 28-question set, the RapidFire AI-chosen config led on 5 of the 7 shared Ragas metrics, ahead of the stronger of the two incumbents on each (Figure 5). It trailed on 2: Context Relevance, where it sits at 0.67 against 0.68 (a gap of 0.01, essentially a draw), and Answer Relevancy, where it trails 0.45 to 0.59 (a real gap).

Figure 5. Case 2: the three systems scored on the seven Ragas metrics, Vector DBMS 1, Vector DBMS 2, and the tuned winner (Vector DBMS 1 + RapidFire AI best). The RapidFire AI best run leads on 5 of 7 against the stronger incumbent on each metric and trails on 2 (Context Relevance and Answer Relevancy). Dotted markers show the customer’s internal pass targets.

The grouped bars also make a fair point visible: no system, including the winner, clears every internal pass target. So, the caveat here is that sweep found the best available configuration on this corpus, not yet a finished production-ready system.

The Bigger Picture: Another decision-relevant story is consistency. Split across the four domains, the RapidFire AI-selected config held Faithfulness at or above 0.70 in every one, from 0.72 on Powertrain to 0.83 on Welding. The two incumbent stacks were strong in places, Vector DBMS 1 reached an excellent 0.95 Faithfulness on Electronics, but each had a domain where its scores fell to zero on some metrics: Vector DBMS 1 on Powertrain (Faithfulness, Context Recall, and Context Precision all 0.00), and Vector DBMS 2 on Welding (Context Recall 0.00). Evenness across a broad, multi-domain corpus is genuinely hard to reach by hand, and closing that gap is where the sweep added the most value. For a production deployment, where worst-case behavior matters more than best-case, that consistency is the more decision-relevant result (Figure 6).

Figure 6. Case 2: Faithfulness by domain. The RapidFire AI run stays at or above 0.70 in all four domains (dashed line), while Vector DBMS 1 drops to 0.00 on Powertrain and Vector DBMS 2 clears 0.70 in only one domain.

Ablation Study: Because our sweep covered the full grid, it also supports an ablation analysis: given the winning config, change one knob at a time to each of its other values and plot the spread of the results. 

Figure 7 below does this for all ten metrics at once–the 7 Ragas metrics plus 3 LLM-judge metrics): 1 tile per metric, the 7 knobs in the same order in every tile, each knob’s resultant values shown from min to max. The winning config’s result for that metric is the dashed line reference. The y-axes are independent per tile–so read each on its own scale. A tall spread on one metric is not comparable to the spread on another.

Pareto Frontier: The panel makes a point a single headline number might hide: no config is best on everything. The winning config was selected by RapidFire AI’s composite score, which weights Answer Correctness at 50 percent because domain experts often trust it most as a single-number proxy for whether an answer is right. Across the full sweep the winner ranks first of all 432 runs on Answer Correctness, but only about 282nd on Faithfulness. The panel shows the same story locally: on the Faithfulness tile, swapping just the model to gemini-2.5-flash, or the search algorithm  to maximal marginal relevance (MMR), lifts the score past 0.90. Re-weight the objective toward Faithfulness, coherence, or relevance and a different config wins. The winner was the right choice for the objective we set–not a universal optimum. Users can change the weighting based on their business objectives  and obtain a suitable  winner for those objectives–RapidFire AI does not pigeonhole you on objectives.

Importance of Eval Metrics: The tiles also show that which knob is load-bearing depends on the eval metric we look at. A knob that looks safe on one can swing another: the embedder barely moves Faithfulness, about 0.02, yet it swings Answer Correctness by more than 0.2. This is the full-picture map an AI team actually needs–it is usually invisible if you just hand tune a few configs. Each point is a single config scored on the 28-question set, so read these spreads as local sensitivity around the winning config, not sweep-wide averages. Because RapidFire AI scored every config on every metric and keeps all of it, a team can re-weight to its own priorities and re-pick the winner without re-running anything. That is part of the payoff of its hyperparallelism: full transparency and auditability, so AI application developers can prioritize for themselves.

Figure 7. Case 2: A one-knob-at-a-time ablation around the winning config, shown for all 10 eval metrics. Each tile is one metric; for each of the seven knobs the other six are held at the winning run’s values while that knob takes each of its grid values. Every dot is an actual run, the orange dot is the winning config’s own setting, the vertical line spans min to max, and the dashed line marks the winning config’s value for that metric. The winner leads on Answer Correctness, the metric the selection composite weights most, but not on most others. The y-axes are independent per tile.

Caveat on Scope: A note on the scope of these results–and an optimistic one. The 432-run grid was exhaustive over the 7 axes we chose, but it was still just a quick-and-dirty first pass: only about 2 days of work by a single AI engineer, across 7 of the many dimensions worth exploring. These numbers are NOT the ceiling of what RapidFire AI can reach. Exploring more axes and dimensions, running many more configs (potentially even 1000s), and evaluating on the larger full dataset would push the eval metrics significantly higher. The headline is that even a two-day, evidence-driven pass with RapidFire AI already delivered these results, substantial headroom available for more.

Mutual Benefit: The relationship went both ways. Beyond the labeled data and domain expertise the customer’s team shared with us, their detailed feedback also fed straight back into our product, helping us improve RapidFire AI’s API and execution support for complex multimodal data ingestion and experimentation.

Engineered Convergence, Driven by Evidence

At first glance the two cases seem to give opposite advice: Case 1 says no need to change the generator LLM, while Case 2 changes it as part of the win. This is NOT a contradiction. They are two scopes of the same methodology, run against two different kinds of evidence. 

For Case 1, we pinned the LLM and embedder on purpose to isolate the value of retrieval engineering against one baseline on a controlled benchmark. For Case 2, we explored a larger space because the question there was which complete stack performed better on the customer’s expert-curated questions. Same engine, two scopes: a controlled benchmark in the first, customer-curated data in the second.

What unites them is the way the answer was reached.: In both cases the winning config was not the obvious one, and it surfaced only because RapidFire AI’s hyperparallelism enables doing a lot more with the same resources . The decision making moved from vibes on “this output looks good” to “this config wins, here is the evidence, and here are the knobs that drove it.”

The Takeaway

RapidFire AI turns a retrieval and generation stack from a sequential bottleneck into an engineered convergence process. This approach applies to any AI application team facing the same combinatorial space: sweep the configs in parallel, score them the same way, stop the losers early, and let evidence pick the winners. Neither partner in this study needed a bigger budget or a longer timeline: they needed a way to move from comparing just one config at a time to systematic high-throughput comparisons.

Another important takeaway here from the data platforms standpoint:   in neither case did RapidFire AI replace anything the customer had built. Case 1 ran on the data-and-analytics leader’s own stack, with their generator and embedder held fixed. Case 2 ran on Vector DBMS 1, reusing their indexes. That is a crucial point: RapidFire AI is complementary to the data platforms that customers already build on. It sits on top of them and amplifies their reach and impact, empowering the AI solutions teams who put reference architectures in front of customers, and turns a working demo or a solid production stack into a measurably better one. The platforms supply the foundation–RapidFire AI turns that foundation into  concrete, measurable, and repeatable wins for AI applications.

Ready to try it? Run the same kind of retrieval/context engineering sweeps on your own corpus:

• RapidFire AI on Colab, get started with RAG optimization: https://tinyurl.com/rapidfireai-rag-colab

• Code on GitHub: https://github.com/RapidFireAI/rapidfireai

• Documentation: https://oss-docs.rapidfire.ai/