Use case · RAG and retrieval
Scientific research
Research Q&A needs high-precision retrieval. Missing the one critical passage breaks the answer, and every claim has to trace back to its source paper.
The experiment
Tune retrieval in phases, from chunking to embeddings to hybrid retrieval, and track how often the best supporting passage comes back first.
- Dataset
- QASPER: 1,585 NLP research papers with 5,049 expert-annotated questions.
- Agent
- A retrieval-first RAG workflow that finds the single most important supporting chunk, plus a short shortlist, so answers stay grounded.
- Knobs tested
- ChunkingEmbedding modelHybrid retrieval
What the experiment showed
- 01Mean Reciprocal Rank was the guide metric, because surfacing the best passage first is what keeps answers correct.
- 02Locking in a winner at each phase before moving on kept the search manageable.
- 03Getting retrieval right comes before optimizing generation.
How RapidFire helped
Every configuration in one run
Compared the knobs side by side on the same hardware, instead of one run after another.
Live results
Scored each version on slices of the eval set as it ran, so decisions came early.
Weak runs stopped early
Cut directions that were not working and put the compute behind the ones that were.
Winners cloned
Cloned the strongest run and changed one factor at a time to find what drove the gain.
Apply it to your data
- 1Bring papers, protocols, lab notes, or internal wikis.
- 2Test chunking, embedding, and retrieval setups side by side.
- 3Lock in winners phase by phase, then tune generation.