Use case · RAG and retrieval
EdTech
Education assistants have to stay aligned to course materials, so students get answers, explanations, and practice that match what is actually taught, with citations.
The experiment
Tune a retrieval-first RAG system over a course catalog, so advising and study-support answers stay grounded in the right sections.
- Dataset
- The UCSB 2009–2010 General Catalog: a 490-page PDF split into 1,304 chunks of course descriptions, prerequisites, requirements, and policies.
- Agent
- An academic-advisor assistant that answers student questions, such as prerequisites, and cites the catalog sections it used.
- Knobs tested
- Chunk size (128 vs 256 tokens)Reranker top-n (2 vs 5)
What the experiment showed
+16.7%
F1 with 256-token chunks (0.434 vs 0.372)
+20.6%
MRR, so the right section shows up first
+21.8%
recall, with gains on precision and NDCG@5 too
- 01256-token chunks won on every retrieval metric.
- 02Larger chunks keep a whole course description, including title, prerequisites, and units, in one piece, so less gets fragmented.
- 03F1 balanced precision and recall, while NDCG@5 and MRR checked that the first result was relevant.
How RapidFire helped
Every configuration in one run
Compared the knobs side by side on the same hardware, instead of one run after another.
Live results
Scored each version on slices of the eval set as it ran, so decisions came early.
Weak runs stopped early
Cut directions that were not working and put the compute behind the ones that were.
Winners cloned
Cloned the strongest run and changed one factor at a time to find what drove the gain.
Apply it to your data
- 1Bring syllabi, lecture notes, textbooks, rubrics, or LMS pages.
- 2Test chunking and retrieval settings side by side, optimizing for F1 and MRR.
- 3Use the top sections to ground answers, practice questions, and study aids.