Use case · Fine-tuning (SFT)
Customer support
Support AI has to be accurate to your policies, helpful, safe, and consistent at volume all at once. Generic models either invent answers or become rigidly unhelpful.
The experiment
Fine-tune a lightweight chat model under limited compute, then converge on the best training configuration for support Q&A.
- Dataset
- Bitext customer support instructions with reference responses, subsampled to fit a Colab runtime.
- Agent
- A fine-tuned chat model used as a support Q&A assistant, tuned for fluent, well-calibrated answers.
- Knobs tested
- Training formatLoRA capacityLearning rate
What the experiment showed
- 01Matching the training format to how the model naturally talks improved quality.
- 02Adaptation worked best with enough capacity, and optimization tuned to converge quickly without instability.
- 03Overlap metrics like ROUGE can mislead for support, so winners were picked on generalization and response quality.
How RapidFire helped
Every configuration in one run
Compared the knobs side by side on the same hardware, instead of one run after another.
Live results
Scored each version on slices of the eval set as it ran, so decisions came early.
Weak runs stopped early
Cut directions that were not working and put the compute behind the ones that were.
Winners cloned
Cloned the strongest run and changed one factor at a time to find what drove the gain.
Apply it to your data
- 1Turn ticket history, macros, and help-center articles into instruction and response pairs.
- 2Sweep a small set of knobs (format, LoRA capacity, learning rate) to converge quickly.
- 3Pick winners on loss plus human spot checks, not overlap metrics alone.