Use case · Fine-tuning (SFT + LoRA)
Cybersecurity
Security answers have to use precise technical vocabulary, even when the question is asked in plain language, without the model forgetting what it already knows.
The experiment
Fine-tune a small instruction-tuned model for technically accurate security answers, using a 2×2 grid to trade off adaptation depth against learning aggressiveness.
- Dataset
- A cybersecurity question-and-answer set, with 50 rows held out for evaluation.
- Agent
- Qwen2.5-1.5B-Instruct fine-tuned with SFT and LoRA, varying LoRA target modules and learning-rate schedule.
- Knobs tested
- LoRA target modulesLearning-rate scheduleLoRA rank
What the experiment showed
0.8347
BERTScore for the best run, vs 0.7984 baseline
0.2715
ROUGE-L for the best run, vs 0.1839 baseline
4
configurations in a 2×2 grid
- 01Targeting all linear layers, not just the query and value projections, was the biggest lever.
- 02Both knobs moved in one direction: aggressive learning beat stable, and heavy adaptation beat light. The model still had room to absorb more domain signal.
- 03The most aggressive setup carries the most risk of forgetting general skills, so check it on a general benchmark such as MMLU.
- 04With 50 eval rows, treat differences as directional and review sample outputs by hand.
How RapidFire helped
Every configuration in one run
Compared the knobs side by side on the same hardware, instead of one run after another.
Live results
Scored each version on slices of the eval set as it ran, so decisions came early.
Weak runs stopped early
Cut directions that were not working and put the compute behind the ones that were.
Winners cloned
Cloned the strongest run and changed one factor at a time to find what drove the gain.
Apply it to your data
- 1Pick a domain dataset whose vocabulary differs from general pretraining.
- 2Start from a lightweight instruction-tuned model and sweep target modules, learning rate, and optionally rank.
- 3Pick winners on held-out evals plus human review.