Use case · Fine-tuning (SFT + LoRA)
Privacy and compliance
Sensitive data has to be found and masked reliably, without breaking the structure or readability of the document around it.
The experiment
Fine-tune GPT-2 to detect and redact names, emails, phone numbers, and addresses, then run a full-factorial grid across prompt design, model capacity, and learning rate.
- Dataset
- ai4privacy open PII masking dataset, with 64 training and 10 eval examples.
- Agent
- GPT-2 (124M) fine-tuned with SFT and LoRA, sweeping 8 configurations in a 2×2×2 grid.
- Knobs tested
- Prompt scheme (minimal vs one-shot)LoRA rank (8 vs 32)Learning rate (2e-4 vs 5e-4)
What the experiment showed
8
configurations, run in about 7 minutes on free Colab
22.7%
lower eval loss with LoRA rank 32 than rank 8
15.2%
lower eval loss with a one-shot example in the prompt
- 01LoRA rank was the biggest lever: higher capacity captured the variety of PII patterns.
- 02A single worked example in the prompt taught the PII-to-mask mapping.
- 03The faster learning rate won on every configuration with this little data.
- 04The best setup reached 77.1% token accuracy, but exact match was 0%, so human review stays essential.
How RapidFire helped
Every configuration in one run
Compared the knobs side by side on the same hardware, instead of one run after another.
Live results
Scored each version on slices of the eval set as it ran, so decisions came early.
Weak runs stopped early
Cut directions that were not working and put the compute behind the ones that were.
Winners cloned
Cloned the strongest run and changed one factor at a time to find what drove the gain.
Apply it to your data
- 1Structure data as pairs: source with PII, then masked output. Works for support logs, legal documents, and medical records.
- 2Include at least one example in your prompt.
- 3Start with a higher LoRA rank and sweep learning rate.
- 4Scale up training data for production to cover nested PII and edge cases.