Fine-Tuning with Minimal Data
Techniques for achieving high-quality domain adaptation with as few as 500 labeled examples using curriculum learning.
Fine-tuning LLMs typically requires thousands of labeled examples. Many enterprises have valuable but small datasets, 200 medical notes, 500 legal clauses, 300 support tickets. Can we achieve production-quality domain adaptation with minimal data?
We explored curriculum learning: training on easier examples first (high-confidence, prototypical) before harder ones (edge cases, ambiguous). We combined this with selective layer freezing, only unfreezing the last 2–4 layers initially, to reduce overfitting.
We also tested data augmentation: paraphrasing, back-translation, and synthetic generation with a teacher model. The right mix of augmentation plus curriculum extended effective dataset size by 3x without collecting new labels.
On 8 domain-specific benchmarks (medical, legal, finance, support), we achieved 90%+ of the accuracy of full-data fine-tuning with only 500 examples when using curriculum + selective unfreezing + augmentation. We release the training recipes.
Key Findings
- 1Curriculum learning (easy-to-hard) improves few-shot fine-tuning by 15–25% vs. random ordering.
- 2Selective layer unfreezing reduces overfitting; we recommend unfreezing top 2–4 layers for <1K examples.
- 3Synthetic augmentation with teacher models can extend 500 examples to effective 1.5K with minimal quality loss.
Related publications.
The Architecture of Reason
An exploration of reasoning capabilities in modern language models and architectural patterns that enable multi-step logical inference.
Quantum-Classical Hybrid AI
Investigating the intersection of quantum computing and classical neural networks for combinatorial optimization problems.
Interested in collaborating?
We're always looking for research partners and academic collaborators who want to push AI forward.