This article explores how to boost the performance of small language models by using supervision from larger ones through knowledge distillation. The article provides a step-by-step guide on how to distill knowledge from a teacher model (LLama 2–70B) to a student model (Tiny-LLama) using unlabeled in-domain data and targeted prompting.
