LLM Family Expansion via Distillation and Quantization

The wide adoption of LLMs has lead to their use in great variety of applications and scenarios, such as chatbot assistants and data annotation, creating the need for the models to satisfy certain budget and hardware constraints. This has led to the trend of LLMs being released in batches consisting of similar models of various sizes for the family of models to adhere to as wide of a range of constraints as possible. In this paper, we validate distillation and quantization as a cost-effective way to expand model families to new sizes and hardware formats. From the open-recipe Apertus 8B LLM, we pre-train a family of models of up to 4B parameters on 1.7T permissive license tokens, which we call Levis. We demonstrate cost-efficiency and strong accuracy performance of our approach for covering large ranges of hardware and systems requirements.