Towards Resource-Efficient LLMs: End-to-End Energy Accounting of Distillation Pipelines

The rise in deployment of large language models has driven a surge in GPU demand and datacenter scaling, raising concerns about electricity use, grid stress, and the impacts of modern AI workloads. Distillation is often promoted as one of the most effective paths to obtain cheaper, more efficient models, yet these claims rarely account for the full end-to-end energy and resource costs, including crucial teacher-side workloads such as data generation, logit caching, and evaluation. We present a comprehensive energy accounting framework that measures the complete computational cost of distillation pipelines via detailed stage-wise tracking of GPU device power consumption. In our experiments, we separate and log empirical energy use across distinct phases and systematically measure the energy and emissions of two common distillation methods: the classic logit-based knowledge distillation and synthetic-data supervised fine-tuning, constructing energy–quality–throughput Pareto frontiers that expose the previously ignored costs. From these measurements and analyses, we derive practical design rules for selecting distillation methods and hyperparameters under energy and budget constraints, and release an open-source measurement harness and accounting protocol to provide a standardized foundation for comparable, reproducible distillation research, explicitly accountable for complete pipeline energy impact.

Training AI systems like chatbots and language assistants is very expensive and require enormous amounts of money, time, and electricity. One popular strategy for reducing costs is distillation: training a smaller AI model (the "student") to mimic a bigger, more capable one (the "teacher"). The smaller models often end up similar in quality to the bigger model, and supposedly at a fraction of the cost, as they take cheaper to run and serve to millions of users. This has made distillation widely regarded as the greener, more efficient path. But that assumption ignores a hidden cost. Distillation requires the large teacher model to do substantial work upfront: generating text, training data or computing detailed outputs for every example, before the small student model can even begin training, which is often overlooked. When we measure the full pipeline from start to finish, distillation often consumes more total energy than the standard procedure of fine-tuning the small model directly on data. We built a measurement framework tracking electricity use at each stage across model sizes from 1 to 13 billion parameters. The findings show the teacher's upfront cost acts like a fixed tax that only pays off when its outputs are reused across multiple runs, something that is rarely done in practice do. Our results reframe distillation as an inherently efficient technique, and we release an open measurement harness so others can compute honest energy–quality tradeoffs for their own systems.