QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs

Large language models (LLMs) demand substantial computational and memory resources, posing challenges for efficient deployment. Two complementary approaches have emerged to address these issues: token-adaptive layer execution, which reduces floating-point operations (FLOPs) by selectively bypassing layers, and quantization, which lowers memory footprint by reducing weight precision. However, naively integrating these techniques leads to additional accuracy degradation due to reduced redundancy in token-adaptive models. We propose QTALE (Quantization-Robust Token-Adaptive Layer Execution for LLMs), a novel framework that enables seamless integration of token-adaptive execution with quantization while preserving accuracy. Conventional token-adaptive methods reduce redundancy in two ways: (1) by limiting the diversity of training paths explored during fine-tuning, and (2) by lowering the number of parameters actively involved in inference. To overcome these limitations, QTALE introduces two key components: (1) a training strategy that ensures diverse execution paths are actively explored during fine-tuning, and (2) a post-training mechanism that allows flexible adjustment of the execution ratio at inference to reintroduce redundancy when needed. Experimental results show that QTALE enables seamless integration of token-adaptive layer execution with quantization, showing no noticeable accuracy difference, with the gap to quantization-only models kept below 0.5\% on CommonsenseQA benchmarks. By combining token-adaptive execution for FLOPs reduction and quantization for memory savings, QTALE provides an effective solution for efficient LLM deployment.

Large language models (LLMs) are powerful AI systems, but they consume enormous amounts of memory and computing power, making them expensive and difficult to run on everyday devices. Researchers have developed two separate ways to cut these costs: quantization, which compresses the model into a smaller memory footprint, and layer skipping, which avoids unnecessary computation for each word the model processes. However, using both techniques together causes accuracy to drop more than either one does alone — and nobody had a good solution for this. We found the root cause: layer skipping quietly strips away the model's built-in resilience, leaving it fragile and unable to absorb the small numerical errors that quantization introduces. To fix this, we designed QTALE, which does two things. First, during training, it nudges the model to explore many different computation paths rather than settling into one rigid pattern — much like a student who studies a variety of practice problems instead of memorizing a single solution. Second, at deployment time, a single easy-to-tune dial lets users restore just enough computation to keep accuracy stable, without retraining. On standard benchmarks, QTALE cuts computing cost by roughly half and memory use by roughly 68%, while keeping accuracy within 0.5% of the best available baseline. This makes it realistic to run capable AI models on consumer-level hardware, broadening access to these tools.