CLAWS: Calibration-Aware Activation Sparsity for Instruction-Tuned LLMs

Modern LLMs contain gated MLPs as a central component of each transformer block: the output of a gate projection multiplies elementwise with an up projection before a down projection produces the block output. Their per-token activations are highly skewed, creating a natural axis for dynamic top-K sparsification. Practical sparsity requires both accurate routing and a mask that the inference kernel can execute efficiently. CATS provides the hardware-friendly row-wise mask but routes using only local gate information. LaRoSA improves the routing basis through calibration, but induces a column-sparse access pattern that is poorly aligned with quantized row-major K-blocked weight layouts. We introduce CLAWS, a Fisher-informed calibration method for gated-MLP sparsification that keeps the CATS execution pattern but replaces its local gate-only router with a calibrated per-neuron importance score. The method multiplies the runtime gate score by a static Fisher-style constant estimated from calibration trajectories, preserving a row-sparse top-K mask that maps directly to quantized kernels. We also analyze why analogous static Fisher multipliers have little leverage in LaRoSA-style rotated-input sparsity: the rotated-input score is much broader than its modifier, while GELU compresses the native gate score so the CLAWS modifier can meaningfully move rankings. On Gemma-4-E2B-IT at 50% FFN density, CLAWS recovers dense-level Avg5 while surpassing matched-recipe CATS+LoRA by 6.8pp MMLU (49.6 vs. 42.8), and a custom ARM kernel delivers a 1.24x wall-clock speedup on the full MLP block.