How Small Can a Tandem Speech Front-End Be? Diagnosing Front-End Capacity with Layer Removal

Recent tandem speech-to-speech (S2S) dialogue systems delegate knowledge and reasoning to a back-end LLM, leaving a front-end S2S transformer to handle low-latency spoken interaction. This raises a practical capacity question: how small can the front-end become, what performance is preserved, and how does its behavior change as layers are removed? To diagnose this, we randomly remove different numbers of layers from a KAME-style front-end transformer, fine-tune each variant on the same data, and evaluate on speech MT-Bench. The results show that moderate removal largely preserves performance, and even aggressive removal remains above the S2S-only Moshi baseline, although it no longer matches the full front-end. We then analyze individual dialogue turns to identify what fails. First, in low-scoring aggressive-removal cases, the model typically does not fall silent; it keeps answering, but the answers become long, incorrect, or unstable. Second, models with similar aggregate scores can still differ in performance at the level of individual dialogue turns. These diagnostic findings show why future front-end reduction should preserve both aggregate quality and dialogue-level behavior.