Layout and Fusion Trade-offs for Mixture-of-Experts Inference under Single-Node Tensor Parallelism

Mixture-of-Experts (MoE) LLMs are increasingly deployed on a single multi-GPU node with tensor parallelism, a regime that is common in private and on-premise serving but has received less attention than datacenter-scale expert parallelism. This paper argues that this setting is not simply a smaller version of expert-parallel inference. The expert-major pipeline inherited from expert-parallel backends reshapes the bottlenecks of an MoE feed-forward layer under tensor parallelism: turning the pre-permute to an expert-major layout into a dominant cost at large token counts, making down-projection grouped GEMMs unusually thin, and creating a reduction chain between top-k combine and the tensor-parallel AllReduce. Through controlled kernel-level experiments, we find that the best layout flips with workload shape. Token-major variants win in decode-like regimes, while in-flight-permute variants with persistent down projection win in prefill-like regimes. Across representative MoE shapes, the best variant improves MoE FFN latency by up to nearly \textbf{2×} over the conventional expert-major baseline (pre-permute + expert-major up/down). Recent serving systems separate prefill and decode at the request and batch level. Our findings suggest that this separation should extend down to the MoE kernel layout itself, calling not for a single fused MoE kernel but for a workload-conditioned layout policy.