Caracal: Causal Architecture via Spectral Mixing

The scalability of Large Language Models to long sequences is hindered by the quadratic cost of self-attention and the limitations of positional encodings. To address these, we introduce **Caracal**, a novel architecture that replaces self-attention with a parameter-efficient, $\mathcal{O}(L \log L)$ Multi-Head Fourier (MHF) module. Our contributions are threefold: (1) We leverage the Fast Fourier Transform (FFT) for sequence mixing, inherently addressing both bottlenecks mentioned above. (2) We apply a frequency-domain causal masking technique that enforces autoregressive capabilities via asymmetric padding and truncation, overcoming a critical barrier for Fourier-based generative models. (3) Unlike efficient models relying on hardware-specific implementations (e.g., Mamba), **Caracal** uses standard library operators. This ensures robust portability, eliminating common deployment barriers. Evaluations demonstrate that **Caracal** performs competitively with Transformer and SSM baselines, offering a scalable and simple pathway for efficient long-sequence modeling. Code is available in the supplementary materials.

Teaching AI to learn from long documents currently faces two major hurdles: traditional methods (like attention) are too slow and computationally expensive, while newer fast alternatives (like ssm) often require specialized, hard-to-install hardware. This limits the ability to build long-sequence AI to those with massive budgets or very specific technical setups. We developed Caracal, an architecture that replaces slow components with a mathematical shortcut called the Fourier Transform—the same standard tool used to process digital music. We designed a unique way to make this tool work for text generation, allowing the AI to process long sequences efficiently using only standard, widely available software. Caracal matches the intelligence of existing models while being both fast and highly portable. By removing the need for both massive computing power and niche hardware, it provides a practical pathway for any researcher to build AI capable of understanding vast amounts of information on almost any system.