Do Transformers Need Three Projections? Systematic Study of QKV Variants

Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual contribution of these three projections and the impact of omitting some remain poorly understood. We systematically evaluate three projection sharing constraints: a) Q-K=V (shared key-value), b) Q=K-V (shared query-key), and c) Q=K=V (single projection). The last two variants produce symmetric attention maps; to address this, we also explore asymmetric attention via 2D positional encodings. Through experiments spanning synthetic tasks, vision (MNIST, CIFAR, TinyImageNet, anomaly), and language modeling (300M and 1.2B parameter models on 10B tokens), we discovered that our transformers perform on par or occasionally better than the QKV transformer. In language modeling, Q-K=V projection sharing achieves 50\% KV cache reduction with only 3.1% perplexity degradation. Crucially, projection sharing is complementary to head sharing (GQA/MQA): combining Q-K=V with GQA-4 yields 87.5% cache reduction, while Q-K=V + MQA achieves 96.9%—enabling practical on-device inference. We further show that Q--K=V preserves quality because keys and values can share representational space, whereas Q=K-V breaks attention directionality. Our results establish projection sharing as a new optimization axis for memory-efficient transformers, especially for edge deployment.

Modern AI systems like chatbots and image recognizers are built on a technology called the transformer, which processes information by letting each piece of data "pay attention" to every other piece. At the heart of this process are three components — queries, keys, and values — that together determine where to look and what to retrieve, each requiring dedicated memory and computation. We found that two of these components, the key and the value, can safely share the same underlying representation without meaningfully hurting performance, a finding we validated across image recognition, anomaly detection, and large language models trained on billions of words. The practical benefit is substantial: modern AI models are increasingly bottlenecked by the memory required to store intermediate computations during text generation, and merging the key and value components cuts this cost in half — which, when stacked with other existing compression techniques, can reduce it by up to 97% with only modest quality loss, making powerful AI models significantly more practical to run on smartphones and other low-power devices.