Syntax vs. Semantics: How Transformers Learn Deep Dependencies

Large Language Models demonstrate remarkable syntactic fluency, yet the optimization dynamics governing their acquisition of deep semantic dependencies remain poorly understood. We propose a mechanistic framework that models this learning process as a competition between Surface Statistics and Deep Semantics. Our theoretical analysis identifies a "Gradient Starvation" phenomenon where the error signals for sparse semantic dependencies are actively suppressed during early optimization. This suppression impedes the learning of structural reasoning and causes its emergence to manifest as a sudden phase transition. Furthermore, this framework offers a mechanistic basis for the effectiveness of Chain-of-Thought (CoT) strategies. By externalizing intermediate reasoning steps into concrete tokens, CoT effectively bypasses the suppression regime inherent to implicit reasoning. We validate these findings across scales ranging from toy transformers to production models (Llama-3.1-8B, Qwen2.5-Coder-7B). Finally, guided by this theory, we propose a topology-aligned contrastive objective that explicitly rectifies the gradient geometry. Experiments on variable binding tasks demonstrate that our method achieves an improvement that is over 2× larger than that obtained via standard cross-entropy fine-tuning. Code will be publicly available at: https://github.com/jr-zhao/Deep-Dependencies/tree/main.

Large language models can write fluent text, but they often struggle to learn deeper reasoning and long-range relationships. We study why this happens and find that, during training, models tend to focus first on easy surface patterns such as syntax and formatting, while more meaningful dependency structures are learned much later. We call this effect “gradient starvation.” Using both mathematical analysis and experiments on models ranging from small transformers to modern systems like Llama and Qwen, we show that reasoning-related abilities often emerge suddenly through a sharp transition during training. We also find that Chain-of-Thought prompting may help because it makes intermediate reasoning steps explicit, reducing the dominance of superficial patterns. Based on these findings, we design a new training method that encourages models to focus more directly on meaningful dependency structure. Our approach improves performance on both code understanding and natural-language reasoning tasks, suggesting a possible path toward language models with stronger reasoning abilities.