An Empirical Study of Proactive Coding Assistants in Real-World Software Development

Large language model (LLM)-based coding assistants have made strong progress in software engineering tasks. However, most existing systems are reactive and require explicit developer instructions. Proactive coding assistants predict users' latent intent from IDE interaction traces and repository-level code context. This can reduce the cognitive overhead of writing instructions and improve development efficiency. However, proactive assistance requires real-world developer behavior data, while existing studies mainly rely on LLM-generated simulated traces. The fidelity of such simulated traces remains unclear. To study this simulation-to-reality gap, we build a large-scale user behavior trace dataset. It contains real-world IDE interaction traces collected from 1{,}246 experienced industry developers over three consecutive days. Our analysis shows substantial differences in behavioral diversity, temporal structure, and exploratory behavior. Based on the real-world traces, we build ProCodeBench, a benchmark for proactive intent prediction in realistic development scenarios. Experiments show that representative LLM, retrieval-augmented, and agent-based methods remain far from reliable. They also suggest that simulation-based evaluation may overestimate real-world performance. Finally, we show that simulated data cannot replace real-world data, but it can improve performance before real-world fine-tuning.