Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting

While 3D Gaussian Splatting (3DGS) has demonstrated impressive real-time rendering performance, its efficacy remains constrained by a reliance on heuristic density control. Despite numerous refinements to these handcrafted rules, such methods inherently lack the flexibility to adapt to diverse scenes with complex geometries. In this paper, we propose a paradigm shift for density control from rigid heuristics to fully learnable policies. Specifically, we introduce LeGS, a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL). Central to our approach is the tailored effective reward function grounded in sensitivity analysis, which precisely quantifies the marginal contribution of individual Gaussians to reconstruction quality. To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from $O(N^2)$ to $O(N)$. Extensive experiments on the Mip-NeRF 360, Tanks \& Temples, and Deep Blending datasets demonstrate that LeGS significantly outperforms state-of-the-art methods, striking a superior balance between reconstruction quality and efficiency.

3D Gaussian Splatting (3DGS) is a powerful technique for reconstructing and rendering 3D scenes in real time, but it still relies heavily on hand-designed rules to decide how many Gaussian points should be used and where they should be placed. These fixed rules often work well in some scenes, but they can struggle to adapt to complex shapes and diverse real-world environments. We propose $\textbf{LeGS}$, a new framework that replaces these handcrafted density-control rules with a learnable policy trained using reinforcement learning. Instead of manually specifying when to add, remove, or adjust Gaussians, LeGS learns these decisions automatically from data. A key part of our method is a specially designed reward function that measures how much each Gaussian contributes to the final reconstruction quality. We also derive an efficient closed-form computation that reduces the cost of this reward calculation from $O(N^2)$ to $O(N)$, making the approach practical for large scenes. Experiments on several standard 3D reconstruction benchmarks show that LeGS produces higher-quality reconstructions while maintaining strong rendering efficiency. This suggests that learnable density control can make 3D Gaussian Splatting more flexible, robust, and effective for real-world 3D scene modeling.