GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video Diffusion Priors

Reconstructing 3D scenes using 3D Gaussian Splatting (3DGS) from sparse views is an ill-posed problem due to insufficient information, often resulting in noticeable artifacts. While recent approaches have sought to leverage generative priors to complete information for under-constrained regions, they struggle to generate content that remains consistent with input observations. To address this challenge, we propose GSFixer, a novel framework designed to improve the quality of 3DGS representations reconstructed from sparse inputs. The core of our approach is the reference-guided video restoration model, built upon a DiT-based video diffusion model trained on paired artifact 3DGS renders and clean frames with additional reference-based conditions. Considering the input sparse views as references, our model integrates both 2D semantic and 3D geometric features of reference views extracted from the visual geometry foundation model, enhancing the semantic coherence and 3D consistency when fixing artifact novel views. Furthermore, we introduce a reference-guided trajectory sampling strategy that ensures both angular coverage and view quality, further enhancing reconstruction fidelity. Considering the lack of suitable benchmarks for 3DGS artifact restoration evaluation, we present DL3DV-Res which contains artifact frames rendered using low-quality 3DGS. Extensive experiments demonstrate our GSFixer outperforms current state-of-the-art methods in 3DGS artifact restoration and sparse-view 3D reconstruction. Project page: https://github.com/GVCLab/GSFixer.

Creating complete 3D virtual scenes from only a small collection of photos is incredibly difficult because missing visual details often lead to blurry spots, floating artifacts, or glitchy distortions. While current generative methods can try to fill in these gaps, they struggle to make the newly generated parts look realistic and consistent with the original photos. We introduced GSFixer, a framework that acts like a reliabe digital restoration tool. By utilizing a specialized video diffusion model that carefully cross-references the geometric and semantic of the input imgaes, it cleanly repairs visual defects while ensuring that everything remains perfectly aligned and coherent from every angle. This technology enables the generation of high-quality, pristine 3D spaces using minimal camera input. It can drastically simplify and accelerate how we build immersive virtual reality environments, develop detailed video game worlds, or create digital maps from just a few standard phone pictures.