S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal and incremental framework that builds a 3D Gaussian semantic field from image streams without accessing future frames or reprocessing historical observations. S2GS continuously updates scene geometry, appearance, and instance-level semantics through a geometry--semantic decoupled dual-backbone design. The geometry branch performs causal modeling for incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict masks and identity embeddings. Query-level contrastive alignment and lightweight online association with an instance memory further stabilize temporal identities. Experiments show that S2GS matches or outperforms strong offline baselines, while substantially improving long-horizon scalability: it processes over 1,000 frames with much slower runtime and GPU memory growth, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting. Project Page: https://stdcoutzrh.github.io/projects/s2gs/.

AI systems that see the world through cameras need to build 3D scenes while also understanding objects in them. Existing methods often become slow and memory-heavy when the image stream is long, because they repeatedly process many past images. We propose S2GS, a method that updates a 3D scene step by step as new images arrive. It does not need to revisit old frames, but still maintains the scene’s shape, appearance, object categories, and object identities over time. Experiments show that S2GS achieves strong reconstruction and understanding performance while scaling much better to long image streams than previous methods. This makes it more practical for online applications such as robotics, augmented reality, and real-time scene understanding.