Open-Set Semantic Gaussian Splatting SLAM with Expandable Representation

Abstract: This work enables everyday devices, e.g., smartphones, to dynamically capture open-ended 3D scenes with rich, expandable semantics for immersive virtual worlds. While 3DGS and foundation models hold promise for semantic scene understanding, existing solutions suffer from unscalable semantic integration, prohibitive memory costs, and cross-view inconsistency. To respond, we propose Open-Set Semantic Gaussian Splatting SLAM, a GS-SLAM system augmented by an expandable semantic feature pool that decouples condensed scene-level semantics from individual 3D Gaussians. Each Gaussian references semantics via a lightweight indexing vector, reducing memory overhead by orders of magnitude while supporting dynamic updates. Besides, we introduce a consistency-aware optimization strategy alongside a Semantic Stability Guidance mechanism to enhance long-term, cross-view semantic consistency and resolve ambiguities. Experiments demonstrate that our system achieves high-fidelity rendering with scalable, open-set semantics across both controlled and in-the-wild envi-ronments, supporting applications like 3D localization and scene editing. These results mark an initial yet solid step towards high-quality, expressive, and accessible 3D virtual world modeling.

Framework Overview. We enhance existing 3DGS-based SLAM with an expandable semantic representation, using an expandable semantic feature pool for growing semantics and assigning each Gaussianan aggregated semantic from the pool. Moreover, to improve consistency and reduce ambiguity, we introduce Intra-Inter Semantic Consistency Objectives and Semantic Stability Guidance.

Rendering Results on Replica

Reconstruction Results on Replica