SGG: Spacetime Gaussian Grouping for 4D object segmentation

Abstract Novel view synthesis of dynamic 3D scenes is an attractive but challenging problem. Many recent methods extend the 3D Gaussian Splatting model with temporal attributes in order to achieve high-quality and real-time rendered dynamic 3D scene representation. In order to have a more complete understanding of 4D scenes, we propose Spacetime Gaussian Grouping (SGG), for segmenting and tracking the objects spatially and temporally. The proposed method trains the Spacetime Gaussians (STG) model in conjunction with the multi-view consistent segmentation masks corresponding to the input images, which are used to label the 3D Gaussians with instance identities. Our model produces a lightweight, dynamic 3D representation of a scene, enabling users to select and edit specific objects in a 4D manner across various devices, including smartphones, PCs, VR systems, among other devices. Furthermore, a novel dynamic view synthesis segmentation benchmark is proposed to evaluate our method quantitatively.