Can I read ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction on EtoBox?

ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction by Weder, Silvan; Engelmann, Francis; Schönberger, Johannes L.; Seki, Akihito; Pollefeys, Marc; Oswald, Martin R. is a scholarly article available to read on EtoBox.

What is ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction about?

We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality. To overcome the inherent challenges of online methods, we make two main contributions. First, to effectively extract information from the input RGB-D video stream, we jointly estimate geometry and semantic labels per frame in 3D. A key focus of our approach is to reason about semantic entities both in the 2D input and the local 3D domain to leverage differences in spatial context and network architectures. Our method predicts 2D features using an off-the-shelf segmentation network. The extracted 2D features are refined by a lightweight 3D network to enable reasoning about the local 3D structure. Second, to efficiently deal with an infinite stream of input RGB-D frames, a subsequent network serves as a temporal expert predicting the incremental scene updates by leveraging 2D, 3D, and past information in a learned manner. These updates are then integrated into a global scene representation. Using these main contributions, our

Author

Weder, Silvan; Engelmann, Francis; Schönberger, Johannes L.; Seki, Akihito; Pollefeys, Marc; Oswald, Martin R.

Published

2023

Language

EN