Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image


Abstract

We propose a computational framework to jointly parse a single RGB image and reconstruct a holistic 3D configuration composed by a set of CAD models using a stochastic grammar model. Specifically, we introduce a Holistc Scene Grammar (HSG) to represent the 3D scene structure, which characterizes a joint distribution over the functional and geometric space of indoor scenes. The proposed HSG captures three essential and often latent dimensions of the indoor scenes: i) latent human context, describing the affordance and the functionality of a room arrangement, ii) geometric constraints over the scene configurations, and iii) physical constraints that guarantee physically plausible parsing and reconstruction. We solve this joint parsing and reconstruction problem in an analysis-by-synthesis fashion, seeking to minimize the differences between the input image and the rendered images generated by our 3D representation, over the space of depth, surface normal, and object segmentation map. The optimal configuration, represented by a parse graph, is inferred using Markov chain Monte Carlo, which efficiently traverses through the non-differentiable solution space, jointly optimizing object localization, 3D layout, and hidden human context. Experimental results demonstrate that the proposed algorithm improves the generalization ability and significantly outperforms prior methods on 3D layout estimation, 3D object detection, and holistic scene understanding.

Paper

Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, Song-Chun Zhu
European Conference on Computer Vision (ECCV) 2018
Paper / Supplementary / Code / Poster / Bibtex

Team

Siyuan Huang1,2

Siyuan Qi1,2

Yixin Zhu1,2

Yolanda1

Yuanlu Xu1,2

Song-Chun Zhu1,2

1 Center for Vision, Cognition, Learning and Autonomy
University of California, Los Angeles

2 International Center for AI and Robot Autonomy
(CARA)

Parsing & Reconstruction

Given a indoor image, we could reconstruct a full 3D scene composed by CAD models and render it from different viewpoints.

Your browser does not support the video tag.

Your browser does not support the video tag.

Your browser does not support the video tag.
Human Imagination

After infering the 3D room layout and objects, we could sample possible activity groups and human skeletons in the scene.

Code

View on Github
Bibtex

@inProceedings{huang2018holistic,
title={Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image},
author = {Huang, Siyuan and Qi, Siyuan and Zhu, Yixin and Xiao, Yinxue and Xu, Yuanlu and Zhu, Song-Chun},
booktitle={ECCV},
year={2018}
}

More Results

Below shows more results we generated.