Siyuan Huang - Statistics, UCLA

I am a fourth year Ph.D. student in the Department of Statistics at University of California, Los Angeles. I am currently doing computer vision research at the Center for Vision, Cognition, Learning, and Autonomy (VCLA) and am advised by Professor Song-Chun Zhu. Before coming to UCLA, I graduated from Tsinghua University with a Bachelors in Department of Automation.

My research interests lie in computer vision, machine learning, cognition, and robotics.
I currently focus on the problem of 3D scene understanding which contains 3D scene parsing, reconstruction, and synthesis. I would like to develop tools to help machines learn 3D representations, percept 3D world, and interact with 3D environments from images or videos. My long-term goal is to build an intelligent machine that could think and behave like a human. E-Mail / CV / Google Scholar

News

05/2020 NEW Our CVPR worshop will be totally virtual. Please check the scene understanding workshop for more details.
03/2020 NEW I am co-organizing the 2nd CVPR 2020 workshop: 3D Scene Understanding for Vision, Graphics and Robotics.
02/2020 NEW I will intern at DeepMind, London during the summer.
07/2019 One paper on 3D object detection is accepted by NeurIPS 2019.
05/2020 Two papers accepted by ICCV 2019. Check our paper on holistic++ scene understanding ( live demo for 3D human pose estimation) and human gaze communication.
03/2019 I am co-organizing the CVPR 2019 workshop: 3D Scene Understanding for Vision, Graphics and Robotics.
09/2018 One Paper accepted by NeurIPS 2018. Code is released.
07/2018 One Paper accepted by ECCV 2018.
06/2018 One Paper accepted by IJCV 2018.
02/2018 One Paper accepted by CVPR 2018.
04/2017: One paper accepted by ICCV 2017.
09/2016: Started my Ph.D. at UCLA

Publications

Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
Yixin Zhu , Tao Gao , Lifeng Fan , Siyuan Huang, Edmonds Mark, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Josh Tenenbaum, Song-Chun Zhu
Engineering 2020
Paper
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.

Learning V1 simple cells with vector representations of local contents and matrix representations of local motions
Ruiqi Gao , Jianwen Xie , Siyuan Huang, Yufan Ren, Song-Chun Zhu, Ying Nian Wu
Arxiv 2020
Paper
we propose a representational model that couples the vector representations of local image contents with the matrix representations of local pixel displacements. When the image changes from one time frame to the next due to pixel displacements, the vector at each pixel is rotated by a matrix that represents the displacement of this pixel.

A Generalized Earley Parser for Human Activity Parsing and Prediction
Siyuan Qi , Baoxiong Jia , Siyuan Huang, Ping Wei, Song-Chun Zhu
TPAMI 2020
Paper
Propose an algorithm to tackle the task of understanding complex human activities from (partially observed) videos from two important aspects: activity recognition and prediction.

PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Siyuan Huang, Yixin Chen, Tao Yuan, Siyuan Qi, Yixin Zhu, Song-Chun Zhu
Neural Information Processing Systems (NeurIPS) 2019
Paper
To solve the problem of 3D object detection, we propose perspective points as a novel intermediate representation, defined as the 2D projections of locally-Manhattan 3D keypoints to locate an object, and they satisfy certain geometric constraints caused by the perspective projection.

Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
Yixin Chen *, Siyuan Huang *, Tao Yuan, Siyuan Qi, Yixin Zhu, Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2019
* Equal contributions
Paper / Supplementary / Project
Propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction- and (ii) 3D human pose estimation. We incorporate the human-object interaction (HOI) and physical commonsense to tackle this problem.

Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
Lifeng Fan *, Wenguan Wang *, Siyuan Huang, Xinyu Tang, Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2019
* Equal contributions
Paper
Propose a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions.

Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation
Siyuan Huang, Siyuan Qi, Yinxue Xiao, Yixin Zhu, Ying Nian Wu, Song-Chun Zhu
Neural Information Processing Systems (NeurIPS) 2018
Paper / Supplementary / Poster / Video / Code / Project
Propose an end-to-end model that simultaneously solves tasks of 3D object detection, 3D layout estimation and camera pose estimation in real-time given only a single RGB image

Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image
Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, Song-Chun Zhu
European Conference on Computer Vision (ECCV) 2018
Paper / Supplementary / Project / Code / Poster / Bibtex
Propose a computational framework to parse and reconstruct the 3D configuration of an indoor scene from a single RGB image in an analysis-by-synthesis fasion using a stochastic grammar model.

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars

Chenfanfu Jiang *, Siyuan Qi *, Yixin Zhu *, Siyuan Huang *, Jenny Lin, Xingwen Guo, Lap-Fai Yu, Demetri Terzopoulos, Song-Chun Zhu

* Equal contributions
Internatianal Journal of Computer Vision (IJCV) 2018
Paper / Demo
Employ physics-based rendering to synthesize photorealistic RGB images while automatically synthesizing detailed,per-pixel ground truth data, including visible surface depth and normal, object identity and material information, as well as illumination.

Human-centric Indoor Scene Synthesis using Stochastic Grammar
Siyuan Qi, Yixin Zhu , Siyuan Huang, Chenfanfu Jiang , Song-Chun Zhu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018
Paper / Project / Code / Bibtex
Present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, for the purpose of obtaining large-scale 2D/3D image data with the perfect per-pixel ground truth.

Predicting Human Activities Using Stochastic Grammar
Siyuan Qi, Siyuan Huang, Ping Wei, Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2017
Paper / Bibtex / Code
Use a stochastic grammar model to capture the compositional structure of events, integrating human actions, objects, and their affordances for modeling the rich context between human and environment.

Nonlinear Local Metric Learning for Person Re-identification
Siyuan Huang, Jiwen Lu, Jie Zhou, Anil K. Jain
arXiv 2015
arXiv Paper
Utilize the merits of both local metric learning and deep neural network to exploit the complex nonlinear transformations in the feature space of person re-identification data.

Building Change Detection Based on 3D reconstruction
Baohua Chen, Lei Deng, Yueqi Duan, Siyuan Huang, Jie Zhou
IEEE International Conference on Image Processing (ICIP) 2015
Paper / Bibtex
Propose a change detection framework based on RGB-D map generated by 3D reconstruction which can overcome the large illumination changes .