News
- 08/2020 NEW Two paper about neural-symbolic learning and math word problems are accepted by AAAI 2021.
- 08/2020 NEW LEMMA dataset and code released.
- 07/2020 NEW Our paper won the best paper award in ICML2020 Workshop on Bridge Between Perception and Reasoning: Graph Neural Networks & Beyond.
- 07/2020 NEW Two papers accepted by ECCV 2020, one of them as oral presentation.
- 05/2020 NEW One paper about neural symbolic learning accepted by ICML 2020. Check the project and for more details including the code.
- 05/2020 Our CVPR workshop will be totally virtual. Please check the scene understanding workshop for more details.
- 03/2020 I am co-organizing the 2nd CVPR 2020 workshop: 3D Scene Understanding for Vision, Graphics and Robotics.
- 02/2020 I will intern at DeepMind, London during the summer.
- 07/2019 One paper on 3D object detection is accepted by NeurIPS 2019.
- 05/2020 Two papers accepted by ICCV 2019. Check our paper on holistic++ scene understanding ( live demo for 3D human pose estimation) and human gaze communication.
- 03/2019 I am co-organizing the CVPR 2019 workshop: 3D Scene Understanding for Vision, Graphics and Robotics.
- 09/2018 One Paper accepted by NeurIPS 2018. Code is released.
- 07/2018 One Paper accepted by ECCV 2018.
- 06/2018 One Paper accepted by IJCV 2018.
- 02/2018 One Paper accepted by CVPR 2018.
- 04/2017: One paper accepted by ICCV 2017.
- 09/2016: Started my Ph.D. at UCLA
Publications
|
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
Qing Li ,
Siyuan Huang,
Yining Hong,
Song-Chun Zhu
ECCV 2020 (Oral)
Paper
We design a neural-symbolic concept learner for learning the visual concepts and a multi-dimensional Item Response Theory (mIRT) model for guiding the visual concept learning process with an adaptive curriculum.
|
|
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
Baoxiong Jia,
Yixin Chen,
Siyuan Huang,
Yixin Zhu,
Song-Chun Zhu
ECCV 2020
Paper /
Code /
Project /
Bibtex
We introduce the LEMMA dataset to provide a single home to address these missing dimensions with carefully designed settings, wherein the numbers of tasks and agents vary to highlight different learning objectives. We densely annotate the atomic-actions with human-object interactions to provide ground-truth of the compositionality, scheduling, and assignment of daily activities.
|
|
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
Yixin Zhu ,
Tao Gao ,
Lifeng Fan ,
Siyuan Huang,
Edmonds Mark,
Hangxin Liu,
Feng Gao,
Chi Zhang,
Siyuan Qi,
Ying Nian Wu,
Josh Tenenbaum,
Song-Chun Zhu
Engineering 2020
Paper
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to
observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility
inference, and social learning.
|
|
Learning V1 simple cells with vector representations of local contents and matrix representations of local motions
Ruiqi Gao ,
Jianwen Xie ,
Siyuan Huang,
Yufan Ren,
Song-Chun Zhu,
Ying Nian Wu
Arxiv 2020
Paper
we propose a representational model that couples the vector representations of local image contents with the matrix representations of local pixel displacements. When the image changes from one time frame to the next due to pixel displacements, the vector at each pixel is rotated by a matrix that represents the displacement of this pixel.
|
|
A Generalized Earley Parser for Human Activity Parsing and Prediction
Siyuan Qi ,
Baoxiong Jia ,
Siyuan Huang,
Ping Wei,
Song-Chun Zhu
TPAMI 2020
Paper
Propose an algorithm to tackle the task of understanding complex human activities from (partially observed) videos from two important aspects: activity recognition and prediction.
|
|
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Siyuan Huang,
Yixin Chen,
Tao Yuan,
Siyuan Qi,
Yixin Zhu,
Song-Chun Zhu
Neural Information Processing Systems (NeurIPS) 2019
Paper
To solve the problem of 3D object detection, we propose perspective points as a novel intermediate representation, defined as the 2D projections of locally-Manhattan 3D keypoints to locate an object, and they satisfy certain geometric constraints caused by the perspective projection.
|
|
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
Yixin Chen *,
Siyuan Huang *,
Tao Yuan,
Siyuan Qi,
Yixin Zhu,
Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2019
* Equal contributions
Paper /
Supplementary /
Project
Propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction- and (ii) 3D human pose estimation. We incorporate the human-object interaction (HOI) and physical commonsense to tackle this problem.
|
|
Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
Lifeng Fan *,
Wenguan Wang *,
Siyuan Huang,
Xinyu Tang,
Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2019
* Equal contributions
Paper
Propose a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions.
|
|
Configurable 3D Scene Synthesis and 2D Image Rendering
with Per-Pixel Ground Truth using Stochastic Grammars
* Equal contributions
Internatianal Journal of Computer Vision (IJCV) 2018
Paper /
Demo
Employ physics-based rendering to synthesize photorealistic RGB images while automatically synthesizing detailed,per-pixel ground truth data, including visible surface depth and normal, object identity and material information, as well as illumination.
|
|
Human-centric Indoor Scene Synthesis using Stochastic Grammar
Siyuan Qi,
Yixin Zhu ,
Siyuan Huang,
Chenfanfu Jiang ,
Song-Chun Zhu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018
Paper /
Project /
Code /
Bibtex
Present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, for the purpose of obtaining large-scale 2D/3D image data with the perfect per-pixel ground truth.
|
|
Predicting Human Activities Using Stochastic Grammar
Siyuan Qi,
Siyuan Huang,
Ping Wei,
Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2017
Paper /
Bibtex /
Code
Use a stochastic grammar model to capture the compositional structure of events, integrating human actions, objects, and their affordances for modeling the rich context between human and environment.
|
|
Nonlinear Local Metric Learning for Person Re-identification
Siyuan Huang,
Jiwen Lu,
Jie Zhou,
Anil K. Jain
arXiv 2015
arXiv Paper
Utilize the merits of both local metric learning and deep neural network to exploit the complex nonlinear transformations in the feature space of person re-identification data.
|
|
Building Change Detection Based on 3D reconstruction
Baohua Chen,
Lei Deng,
Yueqi Duan,
Siyuan Huang,
Jie Zhou
IEEE International Conference on Image Processing (ICIP) 2015
Paper /
Bibtex
Propose a change detection framework based on RGB-D map generated by 3D reconstruction which can overcome the large illumination changes .
|
|