News
Publications
|
PartAfford: Part-level Affordance Discovery from 3D Objects
Chao Xu ,
Yixin Chen,
He Wang,
Song-Chun Zhu,
Yixin Zhu,
Siyuan Huang
Arxiv
Paper
We present a new task of part-level affordance discovery (PartAfford): Given only the affordance labels per object, the machine is tasked to (i) decompose 3D shapes into parts and (ii) discover how each part of the object corresponds to a certain affordance category.
|
|
Learning V1 simple cells with vector representations of local contents and matrix representations of local motions
Ruiqi Gao ,
Jianwen Xie ,
Siyuan Huang,
Yufan Ren,
Song-Chun Zhu,
Ying Nian Wu
AAAI 2022
Paper
we propose a representational model that couples the vector representations of local image contents with the matrix representations of local pixel displacements. When the image changes from one time frame to the next due to pixel displacements, the vector at each pixel is rotated by a matrix that represents the displacement of this pixel.
|
|
Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
Siyuan Huang*,
Yichen Xie*
Song-Chun Zhu,
Yixin Zhu
ICCV 2021
Paper /
Supplementary /
Project /
Code
We introduce a spatio-temporal representation
learning (STRL) framework, capable of learning from unlabeled 3D point clouds in a self-supervised fashion.
|
|
A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics
Qing Li,
Siyuan Huang,
Yining Hong,
Yixin Zhu,
Ying Nian Wu,
Song-Chun Zhu
ICLR 2021 The Role of Mathematical Reasoning in General Artificial Intelligence Workshop (Short Version)
Paper
we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three different levels: perception, syntax, and semantics.
|
|
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
Pan Lu*, Ran Gong*, Shibiao Jiang*, Liang Qiu, Siyuan Huang, Xiaodan Liang, Song-Chun Zhu
ACL 2021 (Oral)
Paper /
Code /
Project /
Bibtex
we construct a new largescale benchmarkconsisting of 3,002 geometry problems with dense annotation in formal language and propose a novel geometry solving approach with formal language and symbolic reasoning.
|
|
Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
Yaxuan Zhu,
Ruiqi Gao,
Siyuan Huang,
Song-Chun Zhu,
Ying Nian Wu
CVPR 2021 (Oral)
Paper /
Supplementary /
Code
To efficiently represent camera pose in 3D computer vision, we propose an approach to learn neural representations of camera poses and 3D scenes, coupled with neural representations of local camera movements.
|
|
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering
Qing Li ,
Siyuan Huang,
Yining Hong,
Song-Chun Zhu
ECCV 2020 (Oral)
Paper
We design a neural-symbolic concept learner for learning the visual concepts and a multi-dimensional Item Response Theory (mIRT) model for guiding the visual concept learning process with an adaptive curriculum.
|
|
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities
Baoxiong Jia,
Yixin Chen,
Siyuan Huang,
Yixin Zhu,
Song-Chun Zhu
ECCV 2020
Paper /
Code /
Project /
Bibtex
We introduce the LEMMA dataset to provide a single home to address these missing dimensions with carefully designed settings, wherein the numbers of tasks and agents vary to highlight different learning objectives. We densely annotate the atomic-actions with human-object interactions to provide ground-truth of the compositionality, scheduling, and assignment of daily activities.
|
|
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
Yixin Zhu ,
Tao Gao ,
Lifeng Fan ,
Siyuan Huang,
Edmonds Mark,
Hangxin Liu,
Feng Gao,
Chi Zhang,
Siyuan Qi,
Ying Nian Wu,
Josh Tenenbaum,
Song-Chun Zhu
Engineering 2020
Paper
We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to
observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility
inference, and social learning.
|
|
A Generalized Earley Parser for Human Activity Parsing and Prediction
Siyuan Qi ,
Baoxiong Jia ,
Siyuan Huang,
Ping Wei,
Song-Chun Zhu
TPAMI 2020
Paper
Propose an algorithm to tackle the task of understanding complex human activities from (partially observed) videos from two important aspects: activity recognition and prediction.
|
|
PerspectiveNet: 3D Object Detection from a Single RGB Image via Perspective Points
Siyuan Huang,
Yixin Chen,
Tao Yuan,
Siyuan Qi,
Yixin Zhu,
Song-Chun Zhu
Neural Information Processing Systems (NeurIPS) 2019
Paper
To solve the problem of 3D object detection, we propose perspective points as a novel intermediate representation, defined as the 2D projections of locally-Manhattan 3D keypoints to locate an object, and they satisfy certain geometric constraints caused by the perspective projection.
|
|
Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
Yixin Chen *,
Siyuan Huang *,
Tao Yuan,
Siyuan Qi,
Yixin Zhu,
Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2019
* Equal contributions
Paper /
Supplementary /
Project
Propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction- and (ii) 3D human pose estimation. We incorporate the human-object interaction (HOI) and physical commonsense to tackle this problem.
|
|
Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
Lifeng Fan *,
Wenguan Wang *,
Siyuan Huang,
Xinyu Tang,
Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2019
* Equal contributions
Paper
Propose a new problem of understanding human gaze communication in social videos from both atomic-level and event-level, which is significant for studying human social interactions.
|
|
Configurable 3D Scene Synthesis and 2D Image Rendering
with Per-Pixel Ground Truth using Stochastic Grammars
* Equal contributions
Internatianal Journal of Computer Vision (IJCV) 2018
Paper /
Demo
Employ physics-based rendering to synthesize photorealistic RGB images while automatically synthesizing detailed,per-pixel ground truth data, including visible surface depth and normal, object identity and material information, as well as illumination.
|
|
Human-centric Indoor Scene Synthesis using Stochastic Grammar
Siyuan Qi,
Yixin Zhu ,
Siyuan Huang,
Chenfanfu Jiang ,
Song-Chun Zhu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018
Paper /
Project /
Code /
Bibtex
Present a human-centric method to sample and synthesize 3D room layouts and 2D images thereof, for the purpose of obtaining large-scale 2D/3D image data with the perfect per-pixel ground truth.
|
|
Predicting Human Activities Using Stochastic Grammar
Siyuan Qi,
Siyuan Huang,
Ping Wei,
Song-Chun Zhu
IEEE International Conference on Computer Vision (ICCV) 2017
Paper /
Bibtex /
Code
Use a stochastic grammar model to capture the compositional structure of events, integrating human actions, objects, and their affordances for modeling the rich context between human and environment.
|
|
Nonlinear Local Metric Learning for Person Re-identification
Siyuan Huang,
Jiwen Lu,
Jie Zhou,
Anil K. Jain
arXiv 2015
arXiv Paper
Utilize the merits of both local metric learning and deep neural network to exploit the complex nonlinear transformations in the feature space of person re-identification data.
|
|
Building Change Detection Based on 3D reconstruction
Baohua Chen,
Lei Deng,
Yueqi Duan,
Siyuan Huang,
Jie Zhou
IEEE International Conference on Image Processing (ICIP) 2015
Paper /
Bibtex
Propose a change detection framework based on RGB-D map generated by 3D reconstruction which can overcome the large illumination changes .
|
|