From 3D Scene Geometry to Human Workspace

Authors: Abhinav Gupta, Scott Satkin, Alexei A. Efros and Martial Hebert

The Robotics Institute, Carnegie Mellon UniversityPresenter: Jiapei Zhang

Project Undertaking

Image if I were Abhinav

I received a project from my boss about indoor scene understanding

Selection of Subject Material at hand

• A set of indoor-scene images• Motion Capture Database• No LiDAR scanner, not as rich as VCC, China• How I wish to be a VCCer one day

VCC V5

Selection of Project Inspirations from Psychology

Where is the “entrance” of my novelty? Where is the “Observer” for traditional Image Understanding? Put Human back to Image Gestalt Psychologists: Objects are best defined not by their

identity but rather by their function. The meaning or a value of a thingconsist of what it affords. -----JJ Gibson(1979) “Affordance” “functionality”

Selection of Project Shoulder of Giants

Bounding Box Level == Object DetectionSingle-View Indoor Geometry EstimationHuman Pose Analysis via Motion Capture

Selection of Project Give a Final Word

Propose a scene representation that combines 3D scene geometry with a set of possible human actions, to create a joint space of human-scene interactions.

Predict the full human workspace of the scene

3D Scene Understanding

HUMAN ACTION MODELING

Implementation Detail Representing Scene Geometry

An indoor scene is modeled by the layout of room(walls, floors, ceilings) and the location/layout of objects.

The objects are modeled by a set of occupied voxels.

Implementation of Detail Geometry Estimation

Sliding Cuboids

Selection

Implementation of Detail Qualitative Human Pose Representation

Each pose is represented by the occupied blocks in discretized 3D space(red) and the required surfaces of interaction(green).

Implementation of Detail Human-Scene Interactions

Constraints

Binary Occupancy Matrix “Space Search” to fit “Constrains”

Free space constrain

Support constrain

Harvest Season Result Showing

Retrospect and Meditation

• Now return back to me• What useful info. for our Indoor-Scene Project

I could extract from this paper ?

Retrospect and Meditation Incorporate images to our projects Will geometry estimation improve our rough mesh model?

Categorize “mobility” of scene into two:a) Extract from Repetitions: Based on Geometry b) Extract from “Human Pose”: Based on Functionality