Project Tango

Rouyun Pan

What is “Project Tango”• A new kind of spatial perception to the Android

device platform by adding advanced computer vision, image processing, and specialized sensors.

HW Spec.• NVIDIA Tegra K1 processor 4GB of RAM

• 192 CUDA core (~350 GFlop)

• 128GB of storage

• 7.02" 1080p display (323 ppi)

• Stock Android 4.4

• WiFi, Bluetooth LE and 4G LTE

• 120 degree front camera

• 4MP Camera and Depth Sensor

• Motion-tracking Camera

HW Diagram

HW Architecture• No external image

processors needed

• Reduces cost, power, latency

• Odometry

• 30 FPS

• ~3-5ms GPU processing/frame

• GPU load: ~15%

• Depth Decoding

• 5 FPS

• ~13-15ms GPU processing/frame

• GPU load: ~8%

CSI

CSI

CSI

SPI

Software Pipeline• GPU processed sensor data made available to applications Java interfaces or

native access, together with 3D engines such as Unity

• NVIDIA VisionWorks can be used for native vision processing Library of GPU accelerated vision primitives

• Use OpenVX for optimized execution of graphs of primitives

Core functionality• Motion tracking

• Area learning

• Depth perception

Motion Tracking• Project Tango implements motion tracking using visual-inertial odometry, or VIO, to estimate where a

device is relative to where it started.

• It provides the position and orientation in full six degrees of freedom, referred to as its pose .

• The APIs support two ways to get pose data:

• callbacks

• specific timestamp.

Lifecycle of Pose status

Area Learning• Project Tango can remember the key visual features of a

physical space - the edges, corners, other unique features it has visited. it is known as Simultaneous Localization and Mapping, or SLAM.

• The Project Tango supports learning, saving, and loading Area Description Files (ADF).

Localization & Drift correction• Project Tango loads the saved description, it can recognize it is within that

loaded area. This process is called localization.

• Project Tango performs drift correction to fix errors that occur during prolonged motion tracking by recognizing its position within a known physical area.

Depth Perception• Project Tango devices are equipped with integrated 3D sensors that measure

the distance (0.5 to 4 meters).

• Project Tango returns depth information in two main formats: XYZ Point Clouds, and a Project Tango-specific format called XYZij designed to aid meshing.

Coordinate system(1)• Project Tango Coordinate Frames

Coordinate system(2)• OpenGL Coordinate Frames

Coordinate system(3)• Unity Coordinate Frames

Coordinate system(4)• On-Device Coordinate Frames

Frames of reference• The Project Tango APIs provide motion tracking data based on a flexible set of

target and base frame pairs, allowing you to pick the coordinate system that best fits your use case.

Mobile Visual Computing Enables New ExperiencesNeed for advanced sensors

and the GPU throughput

Computational Photography

and Videography

Face, Body and

Gesture Tracking

3D Scene/Object Reconstruction Augmented Reality

Demo

So where do you get a tango?

• Google store (currently available in the US) https://store.google.com/product/project_tango_tablet_development_kit

Reference• https://developers.google.com/project-tango/

developer-overview

• https://plus.google.com/communities/114537896428695886568

• http://stackoverflow.com/questions/tagged/google-project-tango

• https://www.passion47.com/project-tango/

• http://www.ustream.tv/recorded/51294136