augmented reality with the intel® realsense™ sdk and r200 camera

Intel® RealSense™ Technology | Intel® Software

Kevin Arthur, Senior User Experience Researcher

Meghana Rao, Developer Evangelist

Augmented Reality with the Intel® RealSense™ SDK and R200 Camera

User Experience and Development Best Practices

Intel® RealSense™ Technology | Intel® Software 2

Outline

Part 1, Kevin

•  Overview of R200 camera and tablet augmented reality use cases

•  User experience guidelines highlights

Part 2, Meghana

•  SDK overview

•  Sample code and demos


New R200 Depth Camera For Tablets, Peripheral Dev Kit Available Now


R200 View Volume and SDK Features

R200 with Intel® RealSense™ SDK

•  Scene Perception Module, enables scene-aware AR

•  Camera tracking and localization

•  Mesh reconstruction

•  Other Modules

•  3D capture

•  Depth-enhanced photo and video

•  Measurement

•  Face detection and tracking

•  Speech (Windows SDK only)


Use Cases for R200 Mixed and Augmented Reality

Gaming and Play Education and Training Visualization


Video


Video – ToyZ Game

Shows real-time scene perception for collision and occlusion (no pre-scan)

Try it at the Intel booth

By Shachar Oz, Omek Studio at Intel


Video – Procedural Island

Shows scan as part of “capture and play”

Illustrates procedural shaders and set dressing

By Eddy Ortega, Garrett Stevens, Perceptual Computing at Intel


UX Guidelines for R200 Tablet ARDesigning real, usable apps for a mass market


Lesson 1: Give People a Reason to Move, or They Won’t

Tablet as window into a virtual space vs. tablet as fixed screen

Address with motion hints


Motion Hints – Explicit

Registered with scene

Or registered with window


Motion Hints – Implicit, Part of Experience

Lead the user with content

Example: “Windy Day” (Google Spotlight Stories)


Lesson 2: But Let People Relax Too

Support bothActive Camera Mode

•  Tiring

Inactive Camera Mode

•  Less tiring


Active-Camera and Inactive-Camera Modes

Active Camera Mode

•  Touch interaction less comfortable, less precise

Inactive Camera Mode

•  Touch interaction more comfortable, more precise

Make the main controls easy to reach with the thumbs during Active Camera modes .

During Inactive Camera modes, placing controls elsewhere is acceptable.

AVOID OKOKOK

Make the main controls easy to reach with the thumbs during Active Camera modes .

During Inactive Camera modes, placing controls elsewhere is acceptable.

AVOID OKOKOK

Touch Zones Touch Zones


Two Styles of Mixed-Reality Games

Augmented Reality Capture and Play

Active Camera

Inactive Camera


Lesson 3: Plan for the Scene

•  Consider size of play space, and use appropriate voxel resolution


Game Design Considerations

•  What objects does the user need?

•  Level design has more unknowns

•  Enhance and transform everyday objects in interesting ways


Procedural Set Dressing


Plan for the Scene

•  Understand the camera limitations. Depth data is less accurate on

•  Very bright areas

•  Clear glass

•  Black surfaces

•  Give relevant feedback

•  Fail gracefully, don’t prevent play


More UX Guidelines

software.intel.com/articles/realsense-ux-design-guidelines


Developing Augmented Reality applications with R200


What is Scene Perception?•  Provides Augmented Reality experiences through

•  Adding virtual content in real time to live camera feed

•  Handling occlusions of virtual objects with physical objects

•  Providing motion estimation and metric measurement

•  Intel® RealSense™ SDK library for Scene Perception

•  libPXCScenePerception


Scene Perception in Action


Augmented Reality in Action


Metric Measurements in Action


Requirements for Scene Perception•  The scene being captured has to be static

•  Works within the bounds of a close object / cubicle space/ living room

•  You can track camera movement in 3D space along six degrees of freedom

-courtesy: http://en.wikipedia.org/wiki/Six_degrees_of_freedom


Modalities you will learn about ( R200 )•  Using Scene Perception for Augmented Reality

•  Configuring Scene Perception

•  Setting Initial Camera Pose

•  Setting Meshing Threshold

•  Setting Voxel Resolution

•  Starting Scene Quality

•  Checking Scene Quality

•  Enabling Scene Reconstruction

•  Tracking Data

•  Camera Pose

•  Volume Preview

•  Mesh Data


Guidelines for using Scene Perception

•  Works best in the depth camera working range

•  The observed environment is assumed to be containing some visual texture and/or having some 3D structure:

•  Tracking may be inaccurate when tracking reflective or monotonous surfaces, flat walls, dark black areas or scene with limited geometric structure

•  Using appropriate VoxelResolution::XYZ_RESOLUTION settings:

•  VoxelResolution::HIGH_RESOLUTION for object size use cases

•  VoxelResolution::LOW_RESOLUTION for room sized environments


Intel® RealSense™ Front-Facing 3D Camera (F200) and Augmented Reality Usages


Augmented Reality apps can make use of:

•  Contour Mode

•  Hand Data

•  Joint Data

•  Finger Data

•  Extremity points

•  Hand Openness

•  Usages

•  Nail Salon

•  Virtual Shopping – Jewelry

•  Games

AR Usages Using Hand Module


Augmented Reality apps can make use of:

•  Landmark

•  Pose detection

•  Eye Movement

•  Expression Detection

Usages:

•  Face Makeup

•  Virtual Shopping for glasses

•  Games/Entertainment

•  Plastic Surgery

AR Usages using Face Module


Resources

software.intel.com/realsense

software.intel.com/articles/realsense-ux-design-guidelines

[email protected], @karthur

[email protected], @RaoIntel

augmented reality with the intel® realsense™ sdk and r200 camera

Software

intel booth

active camera inactive

inactive camera modes

r200 camera user experience

kevin overview of r200

new r200 depth camera

sceneaware ar camera

r200 tablet ar