Support gratefully acknowledged: CISCO, Microsoft Research, NIH, NVIDIA, NSF Awards IIS-CHS-1423059, HCC-CGV-1319567, II-1405847 (“Seeing the Future: Ubiquitous Computing in EyeGlasses”), and the BeingThere Int’l Research Centre, a collaboration of ETH Zurich, NTU Singapore, UNC Chapel Hill and Singapore National Research

Foundation, Media Development Authority, and Interactive Digital Media Program Office.

ARPA-E Telepresence Workshop April 26, 2016

SurgicalConsulta/onTelepresence

AndreiState(UNC)1994

Henry Fuchs UNC Chapel Hill

AdvancedDisplaysandTechniques

forTelepresenceCOAUTHORSforpapers

inthistalk:

Young-WoonCha

RohanChabra

NateDierk

MingsongDou

WilGarreM

GentaroHirota

Kur/sKeller

DougLanman

MarkLivingston

DavidLuebke

AndrewMaimone

FedericoMenozzi

EMaPisano,MD

KishoreRathinavel

AndreiState

EricWallen,MD

GregWelch

MaryWhiMon

XuboYang

1

VideoTeleconferencingvsTelepresence

2

CiscoTelePresence3000

•  VideoTeleconferencing

–  Conven/onal2Dvideocaptureanddisplay

–  Singlecamera,singledisplayateach

siteiscommonconfigura/onfor

Skype,GoogleHangout,etc.

Threedistantroomscombinedintoasinglespace

withwall-sized3Ddisplays

•  Telepresence

–  Providesillusionofpresenceintheremoteorcombinedlocal&remote

space

–  Providesproperstereoviewsfromthe

preciseloca/onoftheuser

–  Stereoviewschangeappropriatelyasusermoves

–  Providespropereyecontactandeyegazecuesamongallthepar/cipants

TelepresenceComponentTechnologies

•  Acquisi/on(cameras)

•  3Dreconstruc/on•  Communica/onnetwork

•  Viewerviewpointtracking•  Genera/onofimage(s)fordisplay

•  Displaypresenta/on

3

CiscoTelePresence3000

Localsurgeon

Remoteconsultants

3DTelepresenceComponentIssues

•  Acquisi/on(cameras)

–  RGBordepthcameras,number,placement

(fixed,moving,head-mounted,an/cipa/ng

theposi/onofremoteusers,...)

•  3Dreconstruc/on

–  Combineinputfrommanycolor+depthcameras

–  Viewthesetof3Dcolorpointsinspace

–  Fitsurfaceofpolygonsaroundthe3Dpoints

–  Enhancepolygonsurfaceswithmodel-basedinfo

(e.g.,humanskeleton)or/me-based

integra/on,light-fieldinfo,..

•  Communica/onnetwork

–  Bandwidth,compression-decompression,..

•  Viewerviewpointtracking

–  Trackedareainstrumentedornot;

indoors,outdoors

–  Lowlatency(30msec,…100microseconds)

•  Genera/onofimage(s)fordisplay

–  Renderingquality,latency,powerusage

•  Displaypresenta/on

–  Largeformat:Differentstereoimagepairforeach

viewer(requirestereoglassesornot)

–  Near-eye:eyeglassesformfactor,widefieldof

view

4

3Dreconstruc/onfrom~10Kinectcolor+depth

cameras(Douetal,IEEEVR2014)

DisplayAlterna/ves*

5*Onlynear-termdeployabletechnologies

Holografika

Oculus

MicrosoiHololens

Large,fixed:pro:nothingtowearontheface(atmost,wearsunglasses,likeat3Dmovies)

con:onlyseetheremotepeopleonfarsideofthedisplay;

local&remotepar/cipantscan’tbeinsamesharedspace

Head-worn:VirtualReality:pro:cheap,immersive

con:cannotseeownbody,localpeopleandspace

AugmentedReality:pro:seeownbodyandlocalenvironment

con:widefieldofviewjustoutofreach,clunky-virtualobjects(distantpeople)cannotoccludereal

world,eitherappeartransparentorrealworldhastobedark

LargeFormat3DDisplays:

withoutglasses

Challenge:Differentimagetobeseenbyeachviewer

emitadifferentcolortoeachdirec/on

Solu?on:~100rear-projectorsforahumanlife-

sizedisplay

Alterna?ves:ifonlyafewviewers,steerthebeamtoonlythoseplaceswherethereisa

viewer

mul/-layerdisplays:

passiveorac/vebarrierdisplays

compressive,tensordisplays

steerablebacklightdisplays

RemainingChallenges:compressionar/facts,

insufficientswitchingspeedofspa/allight

modula/ondisplays(LCDs),lowlightefficiency

6

Holografika

Arrayofprojectors

Anisotropicscreen

Large,FixedDisplayswithShuMerGlasses

7

•  Easierthaneyeglass-freemul/view

displays

•  ShuMerglassesselectwhichuser(both

eyes)seesapar/cularsub-frame

–  Likestereo,just6subframesinsteadof2

•  Future:couldreducenumberofprojectors

bysacrificingsomecolorresolu/on

•  Future:fasterdisplayspeeds

(micromirrors)willenablelessexpensive

solu/ons(e.g.,4usersystemwith2

projectors)

Becketal,VR2013

6stereoprojectors,eachassigned

permanentlyto1of6func/ons:

1:LeieyeRed

2:LeieyeGreen

3:LeieyeBlue

4:RighteyeRed

5:RighteyeGreen

6:RighteyeBlue

Eachof6subframesassignedtoone

of6users(wouldn’tworkfor7users)

PolarizerAfilter

PolarizerBfilter

FixedDisplaysvsNear-eyeDisplays

8

•  Withfixeddisplays,par/cipants(localand

remote)cannotallbeinthesameshared

space,sitnexttoeachother,..

•  Forsharinglocalspace,needdisplaysthat

userswear,sovirtualobjects(distantpeople)

canbelocatedanywhere

•  Requirementsfornear-eyedisplays

–  Seelocalsurroundings(ownbody,table,material)andvirtualobjects

–  Properocclusionbetweenrealandvirtualobjects

Near-EyeDisplayChallenges

•  ClosedVRheadsetsprobablyunacceptablefortelepresence

•  OpenAugmentedRealityheadsets

sizeokfornarrowFieldofView(40degrees)

sizebulkyforwideFoV(90-100degrees)

•  Nonehaveocclusionofrealworldbyvirtual,sorealworldhastobekeptdark

–  Notgoodfortelepresence:eitherlocalenvironmentisdarkorvirtualimageryisvery

bright

9

MagicLeap

LumusDK-32 MicrosoiHololens

Meta2PhaseSpaceSmokeMeta2

WideFoVEyeglassARDisplay:

Maimoneetal,Siggraph2014

10

WhereareWe?WhichwayForward?1of2

•  Nowisanopportune/metoTelepresence

–  CanridetheVRwave•  Manydevelopingtechnologicalpieces,

–  butnocollec/onofthemgivesacompletesolu/ontodayfor

Telepresence

•  Similartopersonalcomputertechnologyca.1970

–  WantAlanKay’sDynabookvision(8½”x11”x1”withfull-color

display,touchscreen,radiocommunica/ontoARPAnet,removable

secondarystorage)-howtomakeaDynabook

–  Challenge:howtogettoaneffec/veTelepresencesystem

•  Systemdevelopment:

–  Giveupsomerequirements:cost,bulk,weight

–  Fortelepresence:usebestavailableARplasorm&add

temporaryenhancements/workarounds

•  (Cont.)

11

MicrosoiHololens

Meta2

WhereareWe?WhichwayForward?2of2

•  Technologydevelopment:workoneach

componentproblem;integrateintorest

ofdevelopmentsystem.

•  Fuchsexamples:

–  Widefieldofvieweyeglasses

•  Maimone’spinlightdisplays:resolu/on,

diffrac/on,occlusion,..

–  Occlusion:addtocurrentARdisplay•  Fortemporarysystemdevelopment:

ligh/ngcontrolincontrolledspaces

•  Forpermanenttechnologicalsolu/on:

mul/-layerlightfielddisplays

–  Head-gearTracking:•  Goanywherewithaccuracyandlow

latency:GPS,mul/pleoutward-looking

rolling-shuMercameras,mul/pleIMUs

12

ARwithoutocclusion ARwithocclusion

MagicLeapwithoutocclusion(?)

Headsetimagebright;roomisdark

13

MagicLeap

LumusDK-32Meta2

PhaseSpaceSmoke

Meta2

ThankYou

MicrosoiHololens