i3 george papagiannakis mobile crossplatform ar eva/minerva 2014

EVA/MINERVA 2014, Slide 1 University of Crete & Foundation for Research and Technology - Hellas

��

Mobile, cross-platform, life-size animated virtual characters in indoors and outdoors AR scenes

George Papagiannakis, Greasidou Elissavet, Panos Trahanias, Michalis Tsioumas


Overview • State-of-the-art in mobile AR

•  How many Realities?

•  Presence?

•  Our previous AR systems for cultural heritage

• Current motivation and research questions • Methodology • Results & Conclusions


How many Realities? Mixed Reality = AR + VR and AV, CR, IR, HR…

The Mixed Reality (MR) Continuum [Milgram99] [Azuma01]

MR Reality-Virtuality Continuum

Extent of World KnowledgeWorld

UnmodeledWorld Fully

Modeled

Real Environment

Virtual Environment

AR VR

AR-VR: Papagiannakis, G., Schertenleib, S., O'Kennedy, B., Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D.,. 2005. Mixing Virtual and Real scenes in the site of ancient Pompeii. Computer Animation and Virtual Worlds, John Wiley and Sons Ltd 16, 1, 11–24. Cross-Reality (CR): Davies, C.J., Miller, A., and Allison, C. 2012. Virtual Time Windows: Applying cross reality to cultural heritage. Proceedings of the Postgraduate Conference on the Convergence of Networking and Telecomunications, ISBN: 978-1-902560-26-7. Hybrid-Reality (HR): Reda, K., Febretti, A., Knoll, A., et al. 2013. Visualizing Large, Heterogeneous Data in Hybrid-Reality Environments. Computer Graphics and Applications, IEEE 33, 4, 38–48. Indirect Reality (IR): Wither, J., Tsai, Y.-T., and Azuma, R. 2011. Indirect augmented reality. Computers & Graphics 35, 4, 810–822.

CR-IR

HR AV


State-of-the-art and recent advances in AR •  Latest research surveys in the field ①  Presence and Interaction in Mixed Reality Environments by Egges et al 2007

②  A survey of mobile and wireless technologies for AR Papagiannakis et al 2008

③  AR technologies, systems and applications by Carmizziani et al 2011

④  AR in built environment: Classification and implications Wang et al 2013

⑤  Mobile AR survey: a bottom-up approach Huang et al 2013

⑥  Kick-starter camera tracking projects and novel glasses: Meta-View, Occipital, Atheerlabs, PrimeSense Capri, Google™glasses and recent Samsung, Apple, Microsoft patents on AR glasses


What is Presence •  “Immersive virtual environments can break the deep, everyday connection between

where our senses tell us we are and where we are actually located and whom we are with”.

•  Typically, the visual fidelity of a VE display is low compared with physical reality.

•  The concept of ‘presence’ refers to the phenomenon of behaving and feeling as if we are in the virtual world created by computer displays [Sanchez-Vives05].

•  Another approach is that the sense of ‘being there’ in a VE is grounded on the ability to ‘do’ there [Sanchez-Vives05] and recently extended to MR [Egges07].

Sanchez-Vives M., and Slater, M., “From presence to consciousness through virtual reality”. Nature Reviews Neuroscience (2005) vol. 6 (4) pp. pp. 332-339 Egges, A., Papagiannakis, G., Magnenat-Thalmann, N., “Presence and Interaction in Mixed Realities”, The Visual Computer, Springer-Verlag, Volume 23, Number 5, May 2007


The importance of Tracking •  Tracking is the basic enabling technology for Augmented Reality •  Without accurate tracking you can’t generate the merged real-virtual

environment •  Tracking is significantly more difficult in AR than in Virtual Environments

generate VR

track camera

Papagiannakis, G., Schertenleib, S., O’Kennedy, B., Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D., “Mixing Virtual and Real scenes in the site of ancient Pompeii”, Journal of Computer Animation and Virtual Worlds, pp. 11-24, vol. 16, issue 1, John Wiley and Sons Ltd, February 2005


2. MOBILE AR CULTURAL HERITAGE GUIDE ON A SINGLE LAPTOP��

Papagiannakis, G., Schertenleib, S., O’Kennedy, B., Arevalo-Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D., “Mixing Virtual and Real scenes in the site of ancient Pompeii”, Computer Animation and Virtual Worlds, p 11-24, Volume 16, Issue 1, John Wiley and Sons Ltd, February 2005.

Papagiannakis, G., Magnenat-Thalmann, N., “Mobile Augmented Heritage: Enabling Human Life in ancient Pompeii”, International Journal of Architectural Computing, Multi-Science Publishing, July 2007, issue 02, volume 05, pp.395-415, 2007.


3. AR PRESENCE AND INTERACTION BASED ON A SINGLE MOBILE WORKSTATION ��

Egges, A., Papagiannakis, G., Magnenat-Thalmann, N., “Presence and Interaction in Mixed Realities“, The Visual Computer, Springer-Verlag, Volume 23, Number 5, May, 2007.


4. AR SELF ADAPTIVE ANIMATION ON A UMPC��

Chaudhuri, P., Papagiannakis, G., Magnenat-Thalmann, N., “Self adaptive animation based on user perspective”, The Visual Computer, Springer-Verlag, 24(7-9), pp. 525-533,July 2008


Motivation: Research question I

• Limitations of CG linear algebra representations •  Points, lines, areas, vector products

•  E.g. cross product not defined in 3D

• CG scientists apply several fixes: •  Separate data structures (e.g. quaternions)

•  Different combinational programming procedures (line-sphere-triangle-plane intersections)

•  Is there an alternative?


Motivation: Research question II •  In [Kavan et al. 2008] dual quaternions were employed for geometrical

skinning (but not for animation).

•  In [Chaudhuri et al. 2008] we employed them for animation blending

•  Dual quaternions: •  where qε and q0 are real quaternions and ε is the dual number where ε2 = 0 . This looks

just like Clifford’s biquaternion q + ωr where ω2 = 0 [Clifford 1873]. •  In our recent geometric algebra (GA) [Papagiannakis 2013] framework we

replaced quaternions with GA: •  Character animation blending

•  Drop-in, efficient replacement

of existing quaternion spherical interpolation methods

•  Can this GA framework be employed in modern ARM architectures for mobile AR?

q̂ = q1,q2,q3,q4( )+ε q01,q02,q03,q04( ) = q0 +εqε

[Clifford 1873] CLIFFORD, W.K. 1873. A preliminary sketch of biquaternions. Proc. London Math. Soc. 4:381–395 [Kavan 2008] KAVAN, L., COLLINS, S., ZARA, J., AND CO'SULLIVAN. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics. [Chaudhuri 2008] CHAUDHURI, P., PAPAGIANNAKIS, G., AND MAGNENAT-THALMANN, N. 2008. Self Adaptive Animation based on User Perspective. The Visual Computer, Springer-Verlag 24, 7-9, 525–533. [Papagiannakis 2013] PAPAGIANNAKIS, G. 2013. Geometric algebra rotors for skinned character animation blending. Technical Brief, ACM SIGGRAPH ASIA 2013, Hong Kong, November 2013, 1–6.


Contribution: Main novelties •  Use of GA for encoding

orientations, for mobile AR devices (ARM architecture)

•  The GA rotors e(φ I*α/2n) can convert any axis α (not only at the origin) into a rotational operator. •  Quaternions can be used only on

other quaternions.

•  GA rotors are universal operators capable of rotating other subspaces: lines, planes, and volumes as first class operators.


Results on mobile AR


glGA Framework: architecture I

21

glGA

Utilities

glGARigMesh

PlatformWrapper

glGAMesh

glGAHelper Application

Init

Display

Windows Mac Linux IOS

Platforms

ImageMagick

AntTweakBar

GLFW

GLEW

Assimp

GLM

External Libraries

Textures 3D Models Shader Programs

Resources


Future Work • Finalize the Roman Forum virtual characters according to

archaeologists •  fully subsume existing different algebras (linear algebra, (dual)

quaternion) under a single representation in a GA algebraic framework,

•  including translation and scaling transformations

• Employ GA rotors for real-time gesture tracking and area light rotation and rendering


Computer graphics modelling, animation and rendering technology tools demo, Whiskytree.com, ACM SIGGRAPH 2013

Presence or Realism for digital heritage?


Thank you!

i3 george papagiannakis mobile crossplatform ar eva/minerva 2014

Technology

virtual reality

cross reality

physical reality

ar papagiannakis et

ar technologies

ar vr

mixed reality environments

crossreality cr