gamification with virtual characters at the borders of mixed realities and algebras

CASA, Geneva, 2016 Slide 1University of Crete & Foundation for Research and Technology - Hellas

Gamificationwithvirtualcharactersatthebordersofmixedrealitiesandalgebras

Prof.GeorgePapagiannakisUniversityof Crete

&Foundation for Reserach and TechnologyHellas

Heraklion,Crete,Greece


From Augmenta

http://whatisaugmenta.com


To Magic Leap

http://www.magicleap.com/#/home


From J.A.R.V.I.S.

http://marvel-movies.wikia.com/wiki/J.A.R.V.I.S.


To Hololens

https://www.microsoft.com/microsoft-hololens/en-us


& Asobo

http://www.asobostudio.com


Overview• A journey in Mixed Reality

(MR) and Algebras• How many Realities?• Gamification• Presence?

• Different algebras for character simulation• Dual quaternions• Geometric algebra?

• Current motivation and MR research questions

• MR character enabling technologies

• Conclusions and future work


How many Realities? Mixed Reality = AR + VR + AV+ CR+ IR+ HR+ SR…

MR Reality-Virtuality Continuum

Extent of World KnowledgeWorld

UnmodeledWorld Fully

Modeled

Real Environment

Virtual Environment

AR VR

AR-VR: Papagiannakis, G., Schertenleib, S., O'Kennedy, B., Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D., 2005. Mixing Virtual andReal scenes in the site of ancient Pompeii. Computer Animation and Virtual Worlds, John Wiley and Sons Ltd 16, 1, 11–24.Cross-Reality (CR): Davies, C.J., Miller, A., and Allison, C. 2012. Virtual Time Windows: Applying cross reality to cultural heritage. Proceedings of thePostgraduate Conference on the Convergence of Networking and Telecomunications, ISBN: 978-1-902560-26-7.Hybrid-Reality (HR): Reda, K., Febretti, A., Knoll, A., et al. 2013. Visualizing Large, Heterogeneous Data in Hybrid-Reality Environments. ComputerGraphics and Applications, IEEE 33, 4, 38–48.Indirect Reality (IR): Wither, J., Tsai, Y.-T., and Azuma, R. 2011. Indirect augmented reality. Computers & Graphics 35, 4, 810–822.Substitutional Reality (SR): K. Suzuki, S. Wakisaka, and N. Fujii, “Substitutional Reality System: A Novel Experimental Platform for ExperiencingAlternative Reality,” Sci. Rep., vol. 2, pp. 1–9, Jun. 2012.

CR-IR

HRAV

The Mixed Reality (MR) Continuum [Milgram99] [Azuma01]

SR


Gamification, Serious Games• “gamification”: as the use of game design

elements in non-game contexts [Deterdinget al 11]

• Use of game dynamics (e.g. plot), game mechanics (e.g. rules) and game components (e.g. points, avatars) in order to engage more people

• “Serious games”: computer games that are not limited to the aim of providing entertainment [Macedonia 2000], [Macedonia 2002]

• that allow for collaborative use of 3D spaces that are used for learning and educational purposes in a number of application domains [Macedonia 2002]

S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From game design elements to gamefulness,” presented at the the 15th International Academic MindTrek Conference, New York, New York, USA, 2011, p. 9.Macedonia M (2000) Using technology and innovation to simulate daily life. IEEE Computer 33(4):110–112 Macedonia M (2002) Games soldiers play. IEEE Spectrum 39(3): 32–37


Presence• The concept of ‘presence’ refers to the phenomenon of behaving and feeling as if we

are in the virtual world: ‘being there’ created by computer displays [Sanchez-Vives05]

• Another approach is that the sense of ‘being there’ or ‘being someone’ in a VE is grounded on the ability ‘to do there’ in VR [Sanchez-Vives05] , AR [Torre00][Egges07] and MR [Wagner09]

• …and many more definitions…

- Torre R, Fua P, Balcisoy S, Ponder M, Thalmann D. Interaction between real and virtual humans: Playing checkers. Proc. Eurographics Workshop On Virtual Environments (2000)- Sanchez-Vives M., and Slater, M., “From presence to consciousness through virtual reality”. Nature Reviews Neuroscience (2005) vol. 6 (4) pp. pp. 332-339- Egges, A., Papagiannakis, G., Magnenat-Thalmann, N., “Presence and Interaction in Mixed Realities”, The Visual Computer, Springer-Verlag, Volume 23, Number 5, May 2007- Wagner I, Broll W, Jacucci G, Kuutii K, McCall R, Morrison A, Schmalstieg D, Terrin J. On the Role of Presence in Mixed Reality. PRESENCE: Teleoperatorsand Virtual Environments (2009) vol. 18 (4) pp. 249-276


The VHD++ framework

§M. Ponder, G. Papagiannakis, T. Molet, N. Magnenat-Thalmann, D. Thalmann. VHD++ Development Framework: Towards Extendible, Component Based VR/AR Simulation Engine Featuring Advanced Virtual Character Technologies. IEEE Computer Society Press, Proceedings of Computer Graphics International (CGI), pp. 96-104. 2003

components•Animation blending• Idle motion • skeleton animation• skin deformation• cloth simulation• face simulation• speech simulation• AR image blending• PRT Illumination• hair animatipn• Crowd simulation• Python scripting• motion capture• VR devices• 3D sound• stereo rendering


MOBILE AUGMENTEDREALITY SYSTEM

Real image for tracking

3D Rendering

VR Runtime Engine

Skeleton Animation

Skin Deformation

VRML97 Loader

XML Loader

Real-time, Marker-less Camera Tracking

3D Static Models

Virtual Humans

Occluder Models

VR Content Repository

Binocular HMD

Augmented Scene

FireWire WebCam

Local connectivity

Camera Matrix&

Background image

Cloth Simulation

Face & Speech Simulation

Sound Rendering

Python Script Engine

sounds

Facial expressions

3D clothes

Python Scripts

Animations

User

Mobile Workstation

Scene Tracking features

Papagiannakis, G., Schertenleib, S., O’Kennedy, B., Arevalo-Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D., “Mixing Virtual and Real scenes in the site of ancient Pompeii”, Computer Animation and Virtual Worlds, p 11-24, Volume 16, Issue 1, John Wiley and Sons Ltd, February 2005.


The importance of AR Tracking• Tracking is the basic enabling technology for Augmented Reality• Without accurate tracking you can’t generate the merged real-virtual

environment• Tracking is significantly more difficult in AR than in Virtual Environments

generate character

VR

track camera

G. Papagiannakis and N. Magnenat-Thalmann, “Mobile Augmented Heritage: Enabling Human Life in ancient Pompeii,” The International Journal of Architectural Computing, Multi-Science Publishing, vol. 5, no. 2, pp. 395–415, 2007.


MOBILE AR CULTURAL HERITAGE GUIDE: LIFEPLUS EU PROJECT

Papagiannakis, G., Magnenat-Thalmann, N., “Mobile Augmented Heritage: Enabling Human Life in ancient Pompeii”, International Journal of Architectural Computing, Multi-Science Publishing, July 2007, issue 02, volume 05, pp.395-415, 2007.

G. Papagiannakis, S. Schertenleib, B. O'Kennedy, M. Arevalo-Poizat, N. Magnenat-Thalmann, A. Stoddart, and D. Thalmann, “Mixing Vir tual and Real scenes in the site of ancient Pompeii,” Computer Animation and Virtual Worlds, John Wiley and Sons Ltd, vol. 16, no. 1, pp. 11–24, Feb. 2005.


MOBILE AR CULTURAL HERITAGE GUIDE: LIFEPLUS EU PROJECT

G. Papagiannakis, S. Schertenleib, B. O'Kennedy, M. Arevalo-Poizat, N. Magnenat-Thalmann, A. Stoddart, and D. Thalmann, “Mixing Vir tual and Real scenes in the site of ancient Pompeii,” Computer Animation and Virtual Worlds, John Wiley and Sons Ltd, vol. 16, no. 1, pp. 11–24, Feb. 2005.


Dialogue-based interaction and Presence in AR

A. Egges, G. Papagiannakis, and N. Magnenat-Thalmann, “Presence and interaction in mixed reality environments,” The Visual Computer, Springer, vol. 23, no. 5, pp. 317–333, May 2007.


Self Adaptive Animation in AR

• Dual Quaternions 1,2:• Rigid transformation

• Blending• Linear Blending (DLB)• Iterative Blending (DIB)

€

(1+ε2(t0i + t1 j + t2k)).q0

1. Clifford, W.: Mathematical Papers. Macmillan (1882).

2. Kavan, L., Collins, S., Zara, J., O’Sullivan, C.: Skinning with dual quaternions. In: 2007 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pp. 39–46. ACM Press (2007).

Chaudhuri, P., Papagiannakis, G., Magnenat-Thalmann, N., "Self adaptive animation based on user perspective", The Visual Computer, Springer-Verlag, 24(7-9), pp. 525- 533, July 2008, presented also in CGI08


Our current MR Research Questions• Does a single mathematical geometric framework exist, to

combine a virtual human's skinning and animation in MR? • How can we enhance presence with gamification? • What about modern game engines and 3D platforms?


Back to the drawing board…


glGA

Utilities

glGARigMesh

PlatformWrapper

glGAMesh

glGAHelper Application

Init

Display

Windows Mac Linux IOS

Platforms

ImageMagick

AntTweakBar

GLFW

GLEW

Assimp

GLM

External Libraries

Textures 3D ModelsShader Programs

Resources

SmartBody

“Keep it simple” character rendering

framework

G. Papagiannakis, P. Papanikolaou, E. Greassidou, and P. E. Trahanias, “glGA: an OpenGL Geometric Application Framework for a Modern, Shader-based Computer Graphics Curriculum.,” Eurographics’14 Education Track, 2014.


Algebras for Computer Graphics (CG)

• Limitations of CG linear algebra representations• Points, lines, areas, vector products

• E.g. cross product not defined in 3D

• CG scientists apply several fixes:• Separate data structures (e.g. quaternions,

dual-quaternions)• Different combinational programming procedures

(line-sphere-triangle-plane intersections)

• Is there an alternative?


Some history

William Hamilton Hermann Grassmann William Clifford

Chris Doran, Introduction to GA, http://geometry.mrao.cam.ac.uk/home/introduction-to-ga/


Quaternions

Generalises complex numbers, introduced the cross product and some notation still in use today.

Confusion over status of vectors, but quaternions are very powerful for describing rotations.



Grassmann algebra

German schoolteacher (1809-1877) who struggled for recognition in his own lifetime.

Published the LinealeAusdehnungslehre in 1844.

Introduced a new, outer product that is antisymmetric.



Properties of the outer product

The result of the outer product is a new object: a BIVECTOR

Bivectors form a linear space

We can visualise bivector addition in 3D

ab

b+c

c

a



Geometric algebra

W.K. Clifford (1845 –1879) introduced the geometric product of two vectors.

The product of two vectors is the sum of a scalar and a bivector.

Think of the sum as like the real and imaginary parts of a complex number.



Two dimensions

2D sufficient to understand basic results. Construct an orthonormal basis.

Parallel vectors commute.

Orthogonal vectors anticommute.



The bivector

The unit bivector has negative square.

Follows purely form the axioms of geometric algebra.

We have not said anything about complex numbers, or solving polynomial equations.

We have invented complex numbers!



From dual-quaternions to GA• In [Kavan et al. 2008] dual quaternions were employed for geometrical

skinning (but not for animation).

• In [Chaudhuri et al. 2008] we employed them for animation blending

• Dual quaternions:• where qε and q0 are real quaternions and ε is the dual number where ε2 = 0 . This looks

just like Clifford’s biquaternion q + ωr where ω2 = 0 [Clifford 1873].

• In our recent geometric algebra (GA) [Papagiannakis 2013] framework we replaced quaternions with GA:• Character animation blending

• Drop-in, efficient replacement

of existing quaternion spherical interpolation methods

• Can this GA framework be employed in modern ARM architectures for mobile AR?

q̂ = q1,q2,q3,q4( )+ε q01,q02,q03,q04( ) = q0 +εqε

[Clifford 1873] CLIFFORD, W.K. 1873. A preliminary sketch of biquaternions. Proc. London Math. Soc. 4:381–395[Kavan 2008] KAVAN, L., COLLINS, S., ZARA, J., AND CO'SULLIVAN. 2008. Geometric skinning with approximate dual quaternion blending. ACM Transactions on Graphics.[Chaudhuri 2008] CHAUDHURI, P., PAPAGIANNAKIS, G., AND MAGNENAT-THALMANN, N. 2008. Self Adaptive Animation based on User Perspective. The Visual Computer, Springer-Verlag24, 7-9, 525–533. [Papagiannakis 2013] PAPAGIANNAKIS, G. 2013. Geometric algebra rotors for skinned character animation blending. Technical Brief, ACM SIGGRAPH ASIA 2013, Hong Kong, November 2013, 1–6.


Contribution: Main Algorithm for GA animation interpolation

• employment of Euclidean GA rotors• as fast, drop-in replacements for quaternion algebra, during animation

orientation interpolation

• We provide two different approaches to express quaternionsas GA rotors

• fast interpolate between them using the rotor exponentialformula:

RARn = RB ⇒ R = e

−I3aφ(2n)


• Input: quaternion orientation of existing animation & skinning frameworks

• Output: express with different ways input and output quaternions as GA rotors and members of the algebra:

i = e1I3= e2 ^ e3j = e2I3= −e1^ e3k = e3I3= e1^ e2ijk = −1q =1+ i+ j + k

R = ba = b ⋅a+ b ^ a = cosφ − I sinφ⇒

R = e−Iφ ⇒ R = e−I3u

φ2

Contribution: Main Algorithm for GA animation interpolation II

Papagiannakis, G. 2013. Geometric algebra rotors for skinned character animation blending. Technical Brief, ACM SIGGRAPH ASIA 2013, Hong Kong, November 2013


GA for character animation on mobile ARM• Use of GA for encoding

orientations, for mobile ARdevices (ARM architecture)

• The GA rotors e(φ I*α/2n) canconvert any axis α (not only atthe origin) into a rotationaloperator.• Quaternions can be used only on

other quaternions.

• GA rotors are universal operatorscapable of rotating other subspaces:lines, planes, and volumes as firstclass operators.

G. Papagiannakis, G. Elissavet, P. E. Trahanias, and M. Tsioumas, “A Geometric Algebra Animation Method for Mobile Augmented Reality Simulations in Digital Heritage Sites.,” EuroMed, vol. 8740, no. 25, pp. 258–267, 2014.


AR Presence and Gamification

How to augment indoors and outdoors Cultural Heritage sites with real time Virtual characters

Propose a complete methodology for robust authoring of AR virtual characters using only modern mobile devices

Propose an alternative method for smooth rotation of AR scene: Geometric Algebra (GA) rotors


The Smartbody character animation system

Animation system with a wide range of capabilities

Harness its advanced character behavioral capabilities.

SHAPIRO, A. 2011. Building a character animation system, MIG’ 11,Proceedings of the 4th

International Conference on Motion in Games

Locomotion Object manipulation

Gazing

Speech synthesis and Lip Sync

Blending Steering

Eye movementsGestures Head movement


A fast and robust pipeline for populating mobile AR scenes with gamified virtual characters

[Papaefthymiou etal.2015a]Papaefthymiou,M.,Feng,A.,Shapiro, A.,andPapagiannakis,G.(2015)'Afastandrobustpipeline forpopulatingmobileARsceneswithgamified virtualcharacters'.InSIGGRAPHAsia2015MobileGraphicsandInteractiveApplications, SA’15,pages22:1–22:8,NewYork,NY,USA.


Life-sized crowd simulation in mobile AR• Creation of AR environments with real life-sized crowd and group simulation,

on current mobile with cardboard style mobile VR displays [Zikas etal.2016a]• Virtual characters can support many real life behaviors like locomotion and

performing gestures• Extend [Papaefthymiou etal.2015a] to to include group and crowd

simulation in mobile AR.

[Zikas etal.2016a]Zikas Paul,PapaefthymiouMargarita,Mpaxlitzanakis Vasilis,andPapagiannakis George.2016.Life-sizedGroupandCrowdsimulationinMobileAR.In Proceedingsofthe29thInternationalConferenceonComputerAnimationandSocialAgents (CASA'16).ACM,NewYork,NY,USA,79-82.


A Conformal Geometric Algebra code generator comparison for Virtual Character Simulation in MR

• Efficient method for robust authoring (rotation) of Augmented reality scene using Euclidean Geometric Algebra rotors [Papaefthymiou etal.2016a, Papaefthymiou etal.2015b]

• Propose two fast animation blending methods using GA and CGA and compare them with:• Quaternions

• Dual Quaternions

• Compare efficiency of GA code generators:• Gaigen library

• Versor library

• Gaalop C++ code generator[Papaefthymiouetal.2015a]Papaefthymiou,M., Papagiannakis,G.,Aristidou,A.,Ioannides,M.,“Aconformalgeometricalgebraframeworkformixedrealityandmobiledisplay”,AppliedGeometricAlgebrainComputerScienceandEngineering2015,AGACSE2015,Barcelona,July2015[Papaefthymiouetal.2016b]Papaefthymiou M.,Hildenbrand D.andPapagiannakis G.,“AConformalGeometricAlgebracodegeneratorcomparisonforVirtualCharacterSimulationinMixedReality”,AdvancesinAppliedCliffordAlgebrasJournal(accepted),alsopresentedinGACSEworkshop,CGI2016,Heraklion,Greece,June2016


An inclusive Conformal Geometric Algebra GPU animation interpolation and deformation algorithm

• An inclusive, single mathematical framework[Papaefthymiou et al. 2016c] replacing previous linearalgebra, matrices, euler angles, single and dual-quaternions used for:

• character animation interpolation and

• skinning (rotation, translation and dilation)

• GA Motor to represent (translation and rotation)

• GA Dilator to represent uniform scaling

• Final transformation is computed by the geometricproduct of Motor and Dilator

[Papaefthymiou etal.2016c]PapaefthymiouM.,Hildenbrand D.andPapagiannakis G.,“AninclusiveConformalGeometricAlgebraGPUanimationinterpolationanddeformationalgorithm”,TheVisualcomputerJournal(accepted),Springer-Nature,alsopresentedinCGI2016,Heraklion,Greece,June2016


An inclusive Conformal Geometric Algebra GPU animation interpolation and deformation algorithm

[Papaefthymiou etal.2016c]PapaefthymiouM.,Hildenbrand D.andPapagiannakis G.,“AninclusiveConformalGeometricAlgebraGPUanimationinterpolationanddeformationalgorithm”,TheVisualcomputerJournal(accepted),Springer-Nature,alsopresentedinCGI2016,Heraklion,Greece,June2016


Mixed Reality Serious Games (MRSGs) for smart education

• Creation of two MRSGs [Zikas etal.2016b] with similar educational content for both VR and AR technologies in order to compare:

• gamification elements

• learning experience

• gameplay

• Dynamics: are dependent on the used AR or VR display technology (META AR glasses or Google cardboard)

• Mechanics and Components: can absorb minimal display changes and not affect the game’s core.

[Zikas etal.2016b]PaulZikas,Vasileios Bachlitzanakis,MargaritaPapaefthymiou,SteveKateros,StylianosGeorgiou,NikosLydatakis,GeorgePapagiannakis,“MixedRealitySeriousGamesforsmarteducation”EuropeanConferenceonGamesBasedLearning,,Paisley, Scotland,October 2016(submitted)


Summary & Future Work

• Our work is formulated as a human-centric, technology-fuelledmobile, mixed reality R&D. Its main idea is summarized as:

•Mixed Reality Character Enabling Technologies:• EmployingGamification research,

• for immersive, integrated simulation employing geometric algebra

• In the future we aim to employ GA for • real-time camera tracking• area light spherical harmonic rotation and rendering


AcknowledgementsFunded by:

• Marie Curie Initial Training Network, FP7-PEOPLE-2013-ITN, under grant agreement n° 608013

• Initial Training Network for Digital Cultural Heritage: Projecting our Past to the Future: ITN-DCH

• Marie-Curie Intra-European Fellowship FP7-PEOPLE-2010-IEF, under grant agreement n° 274669

• “hifi-PRINTER”: High fidelity Presence and Interaction: convergence of computer graphics, vision and robotics for improving human-robot and human-computer interaction

• nVIDIA Academic Partnership h/w grant 2012, 2016


and one last thing…


http://www.ics.forth.gr/CGI2016

CASA, Geneva, 2016 Slide 45University of Crete & Foundation for Research and Technology - Hellas Sponsored by

Thankyouforyourattention!

Specialthanksto:

Prof.NadiaMagnenat-Thalmann

Prof.DanielThalmann

CASA’16conferencecommittee

M.Papaefthymiou,S.Kateros,S.Georgiou,N.Lydatakis,P.Zikas,V.Mpahlitzanakis

FORTH&UniversityofCrete

MIRALab