grasping the future: virtual hands control for fine motor tasks

@IITSEC NTSAToday

Grasping the Future: Virtual Hands Control for Fine Motor Tasks

Ronald Punako Jr., Concurrent Technologies Corporation

@IITSEC NTSAToday

Physical Equipment Training can be Effective, but Cost Prohibitive

2

@IITSEC NTSAToday

How can we Provide Effective Equipment Training at Reduced

Cost?

3

@IITSEC NTSAToday

Use the Same Hands-on Procedures with Virtual Equipment

4

@IITSEC NTSAToday

Overview

Science & Design System Architecture Interactivity Experimental Design Results Future Research

5

@IITSEC NTSAToday

Science & Design – Situated Cognition

Prototype is a virtual reality system that provides both cognitive and psychomotor equipment maintenance training experiences.

Knowing is inseparable from doing. All knowledge is situated in activity bound to social, cultural and physical contexts (Situated Cognition, Gee et al.).

6

@IITSEC NTSAToday

Science and Design – Symbolic Interaction

7

Gestures may be assigned to different actions in a computer based instruction to embody varied content knowledge and to regulate the instruction and learning process (Xu, X., & Ke, F., 2014).

Experience and cognition are grounded in actions and movements, such as simulations, situated actions, or states of body gestures (Barsalou, 2008, 2010).

@IITSEC NTSAToday

Science and Design - Mental Practice

8

When thinking about a saw, the actions and movements using a saw to cut the wood or metal will encode one’s perception on the functionality of a saw.

Body movements and actions also prompt us to recall our prior experiences and cognition, link them to the current situation, and reapply them for future actions and understanding (Barsalou 2010; Mizelle & Wheaton 2010).

@IITSEC NTSAToday

System Architecture – Prototype Client

9

User Interface

Window Manager

CalloutManager

ToolsManager

Scene Graph

EquipmentModel

Parts

Fasteners

Sequencing

Exercise

Sequences

ActionsX

Gaze

Gesture

Talk

@IITSEC NTSAToday

System Architecture - Gestures

10

Human perception of movementFluid motion: 13-50msFrames p/second: 24-60fps

Prototype hand movementFluid motion: 13msFrames p/second: 78fps

@IITSEC NTSAToday

System Architecture - Visualization

11

Helps attain high measures of presence, or the feeling of “being there.”

@IITSEC NTSAToday

System Architecture - Game Engine

12

Unity is the game engine and integration point for the prototype.

@IITSEC NTSAToday

Interactivity - Virtual Environment

13

@IITSEC NTSAToday

Interactivity – Equipment Model

14

@IITSEC NTSAToday

Interactivity - Hands Articulation

15

@IITSEC NTSAToday

Interactivity - Pointing

16

@IITSEC NTSAToday

Interactivity - Pinching

17

@IITSEC NTSAToday

Interactivity - Exercise Progression

18

@IITSEC NTSAToday

Benefits – High-Order Cognitive Skills

19

Application of high-order cognitive activities Apply, Analyze, and Evaluate

Immediate application of procedural knowledge Consistent and constant feedback

during procedure

Prototype

@IITSEC NTSAToday

Benefits – Low-Order Cognitive Skills

20

Application of low-order cognition activities Remember, understand

Why? It’s a virtual instruction manual 3D collapsed\exploded diagram Parts name callouts using gaze

Prototype

@IITSEC NTSAToday

Benefits – Psychomotor Skills

21

Imitation of tool activity improves skilled movements Psychomotor Skills (by rehearsing motions) Visuospatial Skills (by rehearsing motions in

realistic environment) Visual correlation of real-to-virtual hand

movement creates the illusion of “being there”

Hand movement and head movementcreates the illusion the body in VR space

Prototype

@IITSEC NTSAToday

Experimental Design - Environment

22

OculusLeap MotionAudio

Starting location

Head tracker

Alienware PC5 Feet

1in. 2ft.

Gesture frame

@IITSEC NTSAToday

Experimental Design – Sample Frame

23

Participants were highly familiar and experienced with the helicopter mission kit (HMK).

Key: pn = participant code

@IITSEC NTSAToday

Experimental Design – Immersive Tendencies Questionnaire (ITQ)

24

Participants reported low to moderately-low immersive tendencies.

Key: pn = participant code Scale 1 = low tendency, 7 = high tendency

@IITSEC NTSAToday

Experimental Design – Light Bulb Task

Start Locate part 1

Remove 7 fasteners

Decouple part 1

25

Locate part 2

Remove 2 fasteners

Decouple part 2

@IITSEC NTSAToday

Results – Presence Questionnaire (PQ)

26

Participants reported high to moderately-high levels of presence.Recall that ITQ scores are highly correlated to PQ scores.

Key: 1 = Scale reversed

@IITSEC NTSAToday

Results - Thinking Out Loud

27

Key: tn = Time scale

Mostly positive experiences reported for Realism.Some positive and negative experiences reported for Interface Quality.

@IITSEC NTSAToday

Results – Presence Discussion

Low ITQ outcomes are highly correlated to low PQ outcomes, and vice-versa. However, participants reported low ITQ outcomes and high PQ outcomes.

1. Would the same participants report a low PQ outcome without virtual hands?

2. Would a larger and more diverse participant sample report differently?

28

@IITSEC NTSAToday

Results – Skills Discussion

Regarding the efficacy of cognitive and psychomotor skills training.

1. Does use of virtual hands training provide better transfer of cognitive skills than non-virtual hands implementations?

2. Does use of virtual hands provide better transfer of psychomotor skills than mental practice?

29

@IITSEC NTSAToday

Future Research

1. How does tool simulation effect cognitive and psychomotor tasks?

2. How does eye movement support measurement of mastery?

3. How can exercise construction be automated?

4. How may collaboration be supported across realities?

30

@IITSEC NTSAToday

Questions

Thank you for your time!

31

@IITSEC NTSAToday

Contacts

Corey NorrisDirector, Software Solutionsnorrisc@ctc.com814-262-2383

Ron PunakoSenior Software Engineerpunakor@ctc.com814-269-6538

32