grasping the future: virtual hands control for fine motor tasks
TRANSCRIPT
@IITSEC NTSAToday
Grasping the Future: Virtual Hands Control for Fine Motor Tasks
Ronald Punako Jr., Concurrent Technologies Corporation
@IITSEC NTSAToday
Physical Equipment Training can be Effective, but Cost Prohibitive
2
@IITSEC NTSAToday
How can we Provide Effective Equipment Training at Reduced
Cost?
3
@IITSEC NTSAToday
Use the Same Hands-on Procedures with Virtual Equipment
4
@IITSEC NTSAToday
Overview
Science & Design System Architecture Interactivity Experimental Design Results Future Research
5
@IITSEC NTSAToday
Science & Design – Situated Cognition
Prototype is a virtual reality system that provides both cognitive and psychomotor equipment maintenance training experiences.
Knowing is inseparable from doing. All knowledge is situated in activity bound to social, cultural and physical contexts (Situated Cognition, Gee et al.).
6
@IITSEC NTSAToday
Science and Design – Symbolic Interaction
7
Gestures may be assigned to different actions in a computer based instruction to embody varied content knowledge and to regulate the instruction and learning process (Xu, X., & Ke, F., 2014).
Experience and cognition are grounded in actions and movements, such as simulations, situated actions, or states of body gestures (Barsalou, 2008, 2010).
@IITSEC NTSAToday
Science and Design - Mental Practice
8
When thinking about a saw, the actions and movements using a saw to cut the wood or metal will encode one’s perception on the functionality of a saw.
Body movements and actions also prompt us to recall our prior experiences and cognition, link them to the current situation, and reapply them for future actions and understanding (Barsalou 2010; Mizelle & Wheaton 2010).
@IITSEC NTSAToday
System Architecture – Prototype Client
9
User Interface
Window Manager
CalloutManager
ToolsManager
Scene Graph
EquipmentModel
Parts
Fasteners
Sequencing
Exercise
Sequences
ActionsX
Gaze
Gesture
Talk
@IITSEC NTSAToday
System Architecture - Gestures
10
Human perception of movementFluid motion: 13-50msFrames p/second: 24-60fps
Prototype hand movementFluid motion: 13msFrames p/second: 78fps
@IITSEC NTSAToday
System Architecture - Visualization
11
Helps attain high measures of presence, or the feeling of “being there.”
@IITSEC NTSAToday
System Architecture - Game Engine
12
Unity is the game engine and integration point for the prototype.
@IITSEC NTSAToday
Interactivity - Virtual Environment
13
@IITSEC NTSAToday
Interactivity – Equipment Model
14
@IITSEC NTSAToday
Interactivity - Hands Articulation
15
@IITSEC NTSAToday
Interactivity - Pointing
16
@IITSEC NTSAToday
Interactivity - Pinching
17
@IITSEC NTSAToday
Interactivity - Exercise Progression
18
@IITSEC NTSAToday
Benefits – High-Order Cognitive Skills
19
Application of high-order cognitive activities Apply, Analyze, and Evaluate
Immediate application of procedural knowledge Consistent and constant feedback
during procedure
Prototype
@IITSEC NTSAToday
Benefits – Low-Order Cognitive Skills
20
Application of low-order cognition activities Remember, understand
Why? It’s a virtual instruction manual 3D collapsed\exploded diagram Parts name callouts using gaze
Prototype
@IITSEC NTSAToday
Benefits – Psychomotor Skills
21
Imitation of tool activity improves skilled movements Psychomotor Skills (by rehearsing motions) Visuospatial Skills (by rehearsing motions in
realistic environment) Visual correlation of real-to-virtual hand
movement creates the illusion of “being there”
Hand movement and head movementcreates the illusion the body in VR space
Prototype
@IITSEC NTSAToday
Experimental Design - Environment
22
OculusLeap MotionAudio
Starting location
Head tracker
Alienware PC5 Feet
1in. 2ft.
Gesture frame
@IITSEC NTSAToday
Experimental Design – Sample Frame
23
Participants were highly familiar and experienced with the helicopter mission kit (HMK).
Key: pn = participant code
@IITSEC NTSAToday
Experimental Design – Immersive Tendencies Questionnaire (ITQ)
24
Participants reported low to moderately-low immersive tendencies.
Key: pn = participant code Scale 1 = low tendency, 7 = high tendency
@IITSEC NTSAToday
Experimental Design – Light Bulb Task
Start Locate part 1
Remove 7 fasteners
Decouple part 1
25
Locate part 2
Remove 2 fasteners
Decouple part 2
@IITSEC NTSAToday
Results – Presence Questionnaire (PQ)
26
Participants reported high to moderately-high levels of presence.Recall that ITQ scores are highly correlated to PQ scores.
Key: 1 = Scale reversed
@IITSEC NTSAToday
Results - Thinking Out Loud
27
Key: tn = Time scale
Mostly positive experiences reported for Realism.Some positive and negative experiences reported for Interface Quality.
@IITSEC NTSAToday
Results – Presence Discussion
Low ITQ outcomes are highly correlated to low PQ outcomes, and vice-versa. However, participants reported low ITQ outcomes and high PQ outcomes.
1. Would the same participants report a low PQ outcome without virtual hands?
2. Would a larger and more diverse participant sample report differently?
28
@IITSEC NTSAToday
Results – Skills Discussion
Regarding the efficacy of cognitive and psychomotor skills training.
1. Does use of virtual hands training provide better transfer of cognitive skills than non-virtual hands implementations?
2. Does use of virtual hands provide better transfer of psychomotor skills than mental practice?
29
@IITSEC NTSAToday
Future Research
1. How does tool simulation effect cognitive and psychomotor tasks?
2. How does eye movement support measurement of mastery?
3. How can exercise construction be automated?
4. How may collaboration be supported across realities?
30
@IITSEC NTSAToday
Questions
Thank you for your time!
31
@IITSEC NTSAToday
Contacts
Corey NorrisDirector, Software [email protected]
Ron PunakoSenior Software [email protected]
32