Colloquium

Gesture-based interaction for learning: time to make the dream a reality_1288 86..89

Erol Ozcelik and Gokhan Sengul

Address for correspondence: Dr Erol Ozcelik, Computer Engineering Department, Atilim University, X Ankara 06,Turkey. Email: eozcelik@atilim.edu.tr

IntroductionResearch studies have shown that individuals who make hand gestures learn better than the oneswho do not (eg, Alibali & Goldin-Meadow, 1993; Broaders, Cook, Mitchell & Goldin-Meadow,2007). For instance, when children use their hands while they are explaining how they solvemathematical equivalence problems (eg, 6 + 4 +5 = _ + 5), they perform better in post-tests(Broaders et al, 2007). Comprehension and, consequently, memory are improved by these acts(Stevanoni & Salmon, 2005). In addition, gestures facilitate deep and long-lasting learning(Cutica & Bucciarelli, 2008).

Several proposals have been put forward to explain why gestures enhance learning. To begin with,Beilock and Goldin-Meadow (2010) have suggested that including motor actions into mentalrepresentations is responsible for performance improvements. Learners may produce integratedand richer representations when they describe a procedure with gestures (Alibali & Goldin-Meadow, 1993; McNeill, 1992). By means of gestures, people can externalise their thoughts(Clark, 1999), and as a result, more cognitive resources become available for learning from thelimited resources of the mind (Goldin-Meadow & Wagner, 2005).

The embodied cognition theory suggests that the body shapes the mind (Anderson, 2003). Forinstance, pointing by hands reduces the cost associated with maintaining information in ourworking memory (Ballard, Hayhoe, Pook & Rao, 1997). Memory for action events (eg, knockingon the table) is better when the events are performed by the subjects than when they are read orheard (Cohen, 1983). Engelkamp and Zimmer (1989) stated that the motor system is responsiblefor this effect since memory for self-performed events is better than that of experimenter-performed events and imagined events. To support this proposal, a functional magnetic resonanceimaging study demonstrates that the pre-motor cortex is activated when words are encodedthrough gestures (Macedonia, Müller & Friederici, 2010).

Gesture-based interaction in educationIn short, all these studies suggest that gestures enhance learning. In support of this assertion, theHorizon Report (Johnson, Smith, Willis, Levine & Haywood, 2011) identified gesture-based com-puting as an emerging technology that has a great potential to influence education in the nearfuture by providing a novel form of interaction, expression and activity. However, few studies haveemployed gesture-based interaction in education. Hao et al (2010) proposed a vision-basedmotion game detecting gestures of individuals to teach how to write Chinese characters tonon-native Chinese language learners. The results showed that all participants perform well inthe game. Yet, a majority of the learners make a complaint about their frustration with theinability of the programme to sense the movements of their body accurately. In another study,Anastopoulou, Sharples and Baber (2011) have investigated how learners interpret velocity–timeand distance–time graphs formed through their hand movements. For this purpose, they used acommercial motion tracker, which computes the orientation and position of a small receiver

British Journal of Educational Technology Vol 43 No 3 2012 E86–E89doi:10.1111/j.1467-8535.2012.01288.x

© 2012 The Authors. British Journal of Educational Technology © 2012 BERA. Published by Blackwell Publishing, 9600 Garsington Road, OxfordOX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.

attached to the participant’s arm. The results indicate that physically manipulating the graphs bygestures is more effective than passively watching a teacher carry out the right procedure.However, there are two drawbacks regarding the technology used in that study: one is that theequipment is expensive; the other flaw is that the wired sensor employed for the purpose of motiondetection causes the participants’ discomfort by limiting their physical movement.

Taken together, the technologies utilised in gesture-based interaction for learning have limita-tions to accurately detect the gestures made by individuals in a comfortable environment. A newtechnology called the time-of-flight (TOF) camera has the potential to provide a natural andeffective human–computer interaction. The TOF cameras provide not only traditional red, greenand blue images but also depth information (ie, the distance between the measured object and thesensor) for each pixel. This new depth information enables segmenting objects in the foregroundfrom the background in a robust and fast way. On the other hand, intensity information fromtraditional optical cameras is easily affected by lighting, colour, reflection and shadow, which arealways present in normal scenarios (Guomundsson et al, 2010).

Another technology that can be used for gesture-based interaction is the structured-light three-dimensional (3-D) sensor as an input device. A particular example of this sensor is the Microsoft’sKinect, in which patterns of light are projected onto objects and the distortion of the pattern bythe object’s surface is then analysed. Microsoft adapted this sensor to game technology so that

Figure 1: A screenshot of the developed simulation: the user can see depth information in the bottom right panel

Colloquium E87

© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.

users can play games by using their hands and bodies without using a physical controller, suchas a joystick. Furthermore, open libraries (eg, OpenNI Framework) enable programmers todevelop software that recognise gestures in real time. It is also possible to create 3-D games andsimulations by using game engines (eg, Unity).

Proof of conceptIn our laboratory, we tested the feasibility of our proposal by creating a simulation for teaching3-D vectors in physics by using Kinect (Microsoft Corporation, Redmond, WA, USA), OpenNI(Primesense Ltd., Tel Aviv, Israel) and Unity (Unity Technologies, San Francisco, CA, USA) (seeFigure 1). By means of gestures, learners can draw 3-D vectors by controlling the full bodymotion of the avatar in the virtual environment. For this purpose, the individual sets the startposition of the vector by holding the hand still for 3 seconds. When the other hand is held still,too, then the end of the vector in 3-D space is determined and the vector is drawn on the screen.The learner can see the sum of these vectors. Our laboratory tests reveal that our simulation iscapable of detecting gestures accurately and in real time. What is more, since no wires or devicesare attached to the individuals, they can manoeuvre comfortably in such an environment. Asstated by the subjects who participated in the tests, it was enjoyable to control the avatar bymoving their hands and bodies and to draw vectors in the virtual world.

ConclusionBy using 3-D sensors and other related software, it is now possible to develop simulations andgames that are controlled accurately and comfortably by students’ gestures. The authors assertthat when students use gestures while trying to understand a concept, learning is enhanced. It isexpected that simulation environments can be made more flexible to use and cheaper to obtain bythe help of these new technologies. In this respect, these environments can be made moreentertaining and engaging when learners are able to use their hands and bodies. New technolo-gies, such as the ones proposed in this article, can provide opportunities for educators to createnew forms of interaction with users.

AcknowledgementsThis work was partially supported by Atilim University, ATU-LAP-C-1112-11. We want to thankKagan Oktem and Batuhan Erol for the 3D visualizations. We also wish to express our sincerethanks to Cengiz Acarturk and Payam Danesh for their valuable comments on an earlier versionof this manuscript.

ReferencesAlibali, M. W. & Goldin-Meadow, S. (1993). Gesture–speech mismatch and mechanisms of learning:

what the hands reveal about a child’s state of mind. Cognitive Psychology, 25, 4, 468–523.Anastopoulou, S., Sharples, M. & Baber, C. (2011). An evaluation of multimodel interactions with technol-

ogy while learning science concepts. British Journal of Educational Technology, 42, 2, 266–290.Anderson, M. L. (2003). Embodied cognition: a field guide. Artificial Intelligence, 149, 1, 91–130.Ballard, D. H., Hayhoe, M. M., Pook, P. K. & Rao, R. P. (1997). Deictic codes for the embodiment of cognition.

Behavioral and Brain Sciences, 20, 723–767.Beilock, S. L. & Goldin-Meadow, S. (2010). Gesture changes thought by grounding it in action. Psychological

Science, 21, 1605–1610.Broaders, S., Cook, S. W., Mitchell, Z. & Goldin-Meadow, S. (2007). Making children gesture brings out

implicit knowledge and leads to learning. Journal of Experimental Psychology: General, 136, 539–550.Clark, A. (1999). An embodied cognitive science? Trends in Cognitive Sciences, 3, 345–351.Cohen, R. L. (1983). The effect of encoding variables on the free recall of words and action events. Memory

and Cognition, 11, 575–582.Cutica, I. & Bucciarelli, M. (2008). The deep versus the shallow: effects of co-speech gestures in learning

from discourse. Cognitive Science, 32, 921–935.Engelkamp, J. & Zimmer, H. D. (1989). Memory for action events: a new field of research. Psychological

Research, 51, 153–157.

E88 British Journal of Educational Technology Vol 43 No 3 2012

© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.

Goldin-Meadow, S. & Wagner, S. (2005). How our hands help us learn. Trends in Cognitive Sciences, 9,234–240.

Guomundsson, S. A., Pardas, M., Casas, J. R., Sveinsson, J. R., Aanaes, H. & Larsen, R. (2010). Improved 3Dreconstruction in smart-room environments using ToF imaging. Computer Vision and Image Understanding,114, 1376–1384.

Hao, Y., Hong, J. C., Jong, J. T., Hwang, M. Y., Su, C. Y. & Yang, J. S. (2010). Non-native Chinese languagelearners’ attitudes towards online vision-based motion games. British Journal of Educational Technology,41, 6, 1043–1053.

Johnson, L., Smith, R., Willis, H., Levine, A. & Haywood, K. (2011). The 2011 horizon report. Austin, TX:The New Media Consortium.

Macedonia, M., Müller, K. & Friederici, A. D. (2010). The impact of iconic gestures on foreign language wordlearning and its neural substrate. Human Brain Mapping, 32, 982–998.

McNeill, D. (1992). Hand and mind. Chicago, IL: University of Chicago Press.Stevanoni, E. & Salmon, K. (2005). Giving memory a hand: instructing children to gesture enhances their

event recall. Journal of Nonverbal Behavior, 29, 217–233.

Colloquium E89

© 2012 The Authors. British Journal of Educational Technology © 2012 BERA.