an interactive tangram game for autistic children · his work explores the usage of a social robot...

An Interactive Tangram Game For Autistic Children

Beatriz Vila Bernardo

Thesis to obtain the Master of Science Degree in

Information Systems and Computer Engineering

Supervisors: Prof. Dra. Ana Maria Severino de Almeida e PaivaProf. Dr. Francisco António Chaves Saraiva de Melo

Examination Committee

Chairperson: Prof. Dr. Miguel Nuno Dias Alves Pupo CorreiaSupervisor: Prof. Dr. Francisco António Chaves Saraiva de Melo

Member of the Committee: Prof. Dra. Teresa Isabel Lopes Romão

June 2016

Abstract

This work explores the usage of a social robot as an assistive agent during therapy sessions,

in order to assist children with Autism Spectrum Disorder (ASD) through a Tangram puzzle

game. Previous works showed that children with ASD express enthusiasm when interacting with

robots and other technologies. Also, these tools have been used during therapy to help them

overcome their social difficulties. Our aim was to transform the original Tangram, which is played

in therapy sessions of children with ASD, into an engaging tablet game in the same environment.

To achieve this, we included a humanoid robot - NAO.

The game has two conditions: the Tutor Mode and the Peer Mode. In the first condition, the robot

gives graded cueing feedback whenever the child experiences difficulties during the game. It is

directed for children who have difficulty playing this type of games, and it allows us to analyze the

evolution and the concentration of the child in a certain task. In the second condition, the robot

plays with the child in turn-taking: one player places one piece and then the other places another,

and so forth until the game finishes. This condition enables the study of the child’s capacity for

taking turns and robot’s efficiency to establish turns.

Eight children with ASD participated in this study, in order to allow the analysis of the evolution

of their turn-taking skills, performance, and attention on the game, and also the effects of the

robot during therapy sessions. These participants presented various symptoms, so we could not

compare them with each other. Each participant was studied as a single. The results indicated

that in the Tutor Mode the robot was capable of maintaining child’s attention on the game and

to help most of the times it was necessary. In the Peer Mode, the robot also stimulated child’s

concentration on the game and was able to establish turns for the majority of the participants.

External interventions decreased over time but did not disappear completely. This concludes that

the presence of a therapist is crucial in order to have an optimal interaction. However, this study

shows that the targeted children can benefit from robot based therapies.

Keywords: Social Robot , Autism Spectrum Disorder , Children , Therapeutic Intervention ,

Human-Robot Interaction

3

Resumo

Este trabalho explora a utilizacao de um robo social como uma ferramenta auxiliar em sessoes

de terapia, para auxiliar criancas com Perturbacoes do Espectro do Autismo (PEAs) du-

rante o jogo Tangram. Outros trabalhos de investigacao mostraram que as criancas com PEAs

expressam entusiasmo ao interagir com robos e outras tecnologias. Para alem disso, estas fer-

ramentas tem sido utilizadas durante a terapia para ajuda-los a superar as suas dificuldades

sociais. O nosso objectivo era transformar o Tangram original, que e jogado em sessoes de ter-

apia de criancas com PEAs, num estimulante jogo de tablet no mesmo ambiente. Para conseguir

isso, incluımos um robo humanoide - NAO.

O jogo tem duas condicoes: o Modo Tutor e o Modo Colega. Na primeira condicao, o robo

fornece pistas acerca do jogo quando a crianca experiencia dificuldades. Este modo e direc-

cionado para criancas que tem dificuldade a jogar este tipo de jogos, e permite analisar a

evolucao e a concentracao da crianca em determinada tarefa. Na segunda condicao, o robo

joga com a crianca em modo de troca de turnos, um jogador coloca uma peca e o outro coloca

outra, e assim por diante ate ao termino do jogo. Este modo proporciona o estudo da capacidade

da crianca a respeitar a troca de turnos e a eficiencia do robo a estabelece-los.

Oito criancas com PEAs participaram neste estudo, com o objectivo de analisar a evolucao das

suas capacidades de troca de turno, assim como a sua performance e atencao no jogo, e ainda

os efeitos do robo durante a terapia. Estes participantes apresentavam sintomas variados, por

isso nao era possıvel serem comparados entre si. Cada participante foi analisado como um so.

Os resultados indicam que no Modo Tutor o robo foi capaz de manter a atencao da crianca no

jogo e de ajudar na maior parte das vezes que foi necessario. No Modo Colega o robo tambem

conseguiu manter o foco da crianca no jogo e foi capaz de estabelecer turnos para a maior

parte dos participantes. O numero de intervencoes externas decresceu ao longo das sessoes,

no entanto nao desapareceram por completo. Isto conclui que a presenca de um terapeuta e

fundamental para haver uma interaccao optima. Contudo este estudo mostra que as criancas

alvo podem beneficiar do uso de robos para fins terapeuticos.

5

6

Keywords: Robo Social , Perturbacoes do Espectro do Autismo , Criancas , Intervencao

Terapeutica , Interaccao Robo-Humano

Acknowledgements

First of all, I would like to thank my advisers, Prof. Ana Paiva and Prof. Francisco Melo, for

their guidance during this thesis and the opportunity to follow my own ideas.

To my parents and sister, for always believing in me (especially in the demotivating moments),

and for all the support they gave me throughout this long and exhausting phase. To all my friends

who were always present, thank you for the encouragement and support.

To Manuel, for always knowing what to say at the right time, for encouraging me to strive towards

my goals and dreams, and for being my rock.

I would like to thank all my colleagues from GAIPS, Filipa, Tiago, Carla, Patrıcia and so on, that

somehow helped and contributed to the realization of this work.

And finally, I would like to thank the professionals from HGO, CADIn, and Escola Francisco de

Arruda involved in this work for their collaboration and help. And all parents and children who

participated in the studies presented in this thesis, without them none of this would have been

possible.

This work was partially supported by the Portuguese Fundacao para a Ciencia e a Tecnologia and

the Carnegie Mellon Portugal Program and its Information and Communications Technologies

Institute, under project CMUP-ERI/HCI/0051/2013.

7

Contents

Abstract 3

Resumo 5

Acknowledgements 7

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Document Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Background 5

2.1 Autism Spectrum Disorder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 Main characteristics and diagnosis . . . . . . . . . . . . . . . . . . . . . . . 6

2.1.2 Treatments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2 Taking Turns and ASD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Related Work 11

3.1 Screen-Based Technology and Games for Children with ASD . . . . . . . . . . . . 11

3.1.1 Collaborative Tabletop Games . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1.2 ECHOES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Social Robots and Children with ASD . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.2.1 The Aurora Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

9

3.2.2 Tito . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.2.3 Keepon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2.4 Pleo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.2.5 Bubble Blower . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.2.6 NAO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4 Architecture 27

4.1 Tangram Puzzle Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.1 Game Functioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.1.2 Game Difficulty Management . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.3 Initial Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.2 Social Robot - NAO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.2.2 Integration Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.3 Tutor Mode - Prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.3.1 Prompt Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.4 Peer Mode - Turn-Taking Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.4.1 Taking turns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.4.2 Cooperative Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

4.5 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.6 Interface for Tests with Therapist . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5 Evaluation 45

5.1 The Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.2 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.2.1 Single Subject Research Design . . . . . . . . . . . . . . . . . . . . . . . . 47

5.2.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.2.3 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.3 The Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

10

5.3.1 Tutor Condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.3.2 Peer Condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

6 Results 55

6.1 Tutor Condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.1.1 Participant 1 - J.F. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.1.2 Discussion of the Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6.2 Peer Condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

6.2.1 Participant 2 - M.A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

6.2.2 Participant 3 - J.N. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

6.2.3 Participant 4 - H.R. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

6.2.4 Participant 5 - A.S. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

6.2.5 Participant 6 - A.J. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

6.2.6 Participant 7 - M.C. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

6.2.7 Participant 8 - D.B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

6.2.8 Discussion of the Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

7 Conclusions 87

7.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Bibliography 91

Appendices 97

A Parental Consent Form 97

B Child’s Profile 99

C Questionnaires 101

11

List of Tables

4.1 NAO basic technical specifications (Aldebaran Robotics, 2016). . . . . . . . . . . . 36

6.2 Participant M.A.: Percentage of times he played in his turn, times he helped the

robot and the number of times he played out of turn. . . . . . . . . . . . . . . . . . 64

6.3 Participant J.N.: Percentage of times he played in his turn, times he helped the


6.4 Participant H.R.: Percentage of times he played in his turn, times he helped the


6.5 Participant A.S.: Percentage of times he played in his turn, times he helped the


6.6 Participant A.J.: Percentage of times he played in his turn, times he helped the


6.7 Participant M.C.: Percentage of times he played in his turn, times he helped the


6.8 Participant D.B.: Percentage of times he played in his turn, times he helped the


6.9 Time spent (in minutes) in each session for every participant . . . . . . . . . . . . 86

13

List of Figures

3.1 Cooperative gestures on the CPG . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2 SIDES interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.3 Child interacting with Andy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.4 Labo-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.5 Robota . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.6 KASPAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.7 Tito, the robot mediator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.8 Keepon, the interactive robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.9 The mobile robot used in the experiment . . . . . . . . . . . . . . . . . . . . . . . . 24

4.10 Original Tangram puzzle game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.11 Easy Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.12 Medium Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.13 Hard Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.14 Interface with two pieces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.15 NAO robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.16 Thalamus Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.17 Start screen for Tutor and Peer Mode . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.18 Start screen with game settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.19 Visual feedback - Fireworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.20 Screen with name input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.21 Choregraphe - NAO’s programing interface . . . . . . . . . . . . . . . . . . . . . . 43

15

5.22 Trial’s setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.23 Participant J.F.: Total time (in seconds) to complete each puzzle, the average total

time (in seconds) spent per puzzle, and the average time (in seconds) to place a

piece per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.24 Participant J.F.: Average number per puzzle of: wrong or close placement of

pieces, and drags of pieces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.25 Participant J.F.: Average number of external interventions (from the teacher) and

helps (from the robot), per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.26 Participant J.F.: Percentages of eye gaze per session. The baseline and final

session have 2 data points each. The first one corresponds to the original game

and the last one to the tablet game. . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6.27 Participant J.F.: Average number per puzzle of gestures and vocalizations towards

the robot or about the game. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6.28 Participant J.F.: This plot presents the (1) number of received helps, (2) the number

of times already placed pieces were destroyed, (3) the number of times the child

(wrongly and correctly) finger-pointed to a spot (after teacher has asked), and (4)

the number of times of wrong and correct placement of pieces. . . . . . . . . . . . 57

6.29 Participant M.A.: Total time (in seconds) to complete each puzzle, the average total

time (in seconds) spent per puzzle, and the average time (in seconds) of each child

turn per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

6.30 Participant M.A.: Average number per puzzle of: wrong or close placement of

pieces, and drags and rotates of pieces. . . . . . . . . . . . . . . . . . . . . . . . . 62

6.31 Participant M.A.: Average number of external interventions (from the therapist) and


6.32 Participant M.A.: Percentages of eye gaze per session. . . . . . . . . . . . . . . . 63

6.33 Participant M.A.: Average number per puzzle of gestures and vocalizations to-

wards the robot or about the game. . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

6.34 Participant J.N.: Total time (in seconds) to complete each puzzle, the average total


turn per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

6.35 Participant J.N.: Average number per puzzle of: wrong or close placement of


16

6.36 Participant J.N.: Average number of external interventions (from the therapist) and


6.37 Participant J.N.: Percentages of eye gaze per session. . . . . . . . . . . . . . . . . 66

6.38 Participant J.N.: Average number per puzzle of gestures and vocalizations towards


6.39 Participant H.R.: Total time (in seconds) to complete each puzzle, the average total


turn per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

6.40 Participant H.R.: Average number per puzzle of: wrong or close placement of


6.41 Participant H.R.: Average number of external interventions (from the therapist) and


6.42 Participant H.R.: Percentages of eye gaze per session. . . . . . . . . . . . . . . . 69

6.43 Participant H.R.: Average number per puzzle of gestures and vocalizations to-


6.44 Participant A.S.: Total time (in seconds) to complete each puzzle, the average total


turn per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6.45 Participant A.S.: Average number per puzzle of: wrong or close placement of


6.46 Participant A.S.: Average number of external interventions (from the teacher) and


6.47 Participant A.S.: Percentages of eye gaze per session. . . . . . . . . . . . . . . . . 73

6.48 Participant A.S.: Average number per puzzle of gestures and vocalizations towards


6.49 Participant A.J.: Total time (in seconds) to complete each puzzle, the average total


turn per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6.50 Participant A.J.: Average number per puzzle of: wrong or close placement of


6.51 Participant A.J.: Average number of external interventions (from the therapist) and


17

6.52 Participant A.J.: Percentages of eye gaze per session. . . . . . . . . . . . . . . . . 77

6.53 Participant A.J.: Average number per puzzle of gestures and vocalizations towards


6.54 Participant M.C.: Total time (in seconds) to complete each puzzle, the average total


turn per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

6.55 Participant M.C.: Average number per puzzle of: wrong or close placement of


6.56 Participant M.C.: Average number of external interventions (from the therapist) and


6.57 Participant M.C.: Percentages of eye gaze per session. . . . . . . . . . . . . . . . 79

6.58 Participant M.C.: Average number per puzzle of gestures and vocalizations to-


6.59 Participant D.B.: Total time (in seconds) to complete each puzzle, the average total


turn per puzzle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

6.60 Participant D.B.: Average number per puzzle of: wrong or close placement of


6.61 Participant D.B.: Average number of external interventions (from the therapist) and


6.62 Participant D.B.: Percentages of eye gaze per session. . . . . . . . . . . . . . . . . 82

6.63 Participant D.B.: Average number per puzzle of gestures and vocalizations towards


A.64 Consent form given to the participants’ parents . . . . . . . . . . . . . . . . . . . . 98

B.65 Form given to the therapists to fill for each participant . . . . . . . . . . . . . . . . 100

C.66 Questionnaire to be answered by the therapist regarding the outcome of the inter-

action between child, therapist, and tablet . . . . . . . . . . . . . . . . . . . . . . . 102

C.67 Questionnaire to be answered by the therapist regarding the outcome of the inter-

action between child, robot, and tablet . . . . . . . . . . . . . . . . . . . . . . . . . 103

C.68 Questionnaire to be answered by the therapist regarding child’s first impressions

of the robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

18

Chapter 1

Introduction

For several years, robots have been used as assistive devices in the aid of social (Gockley

et al., 2005), physical (Lo et al., 2010), cognitive (Cook et al., 2011) and other abilities

(Trafton et al., 2006). Also, multiple researches have been focusing in the use of robots in autism

therapy (Kim et al., 2013; Kozima et al., 2009; Robins et al., 2009). The interest of autistic children

in this kind of assistive tools has instigated the majority of the aforementioned works.

Nowadays, around 1 in each 68 children have Autism Spectrum Disorder (ASD) (Baio, 2014).

Autism is a complex behavioral disorder that typically manifests prior to age three, and exhibits

behavioral impairment in social interaction and social communication, and presence of repetitive

patterns of behavior, interests or activities (American Psychiatric Association, 2013). However,

people with autism may present difficulties at other levels, such as cognitive disabilities, avoiding

eye contact with others, diminished attention, and other deficits in pragmatic skills (e.g., turn

taking) (Miles & McCathren, 2005; The National Autistic Society, 2015).

In Autism Spectrum Disorder, the expression spectrum corresponds to the extensive range of

levels of impairment that individuals with ASD can have. Some children are moderately impaired

within few skills while others are extremely debilitated. There is no established cure for ASD

yet, although an early diagnosis and intensive behavioral therapy can reduce the symptoms and

increase children’s abilities (Miles & McCathren, 2005).

1.1 Motivation

So far, several studies have shown that children with autism can easily create affective bonds

1

2 CHAPTER 1. INTRODUCTION

with social robots (Kim et al., 2013; Kozima et al., 2009), due to their simple and predictable

characteristics. Thus, the interaction with robots tends to be less intimidating than with humans.

The Aurora Research Project is an excellent example of how robots can be integrated into therapy

sessions with autistic children (Robins et al., 2009). Dautenhahn et al. aimed at developing and

increasing communication and social interaction skills in children with ASD, through collaborative

and turn-taking games with different robots. KASPAR’s (Dautenhahn et al., 2009) (one of the

robots embedded in the Aurora Project) slightly human traits help developing social skills, and,

consequently, it promotes interaction between the child and the therapist. Further studies such

as Bubble Blower by Feil-Seifer & Mataric (2009), AIBO by Stanton et al. (2008) and Keepon by

Kozima et al. (2009) came to the conclusion that robots can assume the role of catalyst for social

interaction between children with autism and co-present others.

Furthermore, some studies such as ECHOES (Bernardini et al., 2014), The Aurora Research

(Dautenhahn et al., 2009) and SIDES (Piper et al., 2006) focused on developing children’s in-

teraction skills. As mentioned earlier, autistic children exhibit impairment in social interactions

and therefore there are certain behaviors that they can not reproduce. For this purpose, the

researchers studied turn-taking and imitation games between the child and the robot or another

child. The use of robots in these cases induced a predictable and controlled environment, favor-

ing a less frightening situation to an autistic child. Additionally, in these studies, some children

exhibited social behaviors such as joint attention, eye gaze, spontaneous imitation and increased

engagement in tasks (Bernardini et al., 2014; Dautenhahn et al., 2009).

Children with ASD are fascinated by computers, tablets and other electronic devices, as their

answers are predictable and consistent. Davis et al. (2007) reported that children with ASD

interacting with touchscreens during the game easily ignored any external distractions. Other

electronic devices such as tabletop technology can be used during therapy to promote collabora-

tion (e.g., through touch games between two or more children). In Battocchi et al. (2009) game,

children discuss the best solution in order to achieve a common goal. If technology succeeds in

engaging children with ASDs in a variety of interactions, then it should be used to contribute to

the development of these children’s skills.

Puzzles are one of the games used during therapy with autistic children, aiming at stimulating

(1) the children’s intellectual capabilities, (2) their attention span, (3) motor coordination, and (4)

visuospatial skills (Kohanova & Ochodnicanova, 2014; Siew & Chong, 2014). This work also

leverages a puzzle game in order to improve the same ASD difficulties. The chosen puzzle was

Tangram1, a Chinese game consisting of seven pieces with different geometric shapes. It is

normally played with wooden pieces, and the goal is to create numerous different silhouettes

1http://en.wikipedia.org/wiki/Tangram

1.2. CONTRIBUTIONS 3

with all of them (there are countless possible puzzles). According to the therapists, it is a game

commonly played during therapy sessions by autistic children starting from 3 years old, and it

attracts them due to its geometrical forms. So we decided to develop a tablet version of the

Tangram puzzle, together with the social robot (NAO1). The robot was programmed to function

as a tutor - helping the children through the game, and a peer - engaging the children in a turn-

taking game.

1.2 Contributions

The studies in this area have increased significantly over the years, and in general, have shown

good results in the usage of robots during the autistic children therapy sessions, especially in

the social interaction skills. Leveraging on the interest of individuals with ASD, on both areas of

social robots and computer games, the present work intended to extend this field of research.

Our goal was to answer the question: How can a social robot may be integrated into therapeutic

intervention with children with ASD as a support tool to engage children in a certain game? To

do so, we:

• Developed a Tangram game in a tablet with a simple interface to be played by autistic

children;

• Developed a social robot that aids the children with ASD in the game - Tutor;

• Developed a social robot that plays with the children a cooperative turn-taking Tangram

game - Peer;

• Evaluated the affective attributes towards the robot, concentration and the game perfor-

mance in each approach.

In order to this game be playable and attractive for most children of the spectrum, it has multiple

levels of difficulty and piece rotation modes. Thus, the game is played by children with few or

many cognitive and motor impairments. The Tutor Condition was developed with the aim of

helping the children who have more difficulties in this game. And the Peer Condition for children

who have mastered the Tangram but have trouble in taking turns during a game.

Our aim was never to use the robot to replace the therapist within therapy, but to serve as an

assistive tool. So we expected to achieve the following purposes: (1) The performance of the

1https://www.aldebaran.com/en/cool-robots/nao

4 CHAPTER 1. INTRODUCTION

children in the Tangram game with a robot is similar than with a therapist ; (2) In Tutor Condition,

the child is as or more autonomous playing Tangram than when playing with a therapist ; and (3)

In Peer Condition, the child respects his/her turn to play as much as with the therapist. Thus,

a single-subject study was conducted at the Hospital Garcia de Orta, CADIn and Francisco de

Arruda school. We evaluated 8 participants individually during a number of sessions which varied

for each participant and compared with the baseline and final results. The baseline measurement

was performed before the robot sessions and consisted of a session with only the child and the

therapist to play together the same Tangram game on the tablet. The therapist, as the robot,

played both the roles of tutor and peer. In the best case, the results with the robot would be close

to the results with the therapist.

1.3 Document Organization

The rest of the document is organized as follows. The concept of Autism Spectrum Disorder

and the main issues related to this work are presented in Chapter 2. Chapter 3 presents the

work related to the main themes of this work. Additionally, Chapter 4 introduces our proposed

architecture. The Chapter 5 presents the evaluation methodology to analyze the suitability of

our work. Chapter 6 presents and discusses the results of each participant. Finally, the main

conclusions of this report, as well as the discussion of some possible future work directions, are

expressed in the Chapter 7.

Chapter 2

Background

This section introduces the concept of Autism Spectrum Disorder (ASD), the symptoms pre-

sented by the individuals, existing methodologies of treatment, and finally, discusses a

difficulty in social interaction relevant for this work.

2.1 Autism Spectrum Disorder

The Autism term was first used by Eugen Bleuler in 1911 to denote one of the symptoms of

schizophrenia Moskowitz & Heim (2011). Although, only in 1943, Leo Kanner defined autism as

a rare syndrome of affective contact disorder, without any connection to schizophrenia (Kanner,

1943). Leo Kanner studied eleven autistic children over several years and described them as

being socially indifferent, with difficulty in communication and resistant to routine changes. A

year later, Hans Asperger reported a group of children that had a good speech but only used it

for monologs, and lacked non-verbal communication skills (Klin, 2006; Wing, 1997). In contrast

to Kanner’s patients, these children were unreserved and developed highly grammatical speech.

Later, this disorder would be called Asperger Syndrome. Over time, autism concept has under-

gone changes as the knowledge on the disease deepened, through several studies. Currently, it

is recognized as part of a spectrum disorder (American Psychiatric Association, 2013; Klin, 2006;

Lindgren & Doobay, 2011).

5

6 CHAPTER 2. BACKGROUND

2.1.1 Main characteristics and diagnosis

Autism spectrum disorder (ASD) is a complex neurodevelopmental disability that begins early in

life and influences how an individual learns, interacts, and communicates with others (American

Psychiatric Association, 2013). According to the American Psychiatric Association (2013), ap-

proximately 1% of the population has ASD and is diagnosed four times more often in males than

in females. Presently, scientists do not know what causes autism, but it is believed that genetic

mutations and other environmental variables (such as low birth weight, advanced parental age,

or fetal exposure to Valproate1 may be the origin (American Psychiatric Association, 2013).

There is no medical test (e.g., a blood test) to diagnose ASD. To make a diagnosis, the specialists

have to analyze the child’s behavior and development. For most children, the symptoms begin

to appear by 12 months, but may be seen earlier if developmental delays are severe. An early

detection and intervention can greatly enhance outcomes, so it is important to search for these

symptoms as soon as possible (American Psychiatric Association, 2013; Lindgren & Doobay,

2011).

According to the American Psychiatric Association (2013), people with this disorder exhibit so-

cial communication and social interaction impairment, and also restricted/repetitive patterns of

behavior, interests, or activities:

• Absence of or Impairment in Social Communication and Social Interaction: Delay in

the improvement of spoken language accompanied by the misreading of nonverbal lan-

guage (such as gestures, facial expressions and volume of voice), and the inability to cre-

ate and maintaining social relationships. ASD patients show signals of deficits in social-

emotional reciprocity (i.e., the capacity to engage and share thoughts and feelings with

other people) that can range from unusual social approach; to reduced sharing of interests,

emotions, or affect; to social aloofness (i.e., children are indifferent in all interactions with

other people). Also, these individuals present deficits in nonverbal communication used for

social interaction, ranging, for example, from ineffectively integrated verbal and nonverbal

communication; to difficulties of maintaining eye contact and understanding body language

or deficits in comprehension and use of gestures; to an absence of facial expressions and

nonverbal communication. Finally, people with ASD show deficits in creating, preserving,

and understanding relationships, ranging, for instance, from difficulties adjusting behavior

to suit different social contexts; to trouble to share imaginative play or in making friends; to

absence of interest in peers.

This trouble with social relationships includes the difficulty of establishing joint attention1Valproate is a medication used to treat bipolar disorder and epilepsy and to prevent migraine headaches

2.1. AUTISM SPECTRUM DISORDER 7

(i.e., child’s capacity to shift gaze between people and objects; pointing, showing, or bring-

ing objects to share emotions and engage in turn-taking interactions, essential to collab-

orative tasks) and reduced or absent imitation of others’ behavior. Many individuals have

language deficits, going from a complete absence of speech through language delays, poor

perception of speech, echoed speech (echolalia), or overly literal language. Even when for-

mal language abilities (e.g., vocabulary and grammar) are intact, their speech in reciprocal

social communication is debilitated.

• Repetitive and Stereotyped Activities: Include basic motor stereotypes (e.g., hand flap-

ping, finger flicking), repetitive utilization of objects (e.g., spinning or lining up objects), and

repetitive speech (e.g., echolalia or immediate repeating heard words; stereotyped use of

words, phrases, or prosodic patterns). Inflexible adherence to routines may be manifested

in the refusal to change or repetitive patterns of verbal or nonverbal behavior (e.g., repetitive

questioning). Some individuals exhibit highly restricted, focused interests that are unusual

in intensity (e.g., strong attachment to or preoccupation with unusual objects). A few fas-

cinations and routines may relate to apparent hyper- or hypo-reactivity to sensory input

manifested through extreme responses to particular sounds or textures, excessive touch-

ing of objects and visual interest with lights. In situations of stress (e.g., when their routines

are disturbed), children with autism spectrum disorders frequently present temper outbursts

or repetitive and stereotyped motor movements (sometimes even self-injurious behaviors).

In addition, children with ASD usually present other disabilities. Such as abnormalities of at-

tention (i.e., overly focused or easily distracted), intellectual disabilities, the co-occurrence of

developmental coordination disorder (disinterest in tasks that requires coordination skills, such

as ball sports, which will disturb test execution and function), and other motor deficits (e.g., odd

gait, clumsiness) (American Psychiatric Association, 2013). Briefly, people within the spectrum

differ greatly from each other, and often present dissimilar characteristics.

ASD is also characterized by the presence of positive symptoms. It involves the existence of un-

usual behaviors for the individuals’ age and development stages, such as prodigious memories,

great meticulousness, well developed visual-spatial skills, strong interest in certain topics, and

the desire to follow rules (Ozonoff et al., 2008).

Before the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (American Psychi-

atric Association, 2013), the ASD was divided into 4 types of autism (Lindgren & Doobay, 2011;

Ozonoff et al., 2008): Autistic Disorder, Asperger’s Syndrome, Childhood Disintegrative Disorder

(CDD) and Pervasive Developmental Disorder - Not Otherwise Specified (PDD-NOS). Currently,

Autism Spectrum Disorder encompasses all the disorders mentioned above.


The Autism Diagnostic Observation Schedule (ADOS) (Lord et al., 2012) is an instrument to ac-

curately assess and diagnose autism spectrum disorders. This method consists in scoring direct

observations of the child’s interactions and correlate the developmental level and age of the child.

There are four ADOS modules, the Modules 1 and 2 are targeted to children with a language level

of fewer than 48 months, the Modules 3 and 4 are designed for older children, adolescents, and

adults who can utilize complex sentences. The examiners observe the person’s behaviors and

coded them using a 0- to 3-point coding system (0 indicating that the behavior is not abnormal and

3 indicating that a behavior is abnormal and interferes with the child’s functioning) (Akshoomoff

et al., 2006).

2.1.2 Treatments

Autism spectrum disorder is not a degenerative disorder and currently there is no cure for Autism.

In general, people with lower levels of impairment may be better able to function individually. How-

ever, even these individuals may remain socially naive and vulnerable, experience difficulties co-

ordinating practical tasks without help, and are susceptible to anxiety and depression (American

Psychiatric Association, 2013). Nevertheless, early therapeutic intervention can greatly enhance

children’s development in order to expand their functional independence and improve their life

quality (The National Autistic Society, 2015). The main goal of ASD therapy is to improve cog-

nitive, language and social skills while minimizing fundamental deficits. The most effective treat-

ment may differ for each patient (Myers & Johnson, 2007). Some of the common approaches are

the Applied Behavior Analysis (ABA), Pivotal Response Therapy (PRT), Floortime, Treatment and

Education of Autistic and related Communication-handicapped Children (TEACCH), SCERTS, et

cetera (The National Autistic Society, 2015). In this Chapter, we focus on some of the approaches

that are relevant to this thesis.

Applied Behavior Analysis (ABA): applies the following principle: When a behavior is followed

by some sort of reward, the behavior is more likely to be repeated. So, whenever a child com-

pletes a task, the therapist reinforces him/her with a positive reward (e.g., through vocalizations

or offering an object of the child’s interest) (The National Autistic Society, 2015). ABA focuses on

the reliable measurement and target assessment of observable behavior within the home, school,

and community. ABA methods are used to increase and maintain adaptive behaviors, reduce ab-

normal behaviors or confining the conditions under which they happen, develop new abilities,

and generalize behaviors to new situations. Functional assessment is an accurate method of

collecting information that can be used to augment the effectiveness and efficiency of behavioral

2.2. TAKING TURNS AND ASD 9

support interventions. It includes defining a clear description of the problem behavior; recog-

nizing the precursors, outcomes, and other natural factors that maintain the behavior; creating

theories that specify the motivating function of the behavior; and gathering direct observational

data to test the hypothesis (Myers & Johnson, 2007).

Floortime: is a specific procedure to both follow the child’s natural emotional interests (lead),

and in the meantime, challenge the child towards an enhancement of the social, emotional, and

intellectual capacities. Parents and therapists engage the children by getting at their level (i.e., on

the floor) to play with them and thereby, helping children to maintain attention during interactions

and logical thinking (The National Autistic Society, 2015). This method emphasizes the critical

role of parents and other family members because of the importance of their emotional relation-

ships with the child. Thus, Floortime method attempt to facilitate emotional and cognitive growth,

and development (Myers & Johnson, 2007).

Treatment and Education of Autistic and related Communication-handicapped Children

(TEACCH): uses visual cues to teach skills making the activities simple and predictable. The

TEACCH method emphasizes structure including organization of the environment, predictable

and structured set of activities, schedules, and routines. There is a priority on both enhancing

skills of people with ASDs and adapting the environment to their deficits (Myers & Johnson,

2007).

SCERTS: The acronym SCERTS mentions the focus on Social Communication (SC), Emotional

Regulation (ER) and Transactional Support (TS) (Prizant et al., 2003). The SC component ad-

dresses the development of child’s capacity for joint attention and symbol use. The ER com-

ponent focuses on enhancing the abilities for mutual and self-regulation, and also to recover

from dysregulation. Finally, the TS component targets the development and implementation of

educational, familiar and interpersonal supports.

In most of the cases, multiple approaches may be needed during therapy, because they com-

plement each other. Also, there are medications that can help to alleviate some symptoms and

aggressive behaviors in people with ASD. For example, medicine might help with the inability to

focus, self-harm, seizures or depression (Lindgren & Doobay, 2011).

2.2 Taking Turns and ASD

Turn-taking is one of the most basic and important social skills required in everyday life. This

ability is fundamental when it comes to developing friendships, communicating with others and

playing games (Sacks et al., 1974). It is not simple to describe turn-taking for being an element


of social life so deeply implanted in common-sense practice. It is necessary to play games, for

managing traffic, to conduct interviews, to serving clients in establishments, and so on. However

is not an innate skill, it needs to be taught.

Autistic children are described to interrupt a speaker improperly or to remain in the speaker

(or the listener) role for too long (Baron-Cohen, 1988). Also, these children are reported to

have difficulties in initiating conversations and reply appropriately to peers. These are clear

evidence that some children with ASD have difficulties in taking turns. Nadel (2004) concluded

that imitation, turn-taking, and recognition of conventional social patterns is a basis for developing

social skills and understanding others’ intentions. Thus, this skill is important to be acquired for

these children in order to improve their social skills, and developing theory of mind1.

Concluding this section, anybody can easily perceive the potential advantages of turn-taking to

promote and develop social skills in autistic children. This work explores the dynamics of turn-

taking during engaging games using a robot to help autistic children with their impairments. In the

next chapter, some works that also address these issues are presented (Duquette et al., 2008;

Robins et al., 2009).

1People with ASDs do not have theory of mind, which is the ability to understand other people’s different beliefs(Baron-Cohen et al., 1985). Therefore, these individuals are unable to anticipate others’ actions, which frequently leadsto inappropriate reactions in social interactions.

Chapter 3

Related Work

This chapter presents the works related to different aspects of my M.Sc. thesis. Firstly, the

most relevant screen-based games for children with ASD are displayed in the Section 3.1.

Then, the emblematic social robots in this context used during therapy sessions are presented in

Section 3.2.

3.1 Screen-Based Technology and Games for Children with

ASD

This section presents works concerning the usage of screen-based games in therapy sessions

with autistic children. Firstly, three works that focus on promoting collaborative and cooperative

skills of children with ASD are presented: CPG (Battocchi et al., 2009), SIDES (Piper et al.,

2006) and Join-In Suite (Zancanaro et al., 2011). These are games with two or more children

that encourage the participants to discuss among themselves the best approach on how they

should proceed during the game. These skills are important for children with ASD to appropriately

behave in social situations. Finally, the ECHOES (Bernardini et al., 2014), a serious game with

an autonomous virtual agent, with the purpose of developing children’s social communication

skills, is presented.

These works are the motivation for this thesis. They demonstrate how screen-based games have

the power to engage and stimulate ASD children on them, in order to develop abilities. They also

show that despite being more challenging for these children, the games managed to encourage

them to overcome their disabilities. However, to make the interaction engaging and coherent, it

is important to provide visual and auditory feedback.

11

12 CHAPTER 3. RELATED WORK

beginning of the game are spread over the game surface inrandom positions; 2) a target picture shown in the upper partof the screen, depicting how the puzzle looks when correctlycompleted; 3) a solution area, positioned in the region ofthe surface proximal to participants and horizontally centred,where pieces have to be dragged to complete the puzzle. Thesolution area is marked by a rectangle with darker bordersand is divided in a number of invisible cells correspondingto the positions where puzzle pieces can be released. Thenumber of the cells is equal to the total number of pieces(Figure 1).The moves that players can perform are simple and clear. Apuzzle piece can be dragged to a new position, either to thesolution area, or away from it. Whenever a piece is releasedwithin the solution area, it anchors to the cell of the solutionarea closest to the releasing position. To remove a piece fromthe solution area, it is sufficient to drag it away and release itat any other position on the surface. Direct piece movementfrom one cell to another within the solution area is also pos-sible using the same pick up, drag and drop technique.A number of animations have been implemented to makethe game more motivating and fun. At the beginning ofthe game, an animation, comprising a short video and back-ground music, introduces the picture to be completed, whichis displayed in the solution area for about 15 seconds. Thepicture then divides into puzzle pieces, which are scatteredover the game surface, accompanied by a drum-roll sound.During the game, an animation is executed when a partici-pant tries to release a puzzle piece over another piece thatis already anchored to the solution area: the piece beingreleased jumps away with a spring sound and moves to arandom position on the surface. Once the last piece hasbeen correctly inserted in the solution area, another anima-tion starts, showing the completed picture together with thesentence ”Good job! Puzzle completed!” with triumphant,happy music. Several visual and auditory feedback signalshave been implemented to make the interaction more under-standable and interactive. Basic feedback includes differ-ent clicks for the picking up or the release of puzzle pieces.This provides confirmation, through the auditory modality,that the interaction with puzzle pieces was effective. A sec-ond level of auditory and visual feedback indicates correctand incorrect positioning of pieces: when a piece is releasedin an incorrect position on the solution area, an unpleasantbuzz is played and a red halo surrounds the piece until itis removed. A pleasant sounding beep and a green halo,which disappears after a few seconds, are associated withthe correct positioning of a piece within the solution area.The sounds used to indicate actions and to provide positiveand negative feedback were based on reports from the litera-ture concerning the effectiveness of audio feedback in regu-lating interaction for colocated collaborative interfaces [16,26]. Their suitability for the study population was verifiedduring pilot testing.

Implementation of Enforced CollaborationThe EC paradigm was implemented through a particular classof low-level actions called Cooperative Gestures [14], whichconstitute the basic elements of Cooperative Gesturing. Co-operative Gesturing is a technique for multi-user interaction

Figure 1. The CPG Interface

on single display shared interfaces that allows the system tointerpret the gestures of more than one user as contributingto a single, combined command [14]. Three particular coop-erative gestures have been used: joint-touch, joint-drag andjoint release. The implementation of these actions createsa play modality in which the simultaneous touch, drag andrelease by the two players is required in order to move onepuzzle piece from its position (inside or outside the solu-tion area). If a piece is being touched by one player only,the piece oscillates and a vibration sound is played and itsmovement is not permitted by the EC rule. If the other playerjoins his peer in the move, by touching the same piece, therestriction on the movement ceases, and the piece can bejointly dragged to a different position. If one of the two play-ers releases the fingers before the dragging has started, thepiece starts again to oscillate and movement is once againrestricted (Figure 2).

Figure 2. Cooperative gestures on the CPG

Hardware: The DiamondTouch TableThe hardware used was the DiamondTouch Table (DT), amulti-user, touch-and-gesture-activated horizontal screen forsupporting small group collaboration developed by MitsubishiElectrics Research Laboratories (MERL) [7]. The DT candetermine, through a capacitive coupling system, which partof the surface is being touched and can identify up to fourusers simultaneously. Its dimensions are 75 X 59 cm. and it

The ACM International Conference on Interactive Tabletops and Surfaces 2009 199

Figure 3.1: Cooperative gestures on the CPG

3.1.1 Collaborative Tabletop Games

Battocchi et al. (2009) decided to create a technology-supported collaborative playful activity in

order to work on the collaborative behaviors and social skills in children with ASD. They developed

the Collaborative Puzzle Game (CPG) whose purpose was to assemble a picture with pieces

left randomly on the board, as in the traditional Jigsaw puzzle game table. In this game, the

pieces are moved to the right place through direct touch and drag. In order to combat social

isolation, Battocchi et al. (2009) decided to introduce Enforced Collaboration (EC) in the game,

which encourages two players to play collaboratively by simultaneously touching and dragging

the pieces, while preventing children from playing individually (Figure 3.1).

During the game, whenever a piece is released within the rectangular solution area, it automat-

ically placed on the closest position. When the child finishes the puzzle, an animation starts,

showing the completed picture with the sentence Good job! Puzzle completed! and a happy

music at the background. This visual and auditory feedback has the purpose of making the

interaction more attractive and comprehensible.

In the CPG context, Encourage Collaboration was implemented through Cooperative Gestures. It

is required that both participants simultaneously touch, drag and release the piece. If only one is

touching it or the other participant releases the piece before the dragging has started, the piece

oscillates and its movement is not permitted.

In order to test CPG, a DiamondTouch Table (Dietz & Leigh, 2001) was used, which is a multi-user

and touch-and-gesture-activated horizontal screen. Two experimental studies were necessary.

Study One tested seventy typically developing children in order to create a baseline to compare.

Study Two tested children and adolescents with ASD. In both studies, the major objective was

3.1. SCREEN-BASED TECHNOLOGY AND GAMES FOR CHILDREN WITH ASD 13

to test the effect of EC. The pairs of children with similar age and level of acquaintance were di-

vided into two major groups. The first played with CPG in the EC while the second group played

without enforcement - free play (FP) in which the participants were free to play independently.

In both studies were considered the following measures: task completion time, the total num-

ber of moves, the number of functional moves, the number of coordination moves, the rate of

coordination moves, the rate of simultaneous activity, and relative interaction time.

In general, the results showed that the children in Study One were interested in the activity and

having fun. It proved that enforced collaboration has an influence on the performance and the

interaction of the participants while playing the CPG. Despite leading to longer completion times

with a higher number of moves to complete the puzzle, the EC affected positively the level of

engagement of the participants, comparing with the results of the FP condition.

Sixteen children diagnosed with ASD, ages between 8 and 18 years old, were tested with the

CPG in the Study Two. The sessions included an EC condition and the Free Play condition

for each pair. In the training phase, the experimenter and the educator demonstrated how to

manipulate pieces until the child understood. The rest of the procedure was similar to Study

One.

In Study Two, the results revealed that CPG was, in general, well accepted amongst the partic-

ipants with ASD, despite some variations on the level of enthusiasm. In order to make children

engaged with the game, the educators had to take more effort with lower functioning children,

unlike higher functioning children who were already attracted while playing. The demonstration

of EC condition, at the beginning of the session, took several minutes before the participants

learned it. However, the children who played this modality did not feel uncomfortable during the

session. As in Study One, the children in the EC condition demonstrated longer completion times

and a higher number of moves to complete the puzzle. Nevertheless, they did not show actual

difficulty during the game. The authors concluded that the negotiation and coordination of the

interaction was challenging for the children. But comparing to the FP condition, higher rates

of simultaneous activity were found in the EC condition. Briefly, EC condition did not influence

the level of engagement, although it encouraged behaviors that involve social interaction and

collaboration with the others.

In short, it was proved that the enforcement of collaboration produces a more challenging inter-

action. However, it appears to increase the level of shared activity between pairs of children with

ASD. This leads to the conclusion that EC is a strong incentive to social interaction (including col-

laborative behaviors) particularly for children with ASD who normally like to work independently.


the activity. Drew1, a seventh grader with AS, suggested several strategic moves to the group but was repeatedly ignored. Later he commented on the group’s final solution, “It’s not exactly like my planned route, but it’s close enough.” Drew’s comment illustrates perspective taking, realizing that other people have different yet valid ideas, a topic that is frequently discussed in group therapy. Drew’s mother observed the testing session from the other side of the lab. After the session she explained,

"I've actually found it rather interesting watching my son because he tends to be decisive about things and be more of a leader, but he's not forcing his will on anyone else here at all. He's listening and seemingly much more socially conscious than I think of him in terms of trying to be involved, but not trying to take over or get angry. So I'm actually quite pleased to see that."

In contrast, some non-cooperative behaviors indicate that additional structure could have helped other adolescents control their impulse to dominate the activity. Several rounds of play were chaotic with kids pushing each others’ hands off the interface and yelling loudly. One outspoken student often took control of the game, reaching across the table to move other players’ pieces without asking and telling others which piece to play next without eliciting input. This student’s father observed the testing session and commented,

"With [my son], tact and making other people feel good about what they're doing doesn't even enter the equation… he'll try to get the ideal result of whatever problem is in front of him and how that impacts other people doesn't even occur to him. That's what he needs to learn more of. Games like this give him more practice."

In the debrief immediately following the gaming session, the students gave an overwhelming response regarding the need for order while playing. One commented, “There always has to be a leader; otherwise it will be wild and nobody will get anything from it.” In response to this comment, Brad, a seventh grade student, stated, “We’re supposed to work together. We’re supposed to be equals.” Brad was the quietest participant during the testing session and quickly became agitated and covered his ears when his peers spoke loudly at each other. During an individual follow-up conversation with Brad, he explained, “Last time it was chaos.” He looked at the ground and paced back and forth, “yeah, it was really chaotic until I got to be the leader.” By “leader” Brad is referring to a point in the session where the therapist closely monitored the students and gave each a chance to make decisions for the group. In this first round of testing, we wanted to assess the appropriateness of tabletop technology for this audience. Our primary concern was whether these adolescents could learn sufficient control over the interface given the tactile input required by tabletop surfaces. Participants answered “How hard was it to move the pieces around on the table?” with a mean of 2.2 (stdev=0.45) on a five point Likert scale (1 = “not at all difficult” and 5 = “extremely difficult”). Based on the self-reported response by students and observations by the two researchers running the session, we conclude that participants found the mechanics of using the touch-sensitive tabletop technology manageable. 1 All names have been changed to preserve anonymity. Photos are

used by permission.

Overall, the students found SIDES to be a highly motivating yet challenging experience. After playing, one eighth grade student remarked, "Are we going to play again? I want to play it in the classroom.” According to the students’ therapist, this excitement carried over into the classroom for several days after the session and caused group discussions about the gaming experience, allowing him to tie the experience back into current classroom social skills topics. Session 1 demonstrated the promise of tabletop computer games as a tool for helping our target user group practice effective group work skills, as these adolescents were highly engaged with each other during the game and motivated by performance.

6. DESIGN ITERATION Play Testing Session 1 revealed that SIDES was motivating for these adolescents and is a promising tool for supporting effective group work among this user population. Session 1 also suggested that explicit game rules such as turn taking and piece ownership might help reduce controlling behaviors of some students and encourage other less engaged members to be more involved in the activity. We revised the game to include computer-enforced turn taking and restricted access to game pieces, as per our observations and feedback from the students’ therapist. After Session 1, the therapist suggested, “Whoever’s turn it is should be the only one who can manipulate the pieces. You can see that the kids can’t keep their hands off. They will reach over and if one kid is too slow or taking in more information, they might not be able to wait and will break the rules by stealing another person’s piece.” The computer provides hard, fast, and consistent rules in a

Figure 4: A “turn taking” indicator is located in front of each player next to their voting buttons. Only one turn taking button highlights at a time to indicate which player’s turn it is. In this image, the green player’s button is highlighted (located on the interface directly in front of this player) and all other turn taking indicators are white and inactive.

Figure 3.2: SIDES interface

Along the same lines, Piper et al. (2006) developed SIDES (Shared Interfaces to Develop

Effective Social Skills), a four-player cooperative computer game with the aim of encouraging

cooperative decision making and equitable participation by group members. To achieve this goal,

the game has to stimulate the learning of basic group work skills, such as negotiating, turn-

taking, active-listening, and perspective-taking. SIDES is a puzzle game where each participant

has limited square tiles with arrows, and they have to assemble an ideal path with them to permit

a frog to travel from the start to the finish lily pad (Figure 3.2). The group must agree on one

path that collects the most points with their given amount of resources. In order to encourage the

participants to discuss their own opinions, each one has a vote button and the path can only be

tested if they vote unanimously.

The first version of SIDES was tested on a DiamondTouch table (Dietz & Leigh, 2001) with five

students with ASD. The players were engaged in the activity the entire time. However, their

excitement led to a chaotic environment, where they talked over each other and did not take

turns. One of the children took control of the game, reaching across the table to move other

players pieces without asking them. For this reason, the design of SIDES was altered in order

to reduce controlling behaviors of some students. In the control panel of each player, was added

a turn taking button that indicates whether or not it is that player’s turn. With this method, Piper

et al. (2006) intended to give players a feeling of ownership over the activity.

SIDES was tested by two groups of four students in the second session of evaluation. They

showed an increase in positive language use, as well as, a decrement in the number of aggres-

sive behaviors. Which shows that the participants were interested in the activity and were also

3.1. SCREEN-BASED TECHNOLOGY AND GAMES FOR CHILDREN WITH ASD 15

conceiving the best possible solution together. The computer-enforced rules were able to encour-

age cooperative group work because they helped prevent dominant players taking control over

the game and each player felt he had contributed to the group’s final product. SIDES showed

that tabletop technology can be used as a tool for supporting collaborative group work, and can

engage students in face-to-face interactions, that they typically find extremely challenging.

Zancanaro et al. (2011) suggested that the facilitator should be part of the interface during col-

laborative games, in order to form and expand the child’s conceptual comprehension of collabo-

ration. Join-In Suite is a 3-user application for a facilitator and two children. It was developed to

address the area of social competence for children with autism, and particularly, their capacity to

collaborate with each other. The application is divided into two parts, learning and experience.

The learning part consists in a social problem-solving technique. The facilitator shows a set of

social vignettes that presents a social problem, then the children had to pick the best solution

that will promote a positive social experience. The experience part is based on behavioral rein-

forcement technique, where the children experience the picked solution which is the social task

to be acquired. In the sessions, the facilitator sits on one side of the DiamondTouch table (Dietz

& Leigh, 2001), where he controls the interface while both children interact with the surface on

the other side. Each decision is made if both children agree, and the therapist has made sure

that everyone is in accordance.

Join-In Suite has constraint patterns in each of the games, in order to make the collaborative

aspect more engaging. In the Apple Orchard story, the Constraints on Objects pattern was

implemented, where if both children drag the basket simultaneously, it moves significantly more

rapidly so that the apples can be collected before they fall to the ground. The Save the Alien

story implements the Different Roles pattern: one child has to tap on the stars to make them fall

toward the sea, while the other has to move a small boat to catch them. Lastly, the Bridge story

implements the Ownership pattern where the two children have to assemble a bridge. However,

each one has different pieces and this pattern avoids them to gather pieces that are on the

opposite side of the river. So they have to request the other child to put it on a transport machine.

Zancanaro et al. (2011) conducted a field study involving 8 children with ASD. Each session

was executed by a therapist and two children playing Join-In Suite. The results showed that this

interface allows the therapist to shape the collaboration dynamics between the two children, in

order to encourage the more passive child to suggest a solution and not follow the instruction of

the dominant child. The capacity to interleave learning and gaming proved to be exceptionally

compelling to maintain the coherence and engagement of the whole experience. Join-In Suite

demonstrated to be extremely helpful using collaborative games as a support for teaching social


! Goal-oriented activities, with clear sequence of steps and an easily identifiable end-goal; and! Cooperative turn-taking activities, with no clear end-goal and whose main objectives are social reciprocity, turn taking,

and mutual enjoyment.

Sorting a set of balls according to their colours and collecting all the flowers on the ground into a basket are examples ofgoal-oriented activities, while taking turns with the agent in order to grow flowers by shaking a cloud that produces rain andthrowing balls thorough a cloud so that they change colour constitute turn-taking activities. All activities are supposed to beperformed by Andy and the child in cooperation, with Andy assuming a more or less prominent role according to the par-ticular learning objective of an activity. For example, if the goal is respond-to-reciprocal-interaction, Andy will adopt a leadingrole in prompting the child to take a turn. If the goal is initiate-reciprocal-interaction, Andy will wait to give the child anopportunity to initiate a bid for interaction, before initiating the interaction if the child is not interacting. In the next section,we shall describe the full repertoire of Andy’s behaviours.

The ideas of a magic garden and of morphing objects as well as the content of the specific activities have emerged fromparticipatory design workshops with both typically developing children and children with ASCs and knowledge elicitationworkshops with autism experts and practitioners. More specifically, fourteen participatory design workshops involvingeighty-seven typically developing children and fifty-three children with ASCs were conducted during the lifespan of the pro-ject with the goal of informing the design of the look-and-feel of the environment including the functionality and the inter-active properties of objects, the appearance of the ECHOES’ agents and other aesthetic decisions [22]. In addition, weconducted two knowledge elicitation workshops involving thirty practitioners with extensive experience of day-to-day aut-ism intervention and in-depth knowledge of a large number of individual children with ASCs and three older (eleven-eigh-teen years old) high functioning children and teenagers with ASCs who acted as consultants. These workshops focused on theappearance of the agent, its specific behaviours toward the children and the narratives underlying the learning activities.Finally, the ECHOES environment and a subset of the learning activities were tested at different stages of the project in fourformative evaluation studies involving forty-six children with ASCs [2]. A number of crucial design decisions, including theappearance of the agent, have been influenced by these formative evaluation studies.

6. The design and implementation of the ECHOES agent

Our goal was to create an artificial social partner that could act credibly both as a peer and as a tutor to children with ASCsand, as a result of this, that could deliver the educational and interpersonal support advocated in SCERTS. As a tutor, ouragent aims to deliver visual and organisational support for:

(i) Expanding and enhancing the development of a child’s expressive communication system’’ [63, p. 309];(ii) Supporting a child’s understanding of language as well as others’ non-verbal behaviour’’ [63, p. 309]; and

(iii) Supporting a child’s sense of organisation, activity structure, and sense of time’’ [63, p. 309].

In addition, the ECHOES agent acts as a peer to the child and aims to provide them with interpersonal support by:

(i) Accommodating the child’s preference for structure and predictability, while fostering initiation, spontaneity, and self-determination; and(ii) Exposing the child to a positive interaction with a peer so that they can ‘‘benefit optimally from good language, social,and play models’’ [63, p. 309].

To achieve this goal, we designed the agent’s physical appearance, its behaviour and its interaction with childrenbased on the SCERTS model and input received from thirty ASCs practitioners, who participated in two workshops

Fig. 1. Children interacting with Andy through the ECHOES multi-touch display.

S. Bernardini et al. / Information Sciences 264 (2014) 41–60 47

Figure 3.3: Child interacting with Andy

competence skills, and that the presence of a facilitator is fundamental in educational interven-

tions for children with ASD.

3.1.2 ECHOES

Bernardini et al. (2014) implemented ECHOES, a serious game with the purpose of helping chil-

dren with ASD gaining social communication skills. Their goal was to create an autonomous

agent competent to interact with a child without the presence of a therapist. In order to struc-

ture the interaction between the agent and the child, Bernardini et al. (2014) leveraged on the

SCERTS educational model. This model divides the required intervention into three develop-

mental stages: (1) Social partner - the child communicates through pre-symbolic means (e.g.,

gestures and vocalizations); (2) Language partner - the child uses early symbolic means to com-

municate (e.g., oral language, sign language, picture symbols); (3) Conversational partner - the

child uses more advanced language abilities.

ECHOES learning activities are displayed in a multitouch LCD display. The 12 activities take

place in a sensory garden occupied by Andy (Figure 3.3), the autonomous virtual agent, and by

interactive objects that react when the agent or the child interacts with them through specific touch

gestures. These action-reaction events intend to promote children’s understanding of cause and

effect. The developing activities are linked in non-verbal narratives. This reinforces the SCERTS

principle, which states that they need to share an obvious unifying theme in order to support

shared attention. ECHOES learning activities’ objectives are to improve joint attention and non-

verbal skills.

Bernardini et al. (2014) planned to create an artificial social partner that could have the role of

a peer and a tutor to children with ASD. As a tutor, the agent has to develop child’s expressive

communication system, support child’s understanding of language and sense of organization.

As a peer, the agent has to adapt itself to the preference of a child regarding structure and

3.2. SOCIAL ROBOTS AND CHILDREN WITH ASD 17

predictability, as well as to expose the child to a positive interaction.

For the intelligence of the agents, FAtiMA (Dias & Paiva, 2005), an Autonomous Agent Architec-

ture with reactive and BDI based deliberative behavior was used. In order to form the ECHOES’

intelligent engine, this model is complemented with two additional elements: (1) the pedagogic

component, which monitors the interaction between the child and the agent to guarantee the

accomplishment of the learning objectives; (2) the child model, a user model that measures the

cognitive and emotional state of the child in real time aiming to feed this information to both agent

and pedagogic component.

The authors emphasized the importance of giving positive feedback to the children, in order to

reduce the anxiety related to social interactions. If Andy has suggested an action and the child

does not perform it, the agent first waits and then interferes by repeating the action and inciting

the child to do it again.

Nineteen children with ASD participated in the study, aged between 4 to 14 years. In the first

sessions, Andy does not interact with the child. However, the child can interact with him (e.g.,

tickle him) in order to get used to his presence. On later activities that require turn-taking, Andy

interacts directly with the child (e.g., making requests, or showing enthusiasm for the actions of

the child).

The children demonstrated a comfort with the ECHOES environment and their responsiveness

was stimulated. Over the three sessions, the number of initiations to Andy and to the practitioner

increased. Curiously, one of the non-communicative children who initially did not demonstrate an

interest in Andy later spontaneously waved and said Hi Andy! when he appeared on the screen.

This increasing of responsiveness level might be elicited by the comfort that children experienced

while playing ECHOES. It was expected a growing disinterest in Andy given the increasing com-

plexity of the activities, but that did not happen. The authors did not expect statistically significant

results, because of their small and heterogeneous group. However, the results were positive and

showed that Andy managed to elicit spontaneous social behaviors in the children.

3.2 Social Robots and Children with ASD

In this section, some previous work is summarized concerning the usage of social robots in

therapy sessions of children with ASD. The works Robota (Billard et al., 1998), KASPAR (Daut-

enhahn et al., 2009) and Tito (Duquette et al., 2008) are firstly presented. Their purpose is to

encourage the children developing their communication and social behaviors, such as imitation


Figure 3.4: Labo-1

and turn taking skills. Then Keepon (Kozima et al., 2009), which helps children understanding

social information, and sharing with others their interests and emotions, was introduced. Some

works used robots like Pleo (Kim et al., 2013) and the Bubble Blower (Feil-Seifer & Mataric, 2009)

with the purpose of mediating social interactions between children and other people. Finally, the

work of Greczek et al. (2014) with Nao, demonstrate that through the graded cueing feedback,

the children could improve their performance over time.

Robots can vary both regarding appearance, or intended purpose, like the ones used in the

works that are presented in this section. They show the children’s enthusiasm in interacting with

robots and demonstrate that robots can provide a focus of attention. Other works also concluded

that robot’s simple appearance can encourage autistic children to engage in various interactions

(Duquette et al., 2008; Feil-Seifer & Mataric, 2009; Kozima et al., 2009). Therefore, these works

motivate our research in terms of using a robot to engage the children in therapeutic context.

Regarding the work of Greczek et al. (2014), it inspired the Tutor Condition of our architecture

(presented in the Section 4.3). The robot performs an adapted system of prompts based on what

was presented by Greczek et al. (2014).

Another great motivation for our work was The Aurora Project concerning their research in social

skills, especially the turn-taking. The researchers showed that robots can elicit taking turns with

autistic children, as it is reported forward.

3.2.1 The Aurora Project

The AuRoRA (Autonomous Robotic platform as a Remedial tool for children with Autism) research

project was initiated in 1998 by Prof. Kerstin Dautenhahn. This project’s aim is to investigate the

potential use of robots as therapeutic or educational toys for children with autism. Since 1998,

Aurora’s researchers have been studying ways in which robots can encourage basic communi-

cation and promote social interaction skills in such children (Robins et al., 2009).


Figure 3.5: Robota Figure 3.6: KASPAR

Labo-1 was the first robot to be used in Aurora project (Werry & Dautenhahn, 1999). The first ver-

sion of the robot was an independent agent, relying solely on its own inputs and decision routines

(Figure 3.4). In the trials, the children interacted freely with the robot (e.g., some children moved

away and looked to see if the robot was following them), and showed no fear of it. Although,

Werry and Dautenhahn concluded that Labo-1 could only provide a small range of interactions

with the children, limited to spatial approach/avoidance turn-taking games.

In 1998, Billard et al. (1998, 2007) decided to build Robota (Figure 3.5), a doll-shaped mini-

humanoid robot, that can engage children in synchronous and imitative interaction games, in

order to develop interaction skills in children with autism (Dautenhahn & Billard, 2002). During

the study, children enjoyed experiencing the robot’s reactions to their touch (Billard et al., 1998).

The complexity of the robot (its movements and speaking) seemed to intrigue the participants,

but not intimidate them.

Dautenhahn & Billard (2002) accepted that imitation has a critical influence in the advancement of

social cognition and communication in all children. The social function of imitation in interactions

is recognized as a means to engage others in interaction, to express interest, and to develop

coordinated interaction which is vital to communication. Along these lines, they proposed to

investigate how imitation and turn-taking games with a humanoid robot (i.e., a robot that can

match some basic behaviors of humans) could empower social cooperation abilities in children

with autism. In order to analyze the potential use of Robota in therapy, the researchers conducted

a set of trials with 14 children with ASD. The participants were encouraged by the teacher to move

their arms and head, in order to observe how the robot mimicked them. Dautenhahn & Billard

(2002) concluded that the children’s participation in these trials was considerably initiated by the

teacher since the setup did not seem to encourage spontaneous interactions by the participants.

However, they observed clear signs of enjoyment by the children such as smiling and kissing


goodbye Robota.

Billard et al. (2007) continued developing studies with the aim of examining the interaction be-

tween Robota and children with autism. In one of the two studies, they investigated what effects

the robot’s appearance has, and the authors concluded that children prefer interacting with a plain

robot over a human-like robot. In the other study, Robota was used to elicit imitative behaviors

in children with autism. The results showed that several participants demonstrated spontaneous

imitation, and understood that their own movements were at the origin of Robota’s motion.

The later research of Aurora include KASPAR, a minimally expressive humanoid robot built by

Dautenhahn et al. (2009) (Figure 3.6). KASPAR’s face has few expressions and few sensors

keeping in mind the end goal of accentuating the most salient human-like traits of the robot.

These characteristics make KASPAR suitable for human–robot interaction research, whereby

they can study the influence of a few salient behaviors, gestures and facial expressions on the

interaction with people. The robot has a static torso and has the size of a small child, in order not

to appear threatening. Under these circumstances, KASPAR is capable of developing social and

communicative skills in children with ASD, through collaborative games. Also, KASPAR can be

manipulated by the therapist or the child, with a remote control, and thus be able to play a variety

of games (e.g., imitation and turn-taking games).

KASPAR has countless applications in research, through robot-assisted plays, in order to help

children with autism surpass some difficulties. The robot was built to encourage tactile explo-

ration on its body, which is important to increase body awareness and sense of self in children

with autism. Another KASPAR’s purpose is to mediate child/adult interaction and thus break the

isolation of the child. Besides that, the robot still seeks to teach social skills, through its facial/-

head and gestural expressions and promote collaborative play with other children and adults.

Robins et al. (2009) exploited the way that KASPAR’s simple movements and facial expressions

encourage basic imitation and turn taking skills in children with autism, as well as sharing and

communication with others. The researchers conducted a study with 3 children with autism,

it consisted of imitation games of facial expressions. Their goal was to investigate how this

socially interactive robotic platform could mediate the interaction between children with autism

and other people, and what effect it had. During the trials, the robot was operated remotely

through a remote control, either by the investigator or by the child. During the interactions with

KASPAR, the children touched and gazed at its facial expressions in detail and also interacted

with the co-present adult. Furthermore, they did not demonstrate aggressive behaviors towards

the robot, they were relaxed and manifested interest in it. In short, it was possible to observe in

all participants that KASPAR helped the children to generalize their behavior with the robot to the

co-present others, by stimulating gazing and touching them.


or if it has flipped over. The robot is also equippedwith ultrasonic range sensors for simple obstacleavoidance. Tito generates vocal requests through pre-recorded messages. A wireless remote control (using avideo game controller) was designed for teleoperation,and an on-board microcontroller enables pre-programmed sequences of behaviors (motion and vocalmessages). Examples of pre-programmed behaviorsare: moving the left arm while saying goodbye,expressing happiness by moving its arms, singing androtating on the spot, or shaking its head to indicate no.Tito records and stores internally the timing betweenthe interactions of the child (from sensory data andaccording to the experimental scenarios). Tito alsoemits a sound when it starts the execution of anexperimental scenario, allowing synchronization ofvideo data recorded with an external camera. Theactivation button of Tito is hidden at the bottom of therobot so that the child is not tempted to play with it.

Fig. 1 – Tito, the robot mediator

3. Results

Three variables were observed in our trials: sharedattention (visual contact / eye gaze directed toward themediator for more than 3 sec; physical proximity;imitation of facial expression or gesture, but notdirected toward the mediator); shared conventions(facial expression, gesture, actions and words, alldirected toward the mediator); absence of sharing (novisual contact, leave the communication area, avoid themediator, sensorimotor play, mannerisms, ritual,aggression). These variables were coded over 12 sec.Windowing, by two coders (98% fidelity) and videofootage of the trials (Camaioni et al., 2001).

We observed that children paired with the robotmediator show better shared attention (visual contact,physical proximity) than the children paired with thehuman mediator in all types of imitation playsincluding facial expressions, body movements, familiaractions with objects or without objects. This validatesthe hypothesis that the robot has appealingcharacteristics for interacting with autistic children.

However, we observed that forms of sharedconventions such as imitation of body movements andof familiar actions are higher with the two childrenpaired with the human. This may be explained byworking with low-functioning autistic children havingmore difficulty understanding communication intentfrom the limited motion capabilities of the robot. Onthe other hand, the two children paired with the robotmediator imitate facial expressions more than thechildren paired with the human mediator. Imitation ofwords only appeared for one participant, paired withthe human mediator. Children paired with the robotmediator were also observed imitating motor noisemade when the robot’s articulations are moving.

4. Conclusion

Our study helps understand the processes fordecreasing autistic children anguish and increasingtheir attention to learn certain forms of communication.Our results are very encouraging and support thecontinuation of work on this research question,repeating the trials with a greater number of subjectsand consolidate these conclusions.

References

Blanc, R., Gomot, M., Gattegno, M.P., Barthélémy, C.,Adrien, J.L. (2002) Les troubles de l'activitésymbolique chez des enfants autistes, dysphasiques etretardés mentaux et l'effet de l'étayage de l'adulte.Revue Québécoise de Psychologie.

Camaioni, L., Aureli, T. (2002) Trajectoiresdéveloppementales et individuelles de la transition versla communication symbolique. Enfance, 3.

Kazdin, A.E. (1976). Statistical analayses for single-case experimental designs. Single case experimental

designs. Eds Hersen and Barlow.

Lemay, M. (2004). L'autisme aujourd'hui. Paris:Éd.Odile Jacob.

Nadel, J. (2002). Imitation and imitation recognition:Functional use in preverbal infants and nonverbalchildren with autism. The Imitative Mind Development

Evolution and Brain Bases. Eds Meltzoff and Prinz.

Robins, B., Dauthenhahn, D., Boedhorst, R., Billard, A.(2004). Effects of repeated exposure to a humanoidrobot on children with autism. Proc. Universal Access

and Assistive Technology.

Zilbovicius, M. (2004). Imagerie cérébrale et autismeinfantile. Revue Cerveau et Psychologie.

Figure 3.7: Tito, the robot mediator

KASPAR’s minimal facial expressive features make its behaviors more predictable, which pro-

vides a safe and enjoyable interactive learning environment for children with autism. It has been

found to be exceptionally alluring to these children and a suitable contender for an instrument to

be used as a part of training and treatment.

3.2.2 Tito

Duquette et al. (2008) built Tito, a custom-made mobile robot with a humanoid shape. It has two

feet, two legs, and it can move the two arms and the head (Figure 3.7). In its simple face, it

has two eyes, a nose, and a mouth. In one of the eyes, it has a camera used to measuring eye

gaze toward Tito. In order to communicate, the robot has several pre-recorded messages in a

masculine voice. It also has a wireless remote control used for teleoperation and a microcontroller

that allows pre-programmed sequences of actions and vocal messages (e.g., raising its left arm

and saying Hello!).

The goal of Duquette et al. (2008) was to analyze how the characteristics of a robot (e.g., his mo-

tion, communication skills, and appearance) could encourage a child with ASD to interact. They

also studied the ability of the robot to maintain the child’s interest and facilitate the development

of social and communication skills.

With the aim of comparing the results of the experimental tests, they split the four participants into

two groups. One of them interacted with the robot and the other with a human mediator. Both

of the mediators performed similar imitation exercises, which consisted of three levels: facial

expressions, body movements, and familiar actions with or without objects. If the child imitated

correctly the mediator smiled, raised both its arms and said Happy!.


The results showed that the participants paired with Tito demonstrated more shared focused

attention, visual contact, eye gaze directed toward the robot and proximity with their mediator

than the ones paired with the human mediator. They also imitated more facial expressions.

However, the children paired with the human mediator demonstrated an increase on the imitation

of words, gestures and actions comparing with the others. They also exhibited more gestures

towards the other person in the room compared to the children paired with Tito.

In general, the study showed that sensory properties of the robot (e.g., movements, colors, lights)

managed to keep children interested in it and, thus, reduced repetitive plays with their favorite toy,

and no repetitive behavior towards Tito. Also, this study could only confirm parts of the research

hypothesis: Tito, when paired with the children, managed to increase their shared focused atten-

tion. However, the children paired with the human revealed higher results on imitation of body

movements and of familiar actions. The authors assigned these results to the children with low-

functioning autism having more difficulty in comprehension the communication intent from the

restricted movement capabilities of the robot.

3.2.3 Keepon

Hideki Kozima and his research group developed interesting studies in the field of social robots

for autism therapy. Previously, Kozima et al. (2004) built Infanoid, a upper-torso humanoid robot

capable of expressing attention, facial expressions and other gestures, with the goal of investi-

gating human social development. The researchers observed that children with autism enjoyed

social interactions with the robot. However, Infanoid expressed excessive information, which led

to anxiety and attention dispersion by the children. Therefore, the researchers developed Keepon

(Kozima et al., 2009), that unlike Infanoid, has a minimal design for facilitating the exchange of

attention and emotion. It allows them to study psychological models of the development of social

intelligence. The robot has a neck/waistline that gives an anthropomorphic impression to children

and provides movement freedom (Figure 3.8). Keepon can be autonomous or teleoperated, and

it can only perform two kinds of actions: attentive action - orienting its face to a certain target; and

emotive action - expressing its emotional states, such as pleasure, excitement, and fear, through

preprogrammed body movements.

The goal placed by Kozima et al. (2009) was analyzing how the minimal design of Keepon can

help children understanding social information and motivating them to share interests and feelings

to others. The robot reacts emotionally whenever the child makes any social interaction, either

verbally or physically. The results demonstrated that in the first contact with Keepon, the children

were aggressive to it or avoided its gaze, but then it managed to arouse curiosity and the children


were gradually approaching the robot. Some of them used the Keepon as a pivot in triadic

interactions (i.e., between the child and co-present others), for sharing the joy and surprise they

found in the dyadic interactions (i.e., between the child and the robot). In conclusion, regardless

Keepon’s limited features, its simple appearance can encourage an unconstrained trade of mental

states in autistic children, such as pleasure, surprise, or frustration.

Leite et al. (2015) also used Keepon to investigate if multiple social robots are as effective in-

teracting with small groups of children as they are in individual interactions. The researchers

conducted a study with 40 normally developed children. They came to the conclusion the partic-

ipants that interacted alone with robots were able to remember the narrative structure and other

details better than the participants in the group condition. This may be due to children being more

attentive in individual interactions. However, both the individual condition and the group condition

demonstrate increasing learning gains.

3.2.4 Pleo

Kim et al. (2013) investigated if social robots can play the role of embedded reinforcers (i.e.,

evoke and reward social behavior in interventions). For this purpose, they compared the effects

of interactions with a social robot on the children with autism, against the effects of interactions

with a human or a touchscreen computer game. The robot chosen was Pleo, a socially expres-

sive dinosaur robot designed to express emotions and attention through body movements and

sounds, like a pet dog. It has 13 pre-recorded synchronized motor and sound scripts, and 10

socially expressive behaviors (e.g., greeting) plus 3 non-social behaviors (e.g., bite).

The authors aimed at studying if interactions between children and social robots motivate the

curiosity and the demonstration of expressions of excitement. Twenty-four children with ASD,

aged between 4 and 12 years, participated in the study. It used a human Wizard-of-Oz style

controller that secretly operates the robot during the sessions, thereby allowing accuracy and

flexibility. The human and the robot interactions consisted of a game with colored blocks. Another

human, the confederate, showed the blocks to the robot or the human, who would have reactions

to each. Then, the participant would have to say if its partners had liked the color of the block and

what was their emotional state, based on their reactions. In order to manifest social expressions,

the robot and the adult used body language, pseudo-verbal or minimally verbal (respectively),

and vocal prosodic indications, in their interactions. In the computer game, the children had to

play a Tangram game.

The tests showed that participants produced more utterances (i.e., a piece of speech beginning

and ending with a pause with more than 2s) with the robot than with the adult, and more in both


Figure 3.8: Keepon, the interactive robot Figure 3.9: The mobile robot used in theexperiment

the robot and adult conditions than in the touchscreen computer game condition. They found that,

generally, children with ASD spoke more while interacting with a social robot than on the other

conditions. In the robot condition, an increase of speech directed toward the confederate, such as

verbalized conjectures and questions about how the robot works was verified. The authors came

to the conclusion that the design of the robot reinforced verbal interaction with the confederate.

These results suggest that a social robot elicits more speech than another human.

3.2.5 Bubble Blower

D. Feil-Seifer and M. Mataric defined Socially Assistive Robotics (SAR) as the intersection of

Assistive Robotics (i.e., robots that support individuals with physical disabilities through physical

interaction) and Socially Interactive Robotics (i.e., robots whose main task was some form of

interaction) (Feil-Seifer & Mataric, 2005). In other words, SAR focuses on giving support to the

user through social rather than physical interaction. Feil-Seifer & Mataric (2008, 2009) devel-

oped studies investigating methods for using socially assistive robots as agents for therapeutic

interaction in ASD.

They composed the treatment methodology in line with the DIR/Floortime intervention, which

is a semi-structured play intervention where a therapist uses a child’s existing social skills as a

premise to grow new social behaviors. In the sessions, the researchers presented the robot in a

place of a toy, as part of the Floortime therapy process, in order to motivate proactive interaction

and mediate joint attention between the child and a peer. Feil-Seifer and Mataric designed the

Behavior-Based Behavior Intervention Architecture (B3IA) that intends to create an architecture,

supporting algorithms for Human-Robot Interaction (HRI) based intervention methods, and to

enable principled evaluation of SAR systems (through user studies) in the context of ASD. This

architecture uses specific modules for activity modeling and history capture to monitor the status

of social interaction between the robot and the user. In B3IA, the robot observes the performance

of the child through a set of sensors. The purpose of this method is to facilitate the robot’s ability


to sense the actions of the child, act autonomously, react to the interplay of interaction over time

and evaluate the quality of human-robot interaction.

The specific research aim was to determine if the presence of the robot has an effect on the so-

cial behavior of the child. In order to achieve that goal, Feil-Seifer & Mataric (2009) conducted a

pilot study involving two children with ASD and one typically developing child, with ages between

1 and 12 years old. The scenario consisted of the game Bubble Play - a computer bubble blower

that can be mounted on the robot (Figure 3.9). During therapy, the Bubble Play incites social

interaction between the child and the individual operating the bubble blower, so, the researchers

expected that, in this project, the role of the robot was as a catalyst for social interaction. To test

if the behavior of the robot influenced the behavior of the child, they compared two conditions,

contingent and random. In the contingent condition, if the child pushes a button, then the bubbles

blow. In the random condition, the robot blows bubbles randomly, regardless of any button being

pushed. The results demonstrated that in the contingent condition, the participants showed in-

creased social behaviors toward the robot and the adult, comparing to the random condition. In

this manner, the obtained outcomes support their hypothesis.

3.2.6 NAO

Greczek et al. (2014) developed a study aiming at analyzing the effects of a humanoid robot,

giving graded cueing feedback, during an imitation game played with a child with ASD. Graded

cueing is a method to improve people’s skills (e.g., social skills) during therapy by giving them

increasingly specific cues or prompts. The therapist requests the patient to perform a task, then

prompts the patient with increasing specificity taking into account how much the patient struggles

with the task. Greczek et al. (2014) were motivated by the fact that children with ASD are normally

impaired in the imitative capacity, which is essential in social interaction. If they practice frequently

with a therapist they can improve that skill. The authors questioned if this graded cueing method

could improve child’s performance and autonomy, with less intervention from the therapist.

In the trials, the child had to play an imitation game with NAO1, a humanoid robot which could play

the role of a social mediator. Also, the robot allows obtaining predictable and concrete feedback

(i.e., using lights, colors and sounds) for the performance of the child. Furthermore, Larson’s

study (Larson et al., 2011) proved that the children with ASD prefer concrete feedback, which

was another proof that NAO would be well suited for this project.

As aforementioned, the child had to play an imitation game - the Copy-Cat - with the robot. NAO

strikes one pose from the 25 possible poses (this game is limited to movements of hands and

1https://www.aldebaran.com/


arms), then the robot encourages the child to imitate it. If the child’s pose is well performed, the

robot gives positive verbal feedback, nods and flashes its eyes green. If the child does not imitate

correctly, NAO begins the prompt system. After the first prompt, if the child needs more prompts,

the robot continued to climb the levels of prompting:

- P0: No prompts given

- P1: Words

- P2: Words + Gesture

- P3: Specific Words + Gesture

- P4: Specific Words + Specific Gesture

In the case where the child has passed through these 4 levels and continued to not imitate

successfully, the robot would change its pose, so that the children, would not lose interest in the

game.

The goal of the graded cueing computational model is to minimize the number of prompts and its

specificity while maximizing the user’s success in performing the task. The model is composed

of N states, representing the level of skill that the child belongs, the N + 1 (P0) is the success

state. The aim of the model is to select the first prompt according to the child’s task ability level.

The probabilistic level can easily adapt to the change of user’s state using a Bayesian approach.

Greczek et al. (2014) conducted a pilot study to test the graded cueing model with a group

of 12 children with ASD, aged between 7 and 10 years. In order to compare the results, the

children were split into two groups. One of them received graded cueing feedback and the other

received non-adaptive feedback (i.e., received P4 feedback from the robot without encouraging

participant’s autonomy).

The group who received graded cueing feedback required 80 prompts, whereas the other group

required 111 prompts. The authors desired that the number of prompts decreased over time,

however, that did not happen for all participants. The result was not expected to have statistical

significance due to the scarce number of participants and sessions. Overall, the children who

received graded cueing increased their pose accuracy over time. Greczek et al. (2014) came

to the conclusion that varied feedback is more effective and less frustrating than non-adaptive

feedback in a child-robot interaction.

Chapter 4

Architecture

In this thesis, we develop a Tangram puzzle game in a tablet that is engaging and playable

for children with ASD. We proposed to use a humanoid robot in order to stimulate children’s

interest and to help them during the game.

This game has two conditions. In the first one, the children are able to play this game with the

help of a robot during their therapy sessions. This condition is directed to children with various

difficulties (e.g., motor, reasoning, and so on) in this type of games. And it allows us to analyze

the influence of the robot in the evolution and the concentration of the child in a certain task.

And in the second condition, they play with the robot as a peer a turn-taking cooperative game.

This last condition is targeted to children who have already mastered this type of games but have

difficulty knowing when it is their turn (this is one of the characteristics present in some autistic

children, as mentioned in Chapter 2). This enables us to study also the level of concentration on

the task and the capacity for the robot to establish turns.

The purpose of our work is not to replace the therapist, or to cease the therapy sessions, but to

create an assistive tool to be used in therapy sessions.

The next sections describe in detail the original Tangram game, the tablet game functioning with

all the different settings and the two initial studies conducted in this thesis. Then, the robot used

in this experiment is presented, as well as, the reasons why it was chosen. Finally the two

developed conditions (Tutor and Peer) are introduced.

27

28 CHAPTER 4. ARCHITECTURE

Figure 4.10: Original Tangram puzzle game

4.1 Tangram Puzzle Game

Tangram is a puzzle game supposedly invented 4000 years ago by a Chinese god named Tan

(Costello, 1988). However, this game was only brought to America in 1815 and nowadays is

played all over the world. It is similar to Jigsaw puzzles1 as both purposes were to place pieces

together, but they are completely different. While the jigsaw puzzle has a larger number of pieces

with complex shapes, which contains a single solution, the Tangram (Figure 4.10) only include

7 geometric and simple pieces that can be arranged in numerous ways in order to obtain more

than 6500 silhouettes of animals, letters, numbers, objects, people, and so on (Read, 1965).

The seven pieces correspond to 2 large isosceles triangles, 1 medium isosceles triangle, 2 small

isosceles triangles, 1 square and 1 parallelogram. The large triangles are two times the area

of the medium one; and the medium triangle, the square, and the trapezoid are twice the area

of each small triangle. It is commonly commercialized in wooden pieces but currently there are

already many computer, smartphone, and tablet games.

This puzzle game has the capacity to improve some skills, such as imagination, visuospatial,

logical and concentration (Kohanova & Ochodnicanova, 2014; Siew & Chong, 2014). And can

also extend child’s knowledge about geometric figures and geometric spatial thinking (Khairiree,

2015). The players have to assemble all seven pieces in order to achieve a figure. They have to

drag, rotate and flip pieces, to understand the spatial relations between them and which shape

stays the same no matter how it is turned. This sort of puzzles requires mathematics knowledge

to solve them (Russell & Bologna, 1982).

For the reasons listed in the previous paragraph, Tangram is frequently played during therapy

sessions by autistic children starting from 3 years old. Therefore, this was the chosen game to

1Traditional puzzle where each piece has a part of a picture on it. Whenever complete, it creates a complete picture

4.1. TANGRAM PUZZLE GAME 29

Figure 4.11: Easy Level Figure 4.12: Medium Level

Figure 4.13: Hard Level Figure 4.14: Interface with two pieces

be developed in this thesis. As it was concluded in the first meeting held in HGO with doctors

and therapists (which is described in subsection 4.1.3), this puzzle attracts ASD children due to

its geometrical shapes. Although, during the game, they usually lose interest on it.

Normal Tangram puzzle is not an engaging game and child’s enthusiasm on it has to be stim-

ulated. Clements (2000) concluded that using recreational devices such as computers may in-

crease child’s level of engagement in learning. We expected that since the game was played on

a tablet, it could overcome this problem, keeping children engaged in it. Also, since the game is

played on a tablet, it can improve the evolution of coordination and fine motor skills (Piper et al.,

2006). It has been discussed by some researchers that the presence of an excess of details can

be over appealing, becoming a factor of distraction, since it leads the children to concentrate on

them instead of in the interaction (Kozima et al., 2004; Wong et al., 2004). So it was important to

have a simple and distractions-free interface.

4.1.1 Game Functioning

As is possible to visualize in the Figure 4.11, the game interface consists of only three compo-

nents: (1) the solution area, (2) the pieces, and (3) a home button. During the game, the player

has to drag the pieces with its finger to the right places. Whenever the player releases a piece

within the solution area, if it is relatively close to the right spot, the piece anchors in it. If not, the


piece returns to its default position. Thus, it is perfectly clear that the piece does not belong to

that place. When all pieces are in their places, the puzzle is completed and after a few seconds

appears a new puzzle.

The following game settings were developed in order for the game to be playable for most children

of the spectrum. And also, so that the game is always evolving as children progress, thus they do

not lose interest in it. These game settings are handled by Game Manager (Subsection 4.1.2).

4.1.1.1 Difficulty Levels

There are three Difficulty Levels: easy, medium and hard. At the easy level, the solution area has

all the board lines of the puzzle, and its interior is transparently colored in the color of the correct

piece (Figure 4.11). The medium level consists only of the puzzle with the uncolored border lines

(Figure 4.12). Finally, in the hard level, the solving area only shows the outline of the shadow

(Figure 4.13). Undoubtedly that the hard level is the most challenging of all. At this level, the

children will have to plan and visualize the solution in their mind, which will allow to long-term

develop their visual-spatial skills.

4.1.1.2 Rotation Modes

In some instances, it is necessary to rotate the pieces to be able to fit them in the right places. In

order to create more challenging levels throughout the game, it was created 3 modes of rotation.

• Simple Mode: It is the first and the easiest mode. In this condition, the pieces are already

with the right angle according to their place in the puzzle (Figure 4.11). The child only

needs to drag the piece to the right spot.

• Button Mode: In this situation, the child has to click the button of a certain piece to rotate

it (Figure 4.12). Each click rotates the piece 45 degrees to the right. The button appears

when the piece is clicked and disappears when is dragged or another piece is clicked.

• Finger Mode: This is the hard mode. The piece is rotated when the child touches one of its

corners and drags the piece around itself (Figure 4.13). When stops the rotating, the piece

anchors to a certain angle multiple of 45.

These modes intended to allow the child to evolve its coordination and fine motor skills (Piper

et al., 2006) by rotating the pieces.


4.1.1.3 Distance Threshold

This threshold defines the distance from the right place that the piece can be dropped. It ranges

from 4 to 1, in the 4 the piece can be dropped far from the place, in other words, if some percent-

age of the piece is within the right place, anchors to it. And in the 1 the piece should be dropped

very close so it anchors. This also promotes child’s fine motor skills.

4.1.1.4 Number of Pieces

The common Tangram Game is played with 7 pieces, but we decided to implement puzzles with

fewer pieces (from 2 to 6) (Figure 4.14). Thus, children with more difficulties in the game can play

a simple puzzle with fewer distraction factors. Our goal was to develop a game that could cover

more children with different characteristics within the spectrum.

4.1.2 Game Difficulty Management

In order to the children do not lose curiosity about the game over time, the difficulty must be grad-

ually increased (Ferrari et al., 2009; Zancanaro et al., 2011). So in each game, the complexity of

the game is gradually increased, to avoid stress or lack of interest by the child. This increment

of difficulty is materialized by (1) increasing the complexity of the puzzle, (2) increasing the diffi-

culty of rotation modes, and finally (3) decreasing the threshold of distance to the right spot. To

manage all the aforementioned settings was developed a Difficulty Manager.

There are 10 puzzles available, five of them are relatively easy, the other five are slightly difficult

(this is only relevant for the hard level games). When the game starts, a puzzle is chosen, by the

program, from within the set of available puzzles. The puzzle is always different from the previous

and only repeats after the child played with all puzzles in one session. So the puzzle is always a

novelty and the results are not influenced by the learning factor. This is also determined in this

Manager. Some puzzles used in this game were based on the book Read (1965).

The difficulty manager decides the characteristics of each game according to the performance

of the child in the previous games. The first game is the easiest one, begins with the easy level

puzzle, in the simple mode rotation, and the distance threshold is 4. During the game, data is

saved to help the manager decide the settings of the next game. To decide the difficulty level, the

manager analyzes how long the game lasted and how many games have been already played

in that level. To decide the rotation mode, the manager analyzes the average time until the child

places each piece and how many games have been already played in that mode. The order is


first the Simple Mode, then the Button Mode and finally the Finger Mode. And lastly, to decide the

threshold, it analyzes the number of failed tries that were relatively close to the correct place. If

the child gives up playing the game, the manager chooses an easier level than the current game

in each of the three features. For the game with fewer pieces, it begins with a puzzle of 2 pieces,

and according to the time spent to finish the game, the number of pieces increases or decreases.

4.1.3 Initial Studies

The current subsection reports the two initial studies developed in the scope of this work. The first

study was conducted before the development of the game, in order to search more information

about the usage of the Tangram game in therapy sessions of children with ASD. The second

study had the aim of testing the performance of the game and to analyze the behavior of the

therapist and the child during the game. These studies were crucial to the development of the

design and functioning of the game.

Both preliminary studies took place in Centro de Desenvolvimento da Crianca Torrado da Silva

(CDC) of Hospital Garcia de Orta1. CDC provides treatment for all kinds of neurological diseases

and developmental disorders.

4.1.3.1 Before the Development of the Game

In May 2015, one child was monitored during a therapy session. Before the session, there was

a meeting with the therapist, in order to outline the procedure. She informed that based on her

experience, the Tangram can captivate children with autism because they like geometric shapes,

though the original game eventually becomes uninteresting due to be composed by unattractive

pieces of wood. The therapist also informed that the game in the tablet can stimulate the children,

but this has to be captivating (with colors and animations) to not lose their interest.

The session took place with a five-year-old boy with ASD. He was able to speak, but he could

not focus on the same task for a long time and also had some difficulty in following instructions.

The interaction with Tangram lasted only five minutes, which corresponded to a single game. The

therapist began by asking which of the shapes he preferred to do. The boy chose a simple puzzle

that matched the letter M (the initial of his name). During the game, the therapist had to call the

child’s attention, for several times, because he easily lost his focus. The therapist had to give

many clues to the boy so that he could complete the puzzle (e.g., How many triangles we need?;

Where does this square belong?; and Do you think that in here belongs a big, medium or small

1http://www.cdc-hgo.com/


triangle?). Each time he placed the piece in the right spot, the therapist gave a positive feedback

(e.g., Nice!, Very Good). When the puzzle was complete, the therapist asked to which object

corresponded, and the child replied that it was the letter M, the initial of his first name. Finally,

when the therapist inquired if the child wanted to play another Tangram game, the boy responded

negatively. He wanted to play on the trampoline, that was his final reward for finishing the puzzle.

With this study, we concluded that the puzzles are games that do not captivate, by themselves,

the attention and interest of autistic children. In this manner, children’s attention must be encour-

aged. On the other hand, we could confirm that is important the child receives tips and clues

throughout the game so that the child does not lose interest and to stimulate child’s reasoning.

Also, positive feedback is fundamental during the game, as has been seen previously. In addition,

the child needs to have a final reward for finishing the puzzle.

4.1.3.2 With the Tablet Game

Before the sessions with the children, we held a meeting with doctors, therapist, and psychologist

of the CDC. The aim of this reunion was to present the game developed so far, to have an

exchange of improvement suggestions, and outline the tests that would be conducted.

About the game, they all agreed that was captivating and interesting for people with ASD. How-

ever, they focused on some aspects that could be improved or added to the game, in order to

provide a better interaction. They suggested that the hard level should have some sort of help

that showed up the puzzle solution for a few seconds. Also, they recommended that the game

should have sounds as positive reinforcement whenever occurred a desired action (e.g., place

the piece in the right spot, or finish the game) to encourage the child. However, there should

be an option to turn off all the sounds, if the child did not react well to them. Another idea that

emerged was the possibility of an even easier level, in that level, the pieces would already be

at the right angle. The child would have to simply drag them to the right place. These sugges-

tions were all taken into consideration and added to this game in the best possible manner, as

presented in subsection .

In this reunion, we scheduled the sessions with three children fit for this game (i.e., children aged

between 3 and 10 years old). A boy with 3 years old, another with 6 and a third boy with 7 years

(all with moderate autism). This session aimed to analyze children’s reaction to the game and

the therapist’s interventions. In order to replicate the therapist behavior in the robot.

The first child, the youngest, was the one that demonstrated more difficulties with fine motor

skills. He held two games in the simple mode (i.e., it would only be necessary to drag the

pieces). He took about three minutes to complete each puzzle, the second game consumed


relatively less time than the first, which can be concluded that improved the understanding of the

game. At the beginning, the child showed signs of being nervous and impatient (e.g., rubbing

his hands together, looking at his father and calling him), perhaps by the presence of new and

strange people, and the unusual activity (i.e., the tablet game) during therapy. The therapist was

essential to calm the child. However when he started playing, he was focused on the game and

started to get excited. As this was a younger child and did not have enough hands dexterity,

he had some difficulty to drag the pieces. The therapist had to grab the child’s hand a few

times to teach him. This participant quickly realized what was the goal of the game, but he took

a lot of therapist’s help throughout the game. He typically repeated the sentences uttered by

the therapist, especially when the piece escaped (e.g., Oops, oh no - negative sentences that

amused him). This child, according to the therapist, had difficulty concentrating on a task. But

throughout the game remained concentrated during most of the time. At the end of each game,

when he heard the sound of victory, he smiled for finishing the puzzle.

The second participant was an older boy that was fascinated by tablets and computers. And

he was excited to play a new game on a tablet. This child held five games: two games on the

easy level and simple mode; another one in the medium level and simple mode; another game

in the easy level and with button rotation mode; and finally a game in the hard level and with

button rotation mode. He usually took 1 to 2 minutes to complete the puzzles, except the last one

that took nearly 4 minutes. This child showed great enthusiasm for the game and, as the first

participant, he was excited every time he finished the puzzle. He also repeated what the therapist

said when occurred something negative in the game. Unlike the former, this participant has not

shown much difficulty at the fine motor level, but he also needed some help from the therapist

for being able to complete the game. At the end of the third game, this child was already more

familiar with the game, then showed a greater outpouring. So he tried to drag the pieces with

too much speed, and they did not respond to that kind of touch. Thus, he began to play with

harder touches on the tablet. The therapist had to intervene in this situation, asking him to calm

down. This boy, unlike the first, played with the piece rotation (trough button) feature. Immediately

realized the goal of this feature, and could successfully rotate the piece to the right angle.

The third and final participant was a boy accustomed to tablets and this sort of games. So the

therapist started the game by saying, For you this game is a piece of cake. He promptly realized

the aim of the game even faster than the other two participants. This child did not need help from

a therapist during all the games. He played 6 games, most of them in the hard level. He was

always focused on the game, and did not react to outside distractions (e.g., people talking at his

side). Of all participants, he was the only one who experienced the rotation option with the finger

(the therapist assumed he was prepared for this feature). Without any difficulty, he managed to

4.2. SOCIAL ROBOT - NAO 35

Figure 4.15: NAO robot

complete the game in the hardest rotation option. This demonstrates how comfortable he was

around tablets and this kind of technology.

To summarize, we can conclude with this study that the game in the tablet is captivating, and the

feedback produces the desired effect. The children realized the goal of the game and improved

slightly over the games. With this study, we realize what kind of utterances and advice the

therapist gives throughout the game so that children can play. Some of the phrases used to

concentrate in the game were Where are you going to place this one?, Where is the green

square?, or Search for a small triangle to place in here. Each time the piece did not fit said Uhoh

it escaped!, Oops, or It was almost there. To positively reinforce, the therapist used phrases such

as Good! You did it, Only 3 pieces left. In order to help to understand the need to rotate the

piece: Do you think it will fit in here? or What do we need to do to the pink triangle?.

However, there are some behaviors that the robot cannot replicate. In the first participant the

therapist had to grab his hand to help him, and in the second had to calm him down due to

the aggressiveness. So, it is always important to have the presence of the therapist during the

interventions.

4.2 Social Robot - NAO

For our work, we decided to use the robot NAO (Figure 4.15), the first humanoid robot launched

by Aldebaran Robotics1. It is a social interaction oriented robot, with an anthropomorphic appear-

ance, perfect for interacting with autistic children as a peer (Dautenhahn et al., 2009). The robot

is 57cm tall, weighing 5.4 kg. NAO has a total of 25 degrees of freedom (DOF), 11 DOF for the1www.aldebaran.com/en/cool-robots/nao


NAO V5 SpecificationsHeight ∼ 57 cmWeight ∼ 5.4 kgAutonomy ∼ 60 min (active use) ∼ 90 min (normal use)CPU Intel Atom 1.6 GHzConnectivity Wi-Fi, Ethernet, USB

Interaction 2 loudspeakers, 4 microphones, 2 HD cameras res. 1280x960,2 I/R and multiple LEDs

Sensors Force sensitive resistors, accelerometer, gyrometer, sonars,joint position, contact and tactile

Built-In OS NAOqi 2.1ProgrammingLanguages C, C++, Python, Java, MATLAB, Urbi, .Net

Table 4.1: NAO basic technical specifications (Aldebaran Robotics, 2016).

lower part (legs and pelvis) and 14 DOF for the upper part (trunk, arms, and head). These char-

acteristics together with the electric motors and actuators, accelerometer, gyrometer, and sonars,

provides the robot more stability and smoothness in its walk and other movements (Shamsud-

din et al., 2011). Furthermore, it has two speakers (in its ears), several microphones and led

lights (RGB Full color) in the eyes that can change its color and form. NAO is equipped with

voice recognition and locates where the sound is coming from, so it can turn its head towards

the speaker (Aldebaran Robotics, 2016; Shamsuddin et al., 2011). Robot’s Intel ATOM 1,6 GHz

CPU is located in the head and an additional ARM-9 processor is in its torso. In the Table 4.1 it

is possible to observe more details of this robot.

NAOqi is the main software that runs on the robot and controls it. This software includes a reliable

cross-platform framework that provides a stable foundation on which can improve NAO’s func-

tionality (Aldebaran Robotics, 2016). NAOqi meets common robotics needs such as parallelism,

resources, synchronization, and events. NaoQi’s functions can be called in Python, Urbi, C, C++

and .Net.

4.2.1 Motivation

Due to unavailability of the robot, we could not use what we had initially proposed - the MOnarCH

robotic platform (Messias et al., 2014). So we had to choose a robot to replace it that was also

suitable for autistic children. The robot NAO was our choice for the reasons that are enumerated

below.

4.2. SOCIAL ROBOT - NAO 37

4.2.1.1 Design

Despite NAO’s anthropomorphic appearance, it has minimal face characteristics with some ba-

sic traits common to people (e.g. lateral symmetry, two eyes, and a mouth). Human-like robots

are suitable to engage the children with ASD in therapy and to generalize the interaction skills

learned. However, the presence of too many details or colors in the robot can over stimulate

children with ASD and have an adverse effect. Furthermore, robot’s appearance should not be

excessively realistic, because making the robot too human-like might diminish the enthusiasm

of the child (Giullian et al., 2010). In addition, Billard et al. (2007) concluded that autistic chil-

dren prefer interacting with a plain robot over a human-like robot. Thus, a minimalist and clean

appearance are optimal to interact with these children.

Moreover, the robot should be the same size or less than the child, in order to be less intimidating

(Giullian et al., 2010). The robot NAO is the size of a two-year-old child, which validates this

requirement.

4.2.1.2 Feedback

In the Larson’s study (Larson et al., 2011) it was proved that the children with ASDs prefer con-

crete feedback (e.g., lights, colors and sounds). NAO can perform countless gestures and move-

ments, it has LEDs that can change color, and moreover has speakers that allow speech and

sounds. Although, its minimalistic face does not allow NAO to express facial emotions. So,

affective behavior can only be achieved by body gestures.

4.2.1.3 Setup

The last requirement equally important was the ease of installation and transportation of the

robot, considering that it would be necessary to transport the robot from the laboratory to the

testing sessions multiple times. As mentioned earlier, NAO weighs 5.4 kg and measures 57 cm

which makes it an object of easy transport. As indicated in the table 4.1, the robot has Wi-Fi

connection which allows it to be connected in a network to easily communicate with the computer

and the tablet used in this thesis. In addition, its autonomy is more than an hour, which is sufficient

for testing sessions.


Figure 4.16: Thalamus Architecture

4.2.2 Integration Layer

The Tangram game was developed in Unity 51 in C# and runs in an Android tablet. To communi-

cate with the rest of the system, it was used a bridge, also developed in C#, that connects with the

game in the tablet, using a C# client within the Thalamus middleware. Thalamus (Ribeiro et al.,

2012) is a cross-media body interface for multiple simultaneous artificial embodied characters

(Figure 4.16). It acts as an abstract interface to the robot’s controls (e.g., reading sensors, deal-

ing with text-to-speech or controlling robot’s body). In our work, it was responsible for dealing with

functions of NAOqi and the Skene module. Skene (Ribeiro et al., 2014) is a semi-autonomous

behavior planner. Its input consists in a high-level behavior description language (Skene Utter-

ances) and perception information (such as target locations). And its output is BML and non-BML

actions (e.g., sounds or application commands).

The Thalamus character contains several modules that exchange perceptions and actions be-

tween them (Figure 4.16). Perceptions are the notification of events and Actions are requests to

accomplish something. These modules deal with behavior, with perception, or even with decision

making. Each module has Sensors and Effectors. A Sensor corresponds to the subscription and

reception of a set of messages, and a Effector corresponds to the announcement and publication

on a set of messages. A module can both subscribe and publish Perceptions and Actions.

4.3 Tutor Mode - Prompting

The interaction between robot and child has two conditions. The Tutor Mode is the first one and

has the purpose of helping and teaching the child during the game. The other is presented in the

next section. This present condition is adequate for children that have several difficulties playing

puzzle games (e.g., motor, reasoning, and so on).

In both modes, the robot begins by presenting itself and then explains the game. After that the1https://unity3d.com/

4.3. TUTOR MODE - PROMPTING 39

Figure 4.17: Start screen for Tutor and PeerMode

Figure 4.18: Start screen with game set-tings

child can click the Play button (Figure 4.17) to start the game. Then NAO explains how to rotate

the pieces, depending on which rotation mode is active.

4.3.1 Prompt Systems

During the game, the robot helps the child whenever is necessary. In the case the child insists

on placing the piece (1) in the wrong place, or (2) with the wrong angle, the agent begins the

prompt system. The prompt system is a set of cues with the aim of helping the player in a certain

task. Greczek et al. (2014) proved that graded cueing has good results and is well suited for most

children with ASD. This case has several prompts:

• P0 - no prompts

• P1 - the agent encourages the child to think about his/her decision (e.g., (1) Are you sure

that this piece is in the right place?, (2) Do you not think that this piece needs a small

change?);

• P2(1) - the agent gives a clue about the right spot (e.g., Why do you not try to put it next to

the yellow square?);

• P2(2) - the agent gives a clue about the right angle (e.g., Try clicking 4 times in the rotate

button, Why do you not try to rotate the piece with your finger to the left?);

• P3 - the correct spot starts to shine.

For the situation of the wrong angle piece, in any of the three prompt levels, when the child hits

the right angle, NAO tells to stop rotating and try to fit it in. For the simpler games (less than 7

pieces), as it is directed to children in the spectrum with fewer capacities, in P1, P2 and P3 are

always displayed visual cues along with the utterance. Also, there is another prompt system in


case the child does not move any piece within a few seconds. These prompts have the purpose

of stimulating the concentration on the game:

• P0 - no prompts

• P1 - random piece shakes and the agent asks where should it go (e.g., Looks like the blue

square really want to escape. Where do you think it should go?);

• P2 - the agent gives a clue about the right spot (e.g., Why not try to put the pink triangle in

the upper right corner?, I think the blue square stood well next to the yellow triangle);

• P3 - the correct spot starts to shine.

The visual stimulation (i.e., piece vibrating) is another form to maintain the child’s focus and

interest in the game. Besides that, in this prompt system for the game with fewer pieces, the

visual cues arrive in the P2. In the hard level (Figure 4.13), the robot can also show the final

puzzle within a few seconds, both in P1 and P2, as the professionals of CDC had suggested.

In both cases, the game starts at P0 level. If any of the three above options arise, the game goes

to P1 level. After some insistence still does not take effect on the child, the agent moves to the

next prompt level, and so on. After some time on the P3 level, if the child does not make any

action, the agent asks if he or she wants to give up on the puzzle, thus getting a new one with a

lower difficulty or if he or she wills at giving up at the game. The agent has to decide what action

it should perform in both prompt systems, so it does not deliver an excess of information to the

child. So it considers the information previously provided and also the current game state (e.g.,

how many mistakes made, how long without playing).

In addition to this two prompt systems, there are also other types of help. Whenever the child has

problems in moving pieces, the robot indicates in detail how he or she should drag it. In a normal

therapy session, the therapist would help the child by holding his or her finger and simulating the

drag (as it was described in Section 4.1.3.2). However, this robot is not capable of performing

that sort of help. This is one of the disadvantages of using robots for helping people to develop

motor skills. In our study, the therapist is always present, so in extreme cases, he or she can

intervene. Another type of help is provided when the child drop the piece near its right place,

but since the distance threshold set, the piece does not anchor to it. This is another case of fine

motor difficulties, so NAO encourages the child to drop the piece again in this place but more

carefully.

Clues should be simple and clear, it is important to have a familiar and basic language. So there

are two sets of utterances, one with normal phrases, and the other with brief and direct phrases.

The second one is a clear set of directions on how to proceed and is aimed for children with

4.4. PEER MODE - TURN-TAKING GAME 41

extensive linguistic impairments. The robot utterances were all based on the ones transmitted by

the therapist in the three observed sessions (Section 4.1.3.2).

4.4 Peer Mode - Turn-Taking Game

As was mentioned in Chapter 2, Turn-taking is an important social skill for communicating and

playing with others, and also developing friendships. However, some children with ASD have

difficulties in developing this particular skill. Thus, we decided to develop a condition in this game

that could enhance this competence. In the Peer Mode, the robot plays a turn-taking cooperative

Tangram game with the child. Teach turn-taking involves an understanding of the reason to take

turns, developing self-regulation aptitudes, knowing what to do (or to do not) when is waiting for

a turn, and knowing when to take a turn. The aim of this condition is to establish taking turns,

teach the child to wait for his/her turn and to incentive the children to help the other even when it

is not their turn.

4.4.1 Taking turns

The robot chooses randomly who gets to play first, and from there on, each one plays alternately.

Each time they switch shifts, the robot explicitly says Now I am playing or It’s your turn to play

followed by a gesture pointing to the child. For children who are not familiar with pronouns, the

set of simpler utterances replace the phrases by NAO’s turn or John’s turn (i.e., the name of

the child). As always it is all said in a familiar language in a relaxed scope. If the child tries to

play in the NAO’s turn, the piece will not move, and the robot will repeat that it is its turn. Thus,

reinforcing the idea that the child should not play in the others’ turn. Whenever the participant

does not play in its shift for several seconds, the robot repeats the establishing turns phrase.

4.4.2 Cooperative Feature

Autistic children are most often observed to be engaged in solitary independent play. To stimulate

child’s cooperative capacities, occasionally NAO asks for help during its turn. It can ask for

touching in the place of the solution where a certain piece fits. Or it can request for the child to

rotate a particular piece so the robot could fit it. In the first case, if the child touches the wrong

place, the robot simulates the drag of the piece to the pointed place and back to its original

spot. Then, it indicates that it is the wrong place and encourages him to say another one. If it is

right, the robot drags the piece and thanks the child enthusiastically. In the second case, NAO


Figure 4.19: Visual feedback - Fireworks Figure 4.20: Screen with name input

waits until the piece is at the right angle to place the piece and show appreciation. If after some

seconds it continues incorrect, the robot reinforces that the piece needs a slight adjustment.

Since this method is suitable for children who have mastered the game, the aid provided by the

robot are not as specific as in the previous condition. But still NAO gives some help, either when

the child drags the piece in the wrong place or when places it with the wrong angle. This help

corresponds to the P1 and P2 levels of the first aforementioned prompt system.

4.5 Feedback

As it was discussed by Bernardini et al. (2014), children with ASD should receive positive feed-

back in order to maintain interest and experience a sense of self-efficacy and accomplishment.

Besides, visual and auditory feedback make the interaction more appealing and comprehensible

(Battocchi et al., 2009).

Whenever the child places a piece in the right spot, the agent gives positive feedback through

congratulations (e.g., It is correct! or Very Good!) and/or other social behaviors, such as ges-

tures. Furthermore, when placed in the right place, the piece has an anchoring animation so that

it can no longer be moved. It is also launched a sound when the piece fits to reinforce this idea.

It is worth to mention that children with ASD should not receive overly penalized negative feed-

back since these children tend to misinterpret this sort of feedback (Zancanaro et al., 2011). So,

when the piece is dragged to the incorrect place, there are no anchoring animation or sound,

and the piece returns to the position where it was before. The robot reacts negatively (depending

on the number of failed attempts) but only with gestures or a negative word/onomatopoeia. This

behavior demonstrates that the piece was dragged to the wrong place.

Once the puzzle is completed, the robot transmits a compliment message towards the child

(e.g., Good job! Puzzle completed!) with enthusiastic gestures. Additionally, the tablet evokes a

congratulation sound and materialize multiple fireworks upon the completed puzzle (Figure 4.19).

4.5. FEEDBACK 43

Figure 4.21: Choregraphe - NAO’s programing interface

This final reinforcement is mightiest than all of the other feedback, to convey the feeling of having

reached the final goal.

Since NAO can not express facial emotions (it just changes the eye color), it was difficult to

demonstrate its feelings. Thus, body gestures were used to transmit this sort of perceptions. All

robot’s behaviors were created in Choregraphe (Figure 4.21), NAO’s user-friendly programming

interface. Choregraphe gives access to all the functions provided by NaoQi through a graphical

environment (Pot et al., 2009). This tool provides a set of pre-programmed behaviors (i.e., a

piece of software controlling the robot) from high-level functions (walk, dance, lie down, stand up,

speech synthesis) to very low-level ones (reading sensors, turning LEDS on and off). It is possible

to create new behaviors by assembling these basic ones, without the need of programming them

in Python or C++.

Regarding NAO’s utterances, as mentioned before, they are all simple and familiar to the children.

There are also another set of utterances which are simpler and with direct instructions, for children

with linguistic impairments. In both set of utterances, in few of them, the robot mentions the

participant’s name, in order to act as an acquaintance of the children and to stimulate them

when they hear their name. For the robot to acquire this information, a screen was added in the

beginning of the game (Figure 4.20). This screen only appears one time and the name can be

entered by the therapist.


4.6 Interface for Tests with Therapist

In order to test this game without the robot, a screen with all the possible settings was created

(Figure 4.18). The puzzles continue to be selected randomly, with no repetitions, by the game

manager. Although every other setting can be specified by the therapist. In the upper left corner

is possible to select the difficulty level (Facil - Easy, Difıcil - Hard) and next to it the drop down list

may be used to set the number of pieces used in the game. Below the play button (i.e., Jogar ),

on the left side, the distance threshold can be set up (Longe - 4, Perto - 1) and in the right side,

the rotation mode. The game is played as in other conditions but without any intervention of the

robot. With only a difference in the hard level, a help button appears in the upper right corner

(Figure 4.13). This button can be used by the therapist to show the puzzle solution for a few

seconds, as suggested by the professionals from CDC, and it only appears in this interface. The

next chapter describes in detail the usefulness of this condition.

Chapter 5

Evaluation

After explaining the architecture used in this thesis, we now present the evaluation method-

ology. We proposed to explore how a humanoid robot could be incorporated as a tutor

or as a peer into therapeutic sessions with children with ASDs. We formulated the following

hypotheses in order to validate our proposal:

H1: A humanoid robot can be integrated into therapeutic intervention sessions with children with

ASDs as a support tool for getting them engaged in the Tangram game. It is expected that the

performance of the children (concentration in the game and task execution) in a Tangram game

with a humanoid robot is similar than with a therapist.

However, we expect to see differences between the two conditions with the robot:

H1.1: In Condition 1, the child is as or more autonomous playing Tangram than when playing with

a therapist.

H1.2: In Condition 2, the child respects his/her turn to play as much as with the therapist.

To validate this, we conducted two studies - one in Tutor Condition (TC) and another in Peer

Condition (PC). This chapter presents a detailed description of the conducted experiments, in-

cluding the setup, the used design and measures, the participants and experiments design for

each condition.

45

46 CHAPTER 5. EVALUATION

5.1 The Setup

Initially, the tests were only meant to be conducted in the CDC1 - Centro de Desenvolvimento

da Crianca Torrado da Silva (Child Development Center) of Hospital Garcia de Horta, Almada,

Portugal. CDC provides prevention, diagnosis, and treatment of all kinds of acute and chronic

neurological disorders and psychomotor developmental disorders. However, since we were lim-

ited in time, only 3 children from this center were able to participate in this study. This limited

number of participants imposed serious challenges on the results we would achieve. Thereafter,

we had to contact other institutions.

One was the public school Francisco de Arruda2 in Lisbon, which has a unit that is designed to

support students with special educational needs. These students attend the same classes as the

others, however, they have a specific educational program guided by a specialized teacher. Two

students of this school participated in our study.

We also developed some studies in the CADIn3 - Centro de Apoio ao Desenvolvimento Infantil

(Center for Support of Infant Development). This is a private institution dedicated to the treatment

and study of neurodevelopmental disorders, such as Autism Spectrum Disorders, ADHD (Atten-

tion Deficit Hyperactivity Disorder), Anxiety Disorders, Dyslexia, and so on. The professionals

of CADIn are specialized in the diagnosis and treatment, both therapeutic and medical. It was

possible to conduct studies with 3 children, 2 from CADIn Cascais and the remaining one from

CADIn Setubal.

The trials lasted about one month and a half. Each participant was chosen by the therapist or

teacher according to their characteristics. They performed 3 to 4 sessions, once a week. These

sessions took approximately 15 to 30 minutes of the therapeutic intervention or the supporting

class. The trials occurred on the same familiar room, with the same habitual therapist/teacher

for each child. Each session of our study involved: one child, one therapist, the robot and one

researcher. For some children, a family member was also present.

In the first session, the tablet is placed on a table and the two video cameras are already strate-

gically positioned in the room for when the child arrives. In the next sessions, the robot is also

already present in the room, in front of the tablet. The figure 5.22 represents the disposal of the

material. The camera 1 is directed to child’s face in order to analyze emotions, gaze, gestures,

vocalizations, and so on. The camera 2 records the interaction with the tablet. Still in the room,

further away from the child, is also present the researcher with two computers and a router, re-

sponsible for maintaining the connection between the Thalamus (in computer 1), the bridge (in1http://www.cdc-hgo.com/2http://aefarruda.pt/portal/3http://www.cadin.net/

5.2. EVALUATION METHODOLOGY 47

Figure 5.22: Trial’s setup

computer 2), the game (in the tablet) and the robot. During the interaction, the therapist maintains

his/her position next to the child, and if necessarily intervenes. The child plays 4 games in each

session, but only if not showing signs of annoyance or discomfort. After the trial has finished, the

researcher leaves the room with all the material, and the therapeutic session or class proceeds

normally. At the final of the session or class, the researcher discusses with the therapist/teacher

about the interaction, after he/she filled in the questionnaires.

5.2 Evaluation Methodology

Since children in the spectrum can be so different and present such various characteristics from

each other, we decided to base our study on Single-subject Design. In the Subsection 5.2.1, we

describe this design and explain in detail the reasons for this being one of the best approaches

to assess autistic children.

In order to evaluate our methodology, we considered quantitative and qualitative assessments.

The quantitative metrics include not only common HRI metrics used in ASD research (Begum

et al., 2015) but also metrics that measure performance in the game. The qualitative assessment

provide an additional comprehension of the obtained outcomes, that may have influenced the

conduct over the trials. Some details are described narratively because they were not possible

to be evaluated.

5.2.1 Single Subject Research Design

Single Subject Research Designs are commonly used to study the behavior transformation and

social learning a given individual exhibits during a treatment (Gast & Ledford, 2009; Horner et al.,


2005). This design is normally applied in studies where the sample size is one, or the individuals

can be considered as a single group. This incorporates the baseline logic principle: the partici-

pants serve as their own control. Single-subject methodology derives from behavioral psychology

and applied behavioral analysis, however, it has been applied by researchers in multiple disci-

plines such as special education, communication therapy, occupational therapy, and so on. As

children with ASD are so different from each other, and one individual does not present the same

characteristics and impairments from another, they can not be compared between themselves or

be considered to be one homogeneous group. Thus, we thought the single subject design would

be the best alternative to study our work and thereby we could model each trial according to each

child.

In single subject design studies, each participant is exposed to a non-treatment condition, the

baseline. This condition consists in repeatedly measuring the performance (target behavior) of a

participant before experimental phases. In order to compare the effects of an intervention with the

performance during baseline, it is important to obtain a stable baseline pattern of performance for

each participant (Rogers & Graham, 2008). Therefore, only when this is verified is that one can

begin the intervention condition. The experimental control is introduced and the target behavior

continues to be observed and recorded in the intervention condition. Each dependent variable is

quantitatively measured over time, in a valid manner and described with replicable precision. The

independent variable is manipulated and controlled by the experimenter, and also described with

replicable precision. The baseline (A) and intervention (B) conditions are gradually alternated

across time, depending on the research design used, for example, A-B-A-B or withdrawal design

or A-B-A) (Gast & Ledford, 2009; Horner et al., 2005). The design strategy used on our research

was the A-B-A, we begin with the baseline, then the intervention, and finally another control

phase after the end of the experimental phase.

The five characteristics that summarize single subject research are:

• Reliable measurement - The study conditions, for example, time of day and location, should

be standardized. It is essential to have consistency in measurement to compare the before

and after the treatment;

• Repeated measurement - The same behavior is measured multiple times to obtain a clear

pattern or consistency in the behavior over the study;

• Description of conditions - To reinforce validity is necessary a detailed description of partic-

ipants and conditions of the measurement;

• Baseline and treatment conditions - In this study, the participant is exposed to, at least,

5.2. EVALUATION METHODOLOGY 49

one baseline and one treatment condition. Both need to be sufficiently long to accomplish

stability in the target behavior;

• Single-variable rule - Only one (at most two, depending on the study) variable should be

changed from baseline to treatment conditions.

5.2.2 Data Collection

Before the trials with each child begin, the therapist fills out a form (presented in Appendix B)

with the child description and his or her autism level according to the ADOS. Since children with

ASD present a variable range or disabilities, this form provides us necessary information about

the child, to correlate to some results obtained (i.e., to realize if those were positive or negative

outcomes) and to adapt the game to their difficulties (e.g., use the simple set of utterances, or

choose puzzles with less pieces).

At the end of the session, the therapist answers the questionnaires (presented in Appendix

C) upon before and after the interaction. There are two different questionnaires depending on

whether the interaction was only with the therapist (Figure C.66) or the robot (Figure C.67). In the

first session, the therapist also fills the questionnaire on the first impressions of the child about

the robot (Figure C.68). Furthermore, at the end of the session, the therapist also discusses

with the investigator some interesting aspects, such as the reaction of children throughout the

game and the emotional state of the child that day, that could influence the results. All this data

is collected. As this is a longitudinal study, the final questionnaire must concern all sessions. So

the therapists respond to three questions: what would be expected, what was verified, and what

was surprising.

Finally, the researcher visualized a total of 10 hours of video recordings to collect information

about eye gaze, turn-taking and game performance, help and external interventions, and others.

The eye gaze time was coded in the video using VCode annotation tool. VCode (Hagedorn et al.,

2008) is an application which provides a tool to annotate events (i.e., by marking action start

and finish) from an observational research video. To evaluate the game performance, the logs

recorded during all games were analyzed.

5.2.3 Measures

All the metrics presented below were chosen for being the most appropriate to our study and

were adapted to this context. Also, they were all approved by the therapists who participated in

the study. Some were measured quantitatively and others qualitatively.


In order to measure child’s task performance, we resort to questionnaires (presented in Appendix

C), game logs and video viewing.

Task Performance

• Time to complete the puzzle - average time spent until the puzzle is complete;

• Time to place the piece (TC) - average time spent to place each piece in the puzzle;

• Turn time (PC) - average time of child’s turn in each game;

• Failed attempts to rotate the piece (only for the finger rotation mode) - number of times the

child failed to rotate the piece;

• Failed attempts to drag the piece - the amount of failed attempts to drag the piece;

• Failed attempts to place the piece - the amount of attempts to put it in the wrong place;

• Attempts to place the piece close to the right place - the amount of attempts to put the piece

in the right spot;

• Failed attempts to finger point (TC) - the number of times the child wrongly finger-pointed

to a spot (after therapist has asked, in the original Tangram game);

• Correct attempts to finger point (TC) - the number of times the child correctly finger-pointed

to a spot (after therapist has asked, in the original Tangram game);

• Attempts to destroy (TC) - the number of times already placed pieces were destroyed (in

the original Tangram game);

• Attempts during others’ turn (PC) - number of play attempts during robot’s/therapist’s turn;

• Responses to help requests (PC) - number of times that the child responds immediately to

the robot requests for help;

• Realize it is child’s turn (PC) - number of times the child realizes immediately it was his turn;

• Number of help - the amount of aid provided by the robot in order to assist the performance,

the motor difficulties, or the attempts out of turn (PC).

• External interventions - the number of external interventions provided by the therapist to

help the child during the game, to stimulate the re-focus on the task, or to reinforce the

actual turn (PC).

5.3. THE PARTICIPANTS 51

In order to measure interest and engagement of the children in the robot, the HRI (Human Robot

Interaction) metrics are the most suitable (Begum et al., 2015; Scassellati, 2005). However in

our study, we choose the ones which were best suited. To measure this, we analyzed the results

from questionnaires (presented in Appendix C) and from the video visualizations.

Affective Attributes Towards the Robot

• Gestures - number of gestures (e.g., pointing, waving) addressed to the robot;

• Vocalizations - number of times the child talked directly with the robot, or about the robot or

the game.

Furthermore, we still used other metric that we think are appropriate to measure the level of

concentration in the task:

• Gaze - the amount of time spent by the child gazing directly to the robot, tablet, therapist,

and elsewhere (measured through video annotations);

At the end of the sessions, the conditions in which the robot is used, the performance of the child

is quantitatively evaluated (i.e., in terms of the level of concentration and success in completing

the task) in similarity to the assessed baseline (i.e., game with the therapist). As mentioned

before, in Tutor Condition the child is as or more autonomous playing Tangram than when playing

with a therapist. This level of autonomy is reported in terms of the number of help and external

interventions and based on task performance metrics. And in Peer Condition, the child respects

his/her turn to play as much as with the therapist. This is evaluated in terms of the number of

times played in place of another and the number of times played in the right turn.

5.3 The Participants

Eight children participated in this study. They were divided into two groups, corresponding to the

two conditions: the children who played in the TC, and the others, in the PC. However, they were

not equally divided, only 1 participated in the TC and the remaining 7 played with the robot in the

PC. These eight children, 3 have therapy in the CDC of HGO, 2 have support classes in Francisco

de Arruda school and the other 3 attend therapy sessions in CADIn. The children were selected

to participate in the experiments by therapists and the teacher. All children were evaluated by

ADOS.

Children with greater difficulties in this type of games were chosen to play in the TC. This also

explains why the only child who played in this condition presented a lower functioning level. In


the PC, children that would already be accustomed to tablets and puzzle games were chosen.

They also presented some or many difficulties in turn-taking. The therapists decided which of the

conditions the child should attend. The discrepancy between the number of participants in each

of the conditions occurred mainly because the limited amount of subjects that could participate

in this study. As this was a longitudinal study, not every parent of the children had availability

(hourly or monetary) to attend therapy sessions every week over one month.

The guardians of all participants signed a consent form authorizing their participation in the study,

and the permission to record the sessions. The consent (presented in Appendix A) consists in

a short description of the work and its aim. Also mentions that all videos recorded will remain

classified and used exclusively for research purposes.

5.3.1 Tutor Condition

Since this condition aim was to help and give instructions to the child during the Tangram game,

target participants would have to manifest difficulties in this sort of games. Only one child fitted

in this condition: the impairments of this child were of lower functioning end. He was chosen to

play in this condition because of his cognitive difficulties.

This condition has fewer participants than the PC as a result of being difficult to find children of

the spectrum who were available to participate in the study and who fit the requirements of this

condition. In addition to this, we thought that in a short period of time, it would not be possible to

verify significant results in this condition due to the severity of impairments of these participants.

It would probably take months of study, with the repetition of the same behavior every week.

5.3.1.1 Experiments Design

In this Condition, the participant goes through 4 sessions. In the first session - baseline, the child

plays with the therapist the original Tangram game, then plays the tablet Tangram game also with

the therapist, and at the end, the robot is presented. In the first phase, the child has to assemble

one puzzle with the 7 physical Tangram pieces, with the help of the therapist. One drawing (in A4

sheet) where the pieces should be placed is provided. This drawing has seven places painted

with the colors of the respective pieces (as in the easy level of the tablet game).

Then the child plays 5 tablet games with fewer pieces or 2 games with seven pieces, depending

on child’s abilities. During the game, the therapist helps whenever it is necessary. Finally, the

robot is introduced to the child, NAO waves its arms as a greeting, as it says hello and its name.

The child is encouraged to touch the robot and explore its body. Children with autism often

5.3. THE PARTICIPANTS 53

have difficulty in tolerating variation in their routine (Robins et al., 2004). So this initial phase is

important to gradually lose their fear and reluctance to change of routines, and also to create

an affective bond with the robot. For some children, this phase may take few minutes as others

may need the entire session. It is important that the distance between the child and the robot

progressively decreases so that he/she can touch the robot without any fear. If in this initial

phase, the child feels comfortable with the robot, the session of the child playing the game in the

tablet with the robot may begin (it is further described in the next paragraph). Ideally, this session

would be held on a separate day of the baseline. However due to time constraints, it would only

be possible to conduct two sessions with the robot. So we decided it would be beneficial that

there were more of these sessions.

The second and third sessions consist in 4 games, played exclusively with the robot. The com-

plexity of the games can increase gradually or not, depending on the performance of the child.

In these sessions, the robot begins by explaining how the game is played and how the pieces

are rotated (depending on the rotation mode). Finally, in the fourth session, phases of the first

session are repeated (i.e., excluding the presentation of the robot, but including the phase robot

where plays with the child, before the games with the therapist). These sessions are conducted

one or more times a week, according to child’s availability.

5.3.2 Peer Condition

Seven children with ASD participated in this experiment. Preferably these participants should be

accustomed to puzzle games and tablets, and manifest some difficulties on turn-taking skills and

completing instructions. Therefore, all these seven participants were diagnosed as moderate or

mild autism. They were all chosen to participate in this study by their therapists.

5.3.2.1 Experiments Design

In the Peer Condition, each participant could perform 3 or 4 sessions, depending on their avail-

ability, performance and the will to continue. The design of this experiment is very similar to the

Tutor Condition, with some exceptions. The baseline, contrary to the previous condition, consists

of only 4 games with the 7 pieces played with the therapist in the turn-taking mode (the therapist

plays one piece, then the child places another). At the end of the session, the robot is also pre-

sented to the child. As in the previous condition, if this initial contact occurs naturally, the game

with the robot can begin.

The first and forthcoming sessions with NAO are also composed of 4 games with the robot in the


turn-taking mode. The complexity of the games can increase gradually or not, depending on the

performance of the child. At the beginning of each session, he explains the turn-taking flow and

the rotation mode, if necessary. Lastly, in the final session the child plays the last four games with

the robot and then plays additional four with the therapist in turn-taking mode.

Chapter 6

Results

In this chapter, we present and discuss the results of each experiment separately. For each

experiment, we report the sessions with all the participants individually and we finish by analyzing

the outcomes in order to comprehend to what degree our solution was capable of meeting our

objectives.

6.1 Tutor Condition

This section presents the results of one participant only. These results are discussed individually

at the end of this section.

6.1.1 Participant 1 - J.F.

J.F. is the oldest participant, he has 14 years old. He has severe autism and attends support-

ing functional behavioral classes at the Francisco de Arruda school. His linguistic, cognitive and

motor development are strongly underdeveloped. Also, he is not capable of taking turns, concen-

trating in a certain task, and focusing on communication. J.F. is interested in tablets and tablet

games, which are frequently played by him. Although showing interest in robots, he has never

interacted with any. He does not have much experience and interest in puzzles and the Tangram.

With this profile, we find more appropriate for this participant to play with Tutor version.

55

56 CHAPTER 6. RESULTS

Figure 6.23: Participant J.F.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) to place a piece per puzzle.

Figure 6.24: Participant J.F.: Average numberper puzzle of: wrong or close placement ofpieces, and drags of pieces.

Figure 6.25: Participant J.F.: Average numberof external interventions (from the teacher) andhelps (from the robot), per puzzle.

6.1. TUTOR CONDITION 57

Figure 6.26: Participant J.F.: Percentages of eyegaze per session. The baseline and final ses-sion have 2 data points each. The first one cor-responds to the original game and the last oneto the tablet game.

Figure 6.27: Participant J.F.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.

Figure 6.28: Participant J.F.: This plot presents the (1) number of received helps, (2) the numberof times already placed pieces were destroyed, (3) the number of times the child (wrongly andcorrectly) finger-pointed to a spot (after teacher has asked), and (4) the number of times of wrongand correct placement of pieces.


Baseline

In the baseline, J.F. played the original Tangram game with the aid of the teacher. According

to the teacher, that day he was well-disposed and curious, so he agreed to play this game that

was not his favorite. He did not give up despite all the difficulties he had throughout the game,

which is not usual for this student. Usually, he rejects completely a task if it is not of his interest

or if he faces many adversities. However, J.F. was not able to finish any puzzle without the help

from the teacher. He was not concentrated on the game (Figure 6.26), he was more interested in

hearing the positive and negative sounds produced by the teacher. This participant spent much

time exploring the pieces in repetitive movements, as it is possible to see in the gaze pieces

percentage in graph 6.26. Due to his motor impairments, this game was even more difficult,

whenever he placed a piece he destroyed all the others already placed (Figure 6.28). During this

initial session, he had to play only once (the rocket puzzle), but when he finished, he wanted to

continue to play another one (the cat puzzle). In the last puzzle, most of the pieces were placed

by the teacher. He played this game for 23 minutes (Table 6.9), the first puzzle took 19m50s and

the second 2m19.

After that, the second part of the baseline - the game on the tablet with the help of a teacher,

started. This session took about 6 minutes (Table 6.9) in which J.F. held five games with few

pieces (from 2 to 6 pieces, increasing the number of pieces in every game). He has always been

very focused on the game and was always willing to play another. The first game took longer

and required more aid, as it was the first time he played this Tangram version. But as can be

seen in the graph 6.23, his performance has improved over the games. He had some difficulties

in moving the pieces due to his fine motor impairments (Figure 6.24), so once in a while, he

dragged the hand of the teacher to move a piece for him. He laughed and was very excited when

saw the fireworks at the end of each puzzle. After 5 games he met NAO, he was very excited and

curious. He tried to establish physical contact with it multiple times, especially with its fingers,

the light button on the chest, and the screws of the legs. Due to the positive outcome of this first

interaction, we decided to start the session with the robot on the same day, so this participant

would have more sessions with it.

Intervention Sessions

The first session with the robot lasted about 3 minutes (Table 6.9). He played 5 games with

the robot (4 with fewer pieces and 1 with 7 pieces) all in the easy level without rotation. During

the interaction, he was very interested in playing with the robot screws, so the teacher had to

intervene multiple times. The interventions were due the rough touch and to make him refocus

6.1. TUTOR CONDITION 59

on the game. Despite being focused on the robot, he was not aware of its utterances, so the

teacher had also to strengthen these guidelines. He seemed to not being able to understand

what the robot was saying to him, nor did what he requested him. Thereafter, the robot could not

help the child throughout the game. He showed enthusiasm whenever he ended the game and

was always willing to play another one.

In the second session, he was still very interested in the game, but more absent, less enthusiastic

and talkative. He had many deconcentration periods in the game, so the robot had to alert him to

focus on the game. In some of these times the robot told him Do not let the orange triangle escape

and J.F. quickly touched the orange triangle and placed it in the right spot. It was not possible

to measure whether it was a result of what the robot said or not, although it happened multiple

times. Much of the external interventions were due to him being distant. In this session his

interest in the robot decreased dramatically, he looked less towards the robot (Figure 6.26) and

tried to touch it fewer times (Figure 6.27). However, after the end of the 4 games, he continued

to touch the tablet, demonstrating his will to play more puzzles, so he played two more. All the

puzzles had 7 pieces, so in average, he took longer than in the previous session (Figure 6.23),

especially on the second puzzle when he was more distracted.

In the third, as in the previous session, J.F. continued to be absent, very focused on the game, but

with no interest in the robot (i.e., he just tried to touch it once). He had more difficulty in dragging

the pieces, as he tried to touch both at the same time, which was not allowed. In these cases,

he did not listen to the robot interventions. In other cases where the robot asked him where he

could put a certain piece, he placed it in the right spot, as in the previous session. The games

were slightly more advanced, but it did not seem to have affected the performance (Figures 6.23

and 6.24). On the contrary, he did not need many aids or external interventions, as it is possible

to see in graph 6.25. He continued to play beyond the 4 games, but in the graph 6.23 are only

presented the results of 6 because he did not finish them all. Occasionally, he clicked the Home

button for no apparent reason. According to the teacher, he was exploring all the possibilities of

the game.

During the fourth session, J.F. was in a better mood and more excited. Again, when the robot

asked him where to put a particular piece, he placed it correctly. He was more focused and

interested in the robot than in the previous sessions (Figures 6.26 and 6.25). He tried again to

play with two pieces at the same time, without listening to NAO’s instructions to not do so. The

teacher had to intervene. Also, he played more games than the 4, however, he continued to tap

the home button occasionally. When at the end the teacher took the tablet to end the interaction,

he clutched to it to continue playing. The teacher mentioned that in this session he spent the

game sucking his tongue, which is a sign of pleasure for the task.


Final Session

After this interaction with the robot, he played 2 games on the tablet with the teacher almost

autonomously (Final Session). Finally, this participant played again the Tangram original puzzle

with the teacher. He was able to finish almost completely alone one of the puzzles, and when

he completed it, he wanted to continue playing. He took less than 7 minutes to finish the first

puzzle, and his performance was substantially better. He was more focused on the puzzle than

in exploring the geometric pieces, when compared to the baseline (Figure 6.26). He started the

second puzzle and played for 3 minutes, then started to get hungry and went away.

6.1.2 Discussion of the Results

As only one participant played in this condition, we can not compare results between participants.

So we only analyze J.F. results. According to the final questions answered by the teacher, she

was expecting that J.F. did not adhere to the game at all, because it was not of his interest.

However this participant enjoyed the game, which helped him to keep focused most of the time

(deconcentration is one of the impairments of this child). In the robot sessions, he was almost

as concentrated as when he was with the teacher. He was so captivated by the game that he

wanted to play more than the four proposed puzzles. It was possible to verify an increase of

autonomy over the games. Initially we thought he could only complete the puzzles with fewer

pieces, but then we realized that they were too easy for him. From the third session onwards,

he played on medium difficulty level and showed no problems with it. Also, this was one of the

sessions that took less time to complete and required less aid. Due to his limited mobility, he did

not play the games with piece rotation. The results in the sessions with the teacher were clearly

better than with the robot. Although, the robot was capable of taking the role of tutor, to direct the

participant’s attention back to the game, when he got distracted.

Throughout the games some external interventions were needed, in order to prevent him for

touching the robot violently and to keep him concentrated on the game. But the aid given by

the teacher and by the robot had similar values. Whereby can be said that his autonomy in both

conditions was similar.

His interest in the robot oscillated between sessions, as we can see in the graph 6.26. Clearly

in the first session, he was very interested in the robot, almost more than in the game, as it was

an exciting novelty. But this interest sharply declined in the second and third sessions. In these

sessions, he was more interested in the game. This can be translated as the habituation to the

robot factor.

6.2. PEER CONDITION 61

In the first game with the original Tangram, he spent much time exploring the shape of the pieces

and destroying all he had already built. Also, he needed much help from the teacher for being

able to complete the puzzle. In the final game with the original Tangram, the results were much

better. He was much more focused on the puzzle, than in analyzing the pieces. This was only

tested in just one day, so we can not confirm that this result was due to the games he had in the

tablet. However, the teacher said that his interest in the game and the results obtained in the

game with the original Tangram compared to baseline were surprising.

6.2 Peer Condition

This condition includes the participation of seven children with ASD. At the end of this section,

these results are analyzed collectively.

6.2.1 Participant 2 - M.A.

This participant has six years and has moderate autism according to ADOS. He frequents the

CDC of HGO to attend occupational therapy every once a week. M.A. presents a very good

cognitive and motor development, and also a good linguistic development for his age. Also, he

does not require great effort to taking turns, to concentrate on a certain task, and to focus on

communication. He is particularly interested in tablets and tablet games and these are tools that

are often used by him. Previously, M.A. had participated in other studies with robots. Also, he

already had played the original Tangram before, but had little interest in the game.

Baseline

M.A. attended 4 sessions - Baseline, First, Second, and Final (Table 6.9). Due to timing issues,

two of them were on the same day (Second and Final). It is possible to see in the graph 6.30

that in the baseline session, he was able to complete the puzzles without difficulty. The ones

with rotation, took longer to complete than the first two (Figure 6.29), as expected. Respecting

turn-taking (Table 6.2), M.A. noticed whenever it was his turn, he only tried to play one time in the

therapist shift in the first game when he did not understand the turn-taking flow. He was always

focused on the game. As it is possible to analyze in the graph 6.32, he spent most of the time

looking at the tablet. At the end of the 4 puzzles, he requested to play once more, however,

he played it alone so we did not analyze those results. Then, the robot was introduced to the

participant. He was not comfortable with its presence and adopted a posture of fear (with his


Figure 6.29: Participant M.A.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) of each child turn perpuzzle.

Figure 6.30: Participant M.A.: Average num-ber per puzzle of: wrong or close placement ofpieces, and drags and rotates of pieces.

Figure 6.31: Participant M.A.: Average numberof external interventions (from the therapist) andhelps (from the robot), per puzzle.


Figure 6.32: Participant M.A.: Percentages ofeye gaze per session.

Figure 6.33: Participant M.A.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.

back to the robot). M.A. said he did not want to look at the robot and asked to turn it off. For

this reason, we decided that the first session with the robot would be just in the following week.

Nevertheless, when NAO started to talk (asking if he liked to play the game on the tablet, and

what had been his favorite puzzle), he replied. After this, M.A. adopted other posture and started

to smile. When the robot said it would return next week to play with him, he was excited.


In the first session, the robot was already in the room when the participant arrived. At the begin-

ning, he was a bit afraid to approach the robot, but quickly lost the fear and was excited play with

him. The performance in the game was slightly lower to the baseline, as it is possible to see on

graph 6.30. However, on average it takes about the same time to complete the puzzles (Figure

6.29). M.A. understood the concept of turn-taking, he played in his turn and waited for the robot

to play in its turn (it was also possible to verify this with the changes of his body posture). He

tried to play in place of the robot only once in the first game. As shown in Figure 6.33, there

was a large number of vocalizations, because this participant answered the majority of robot’s

questions (e.g., if he wanted to start a new game, if he liked the puzzle, and so on). Interestingly

he began to imitate the robot’s request for help. When it was his turn, he asked NAO to help him

with a particular piece (even without requiring any help). Since the robot did not respond to his

request, he continued to play. He repeated this behavior six times in this session. Also, he tried

to help NAO multiple times, even when it did not request.

In the second session with the robot, M.A.’s performance was good taking into account the in-

creased game difficulty. He did not have any trouble with the turn-taking (Table 6.2). In addition,

he continued to copy the help request in his turn (3 times), and also mimicked some movements


performed by the robot. Often, he enthusiastically shared with the therapist that were only a cer-

tain number of pieces left (showing the corresponding number with his fingers). When the robot

said it had to leave he said ohh, sad that the game was already finished.

Final Session

After the 4 games with the robot, he played another 4 games with the therapist. Although the first

three of these games presented a higher level of difficulty, M.A. was able to solve them in a short

time, as we can see on graph 6.29. He always realized whenever it was his turn. Also in this

session, he used the help request phrases three times as in previous, and the therapist replied

that he was capable to do it by himself. Still repeated other phrases spoken by the robot (e.g.,

Only 2 pieces remaining).

The therapist commented that it was very interesting the imitation of the behavior for help re-

questing. These children do not usually express in words the requests for aid. They normally

point or direct the other person’s hand so that they help them. Although this behavior has been

repeated in the two sessions, including the final one with the therapist, it was not possible to ex-

amine the ability of generalization for this action. Many more long-term studies would be needed

to verify it.

Baseline First Second FinalPlayed in His Turn 100% 100% 100% 100%Played Out of Turn 1 1 0 0Helped the Robot - 100% 100% -

Table 6.2: Participant M.A.: Percentage of times he played in his turn, times he helped the robotand the number of times he played out of turn.

6.2.2 Participant 3 - J.N.

The Participant 3 has four years and has mild autism according to ADOS. He also frequents the

CDC of HGO to attend occupational therapy. J.N. presents a good cognitive and motor devel-

opment. However, he is undeveloped linguistically given his age. Also, he has some difficulties

taking turns and focusing on communication. He shows some interest and experience in tablets,

tablet games and puzzles. Although, he despises robots and the Tangram game. As the previ-

ous participant, he had also participated in other studies with robots, but he did not expressed

positive reactions.


Figure 6.34: Participant J.N.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) of each child turn perpuzzle.

Figure 6.35: Participant J.N.: Average num-ber per puzzle of: wrong or close placement ofpieces, and drags and rotates of pieces.

Figure 6.36: Participant J.N.: Average numberof external interventions (from the therapist) andhelps (from the robot), per puzzle.


Figure 6.37: Participant J.N.: Percentages ofeye gaze per session.

Figure 6.38: Participant J.N.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.

Baseline

J.N. began to play 3 games alone in the first session, in order to learn how to play. Only then

he played another two with the therapist in turn-taking mode. In the first 3, he was excited and

played without difficulty. But when he had to share the game with the therapist he began to get

angry (screamed and beat with his feet and hands), because he preferred to play alone, which

is common in these children. He tried to play in therapist’s turns for several times, she had to

say who could play in each turn and still he did not respect (Table 6.3). When he met the robot,

he was very curious. But when NAO said he would play it with him, automatically changed his

mood, he got angry and yelling at the robot to leave because he did not want to play with it. The

therapist insisted about 10 minutes showing the tablet and playing for him. Though he seemed

interested in the game and also tried to play, at the same time he refused to touch the tablet.

NAO said goodbye to him and said he would return next week, he said no. The therapist referred

that this child often had this type of reactions to new changes in his routine that were out of his

control, or when he is thwarted.


In the first session with the robot, he had a similar reaction as in the baseline. When he saw

the robot in the room, he did not want to enter and shouted for him to go away. He repeatedly

pointed to the robot and named it bad. The therapist decided to let him play with his favorite

game to calm him down, but NAO was still there. Later the therapist insisted for him to play with

the robot, he touched a few times on the tablet, but as soon as he realized what he was doing

he fled, and insisted that was not going to play. The therapist decided to cover the robot with a


towel (still possible to hear it) to test whether his reaction is altered. J.N. played 3 games in which

did not respect robot’s instructions, however, he was aware of what it was telling him because

he responded to their help requests. He was able to understand what the robot told him but did

not cooperate because he does not like being contradicted. He wanted to play alone. In the

last game, we uncovered robot’s head and the participant reacted negatively, but he finished the

game and said he did not have wanted to play more. In this session, he requested a lot of help

and external interventions (Figure 6.36).

In the third session the therapist decided to change the order, first he played with his favorite toys

and then played with the robot. When he arrived at the room, the robot was already covered by

a towel. He knew NAO was there, but coped well with it. We uncovered the robot as he happily

played with plasticine, and had a positive reaction - smiled at it. The therapist decided to enter

the tablet so that he could play with the NAO. In the first game, he was not aware of the robot’s

instructions and was always trying to play out of turn. And the therapist had to remind him that

he had to help the robot when it asked him. At the end of this puzzle, he asked to play more,

demonstrating an increased interest. In the third and fourth puzzles, he was more concentrated

on robot’s instructions and continued to help him. At the end of the game, he was visibly tired from

being so focused. This session went better than the others, but we can not conclude whether it

was the habituation to the robot or if it was due to the change of session order.

In the fourth session the robot was already in the room and when he arrived he wanted to go

play with it. In this session J.N. was more focused on what the robot told him to do, he did

not play so often in its turn and he responded to help requests more frequently comparing with

previous sessions (Table 6.3). However from the end of the 3rd game, he started to say that

he did not want to play anymore, but ended up clicking the Play button to the 4th game. He

was very concentrated in the tablet throughout the games (Figure 6.37). Although the increased

game difficulty, he improved his performance as it is possible to see in graph 6.35. The last was

the hardest puzzle and he started to get angry for not being able to place any piece. When the

therapist tried to help, he was angry with her. This puzzle took longer (Figure 6.34) and required

more help and external interventions (Figure 6.36). At the end, he said goodbye to the robot and

waved. The therapist told him that now he was going to play with her, but he got angry again

and said he did not want to play anymore. She insisted for 5 minutes and then gave up because

he was visibly tired and irritable. Therefore, it is not possible to analyze the last session with the

therapist.


Baseline First Second ThirdPlayed in His Turn 100% 42% 47% 69%Played Out of Turn 7 23 30 10Helped the Robot - 100% 60% 75%

Table 6.3: Participant J.N.: Percentage of times he played in his turn, times he helped the robotand the number of times he played out of turn.

Figure 6.39: Participant H.R.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) of each child turn perpuzzle.

6.2.3 Participant 4 - H.R.

H.R. has six years and has mild to moderate autism according to ADOS. As the previous two

participants, he frequents the CDC of HGO to attend occupational therapy. He is usually in a

good mood on these sessions. H.R. presents a good linguistic and cognitive development, and

a very good motor development for his age. Also, he does not require great effort to take turns,

to concentrate on a certain task, and to focus on communication. He is particularly interested in

tablets, and also shows some enthusiasm for tablet games, robots, and puzzles. H.R. had also

participated in other studies with robots. Additionally, he already had played the original Tangram

before and had little interest in this game.


Figure 6.40: Participant H.R.: Average num-ber per puzzle of: wrong or close placement ofpieces, and drags and rotates of pieces.

Figure 6.41: Participant H.R.: Average numberof external interventions (from the therapist) andhelps (from the robot), per puzzle.

Figure 6.42: Participant H.R.: Percentages ofeye gaze per session.

Figure 6.43: Participant H.R.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.


Baseline

In the baseline, the participant 4 played two games alone (with the help of the therapist) in order

to learn the game. After that, he played another two games with the therapist in turn-taking

mode. It was slightly complicated to introduce the turn-taking because he wanted to continue to

play alone. Although he had no trouble finding out who should play in each turn, the therapist

had to always reinforce the turn. He was very focused on the game and got excited every time he

placed a piece or finished the puzzle. At the end of each puzzle, he tried to guess what animal

was. When H.R. met the robot, after the 4 games, he was very curious about it but had fear of

approaching it.


At the same session as the baseline, he played four games with the robot in turn-taking mode.

As it is possible to see in the Table 6.4, he tried to play a few times out of his turn. This happened

just before the robot stated that it was its turn to play. During the game, he was excited and

answered the robot’s questions (Figure 6.43). Sometimes he said good! when the robot put a

piece in the right place. However he was not entirely comfortable with the presence of the robot,

and its movements frightened him up because of the noise its joints made.

In the second session the participant was very cheerful, he was delighted when he came to the

room and saw the robot and the tablet. The difficulty of the games in this session was increasing,

but he showed no difficulty playing them (Figure 6.41), neither in taking turns (Table 6.4). A

few times he pointed to the robot when it was his turn to play (Figure 6.43). When the puzzle

was completed, he raised the arms enthusiastically, as did the robot. And always asked the

therapist to play another game. He was very focused throughout the game, smiled a lot and

began communication with the robot. At the end of session asked to touch the off button of NAO,

which shows that he was comfortable near the robot.

In the third interaction with the robot, H.R. seemed a little less enthusiastic about the robot and

the game than on the other days (Figures 6.42 and 6.43). When he arrived at the room said he

did not want to play, or to approach the robot. However when the robot asked him if he wanted

to play with him, he promptly answered Yes. He smiled a lot and was excited during the game.

He always respected each turn (Table 6.4). The difficulty of the game has increased substantially

due to his good performance in the previous. However he struggled in hard level games with

easier puzzles (Figure 6.41), so he asked the robot how he should put a particular piece. In this

session, he continued to be frightened by the movements of the robot.


In the last session, H.R. was once more afraid that the robot would leave its place. But he was

no longer frightened by their movements. Sometimes he pointed to the robot or to himself, to

indicate who should play. His body posture alternated as each turn. Although the difficulty levels

were similar than the previous session, his performance improved as we can see in the graphs

6.39 and 6.41. He was always focused and happy throughout the game.

Final Session

Finally, he played 4 games with the therapist in turn-taking mode, with an advanced level of

difficulty. He said whose turn it was to play, and mimicked some of the robot’s speech. Sometimes

he tried to help the therapist when she did not know where to put the piece. Interestingly when

the session ended, as the H.R. could solve the most difficult puzzles, the therapist decided to get

the original Tangram game. He could easily solve the first puzzle, and when the therapist wanted

to change the game, H.R. insisted on playing more Tangram puzzles. He wanted to complete the

duckling puzzle that had also done with the robot. During this interaction he used some of the

robot lines, for example, I am thinking (when it was NAO’s turn). Altogether he played 3 puzzles,

and he wanted to continue but the therapist insisted on stopping because he showed signs of

fatigue for being concentrated for so long. This interaction was only witnessed by the researcher,

with no video recording, so it was not possible to draw further conclusions.

During most of the sessions, H.R. seemed fearful of the robot and its movements, despite having

touched him, pressed its off button for several times, and even took pictures hugging NAO. The

therapist said that was amazing that after 4 sessions (Table 6.9) the participant was able to solve

the puzzles on the hard level, and that his concentration level has always been great.

Baseline First Second Third Fourth FinalPlayed in His Turn 100% 100% 100% 100% 100% 100%Played Out of Turn 1 2 0 0 0 0Helped the Robot - 100% 100% 100% 100% -

Table 6.4: Participant H.R.: Percentage of times he played in his turn, times he helped the robotand the number of times he played out of turn.

6.2.4 Participant 5 - A.S.

The participant 5 has twelve years old and presents mild autism. He attends supporting functional

behaviorist classes at the Francisco de Arruda school. He presents a highly cognitive and linguist

development, and a normal motor development for his age. Also, he does not require great effort

to take turns, to concentrate on a certain task, and to focus on communication. A.S. shows a


Figure 6.44: Participant A.S.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) of each child turn perpuzzle.

huge interest in tablets, tablet games, and robots, but a weak enthusiasm for puzzles and the

Tangram game. He is used to playing on tablets, however, he does not have much experience

with Tangram puzzle game.

Baseline

A.S. held 5 sessions, two with the teacher and three with the robot (Table 6.9). The Baseline and

First as Third and Final were held together. In the Baseline, he played 4 games with the teacher

in turn-taking mode, which took about 15 minutes. When the game started he was visibly excited

but a little apprehensive about failing. But when he realized that he was able to play it, he became

more relaxed. A.S. did not have any difficulty to play the game or to realize what was his turn to

play. There were even occasions when the teacher was having trouble playing and he wanted

to help her. As shown in the graph 6.44, he took a long time to finish the puzzle on the most

difficult level. At the end of these 4 games he met NAO, he was very enthusiastic, curious and

eager to interact with it. He asked many questions about the robot and its functioning, to which

the investigator answered.


Figure 6.45: Participant A.S.: Average num-ber per puzzle of: wrong or close placement ofpieces, and drags and rotates of pieces.

Figure 6.46: Participant A.S.: Average numberof external interventions (from the teacher) andhelps (from the robot), per puzzle.

Figure 6.47: Participant A.S.: Percentages ofeye gaze per session.

Figure 6.48: Participant A.S.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.



In the first session with the robot, he played with the robot another 4 puzzles. He played with ease,

always respecting his and robot’s turn (Table 6.5). According to the teacher, during the game A.S.

interacted with the robot almost as if it was a human, established occasional dialogue (Figure

6.48) and even smiled multiple times. Sometimes mimicked some of the robot vocalizations.

These games were too simple for him, as we can see on graphs 6.44, 6.46 and 6.45.

In the second session, he was a bit more passive and absent. Although, he continued speaking

with the robot during the interaction (Figure 6.48). As shown on graph 6.44, the difficulty of the 4

games was gradually increasing, and he did not present struggle, except for the hard level game

(the last one). For this reason, his performance has worsened during this session (Figure 6.45).

In this case, the robot provided some aids that were effective (Figure 6.46). In the third session

he was very focused on the game, but with no interest in the robot, as we can see in the graphic

6.47 he barely looked at the NAO. There were no vocalizations or gestures towards it (Figure

6.48). Although the apparent lack of interest in this session, after the 4 games he wanted to play

one more. The levels of the games were gradually increasing, all of the 5 games were on the hard

level, the first 3 were easy puzzles and the last 2 were the most complicated. Even though, A.S.

was always able to solve them more easily than in previous sessions (Figure 6.44), sometimes

with the help of the robot. In both sessions, he had no trouble realizing when it was his turn to

play.

Final Session

In the same session as the final with NAO he played 4 more games with the teacher in turn-

taking mode. As always he did not have any difficulties solving the puzzles or taking turns. He

even offered help to the teacher when she did not know how to play. His performance has also

improved dramatically (Figure 6.45).

In short, the teacher said that was waiting for him to not adhere so enthusiastically as he did

not appreciate the original Tangram game. Despite having lost some excitement in the third

session, he showed satisfaction at the game and never refused to continue playing throughout

all interactions. He also shared that he had particularly enjoyed the harder levels as they were

challenging. He was always concentrated on the game and on NAO’s instructions. According

to the teacher, the absence of rejection or disapproval to continue the sessions was surprising

because it was not expected.


Baseline First Second Third FinalPlayed in His Turn 94% 100% 100% 100% 100%Played Out of Turn 0 0 0 0 0Helped the Robot - 100% 100% 100% -

Table 6.5: Participant A.S.: Percentage of times he played in his turn, times he helped the robotand the number of times he played out of turn.

6.2.5 Participant 6 - A.J.

This child has 10 years old, has mild autism and frequents psychology sessions at CADIn Cas-

cais. His cognitive and linguistic capabilities are normal for his age. Despite not having much

difficulty to focus on a task, it is hard for him to focus on communication and he does not master

the turn-taking. A.J. has some experience and interest on puzzles, tablets and tablet games. His

mood at the sessions varies according to his temper and the proposed activities.

Baseline

This participant, in the first session, played four games with the therapist. He had some difficulties

on taking turns in the first game (Table 6.6) but then easily realized who should play and when. He

was able to easily solve the most difficult puzzles. As we can see in graph 6.53, he spoke a few

times during the interaction. When the robot was presented he was very curious and surprised.


This participant began the interaction with the robot with games of medium difficulty which were

getting harder according to his performance. The last game proved to be too difficult for him

(Figure 6.49) because the puzzle was the hardest. So it was necessary external interventions

as the robot was not able to help at all times. Throughout the game, he always followed NAO’s

instructions and responded to its questions. He was smiling and having fun with the robot’s

actions.

In the second session, A.J. was in a good mood and happy to see NAO one more time. He played

the 4 games on the hard level, but with easy puzzles. He easily solved the puzzles without much

help from NAO or the therapist (Figure 6.51). He was always enthusiastic and focused during the

interaction, and sometimes even answered NAO questions. He was so engrossed in the game

that when he reached the end of the four games he was surprised because he thought he had

not already played so many. A.J. was interested in continuing the sessions so he was very sad

when he heard that next week would be his last session.


Figure 6.49: Participant A.J.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) of each child turn perpuzzle.

Figure 6.50: Participant A.J.: Average num-ber per puzzle of: wrong or close placement ofpieces, and drags and rotates of pieces.

Figure 6.51: Participant A.J.: Average numberof external interventions (from the therapist) andhelps (from the robot), per puzzle.


Figure 6.52: Participant A.J.: Percentages of eyegaze per session.

Figure 6.53: Participant A.J.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.

In the third session, this participant played again with the robot. He was excited when he arrived

the room. In this session, he played more complicated puzzles, so the total time increased

(Figure 6.49). He was more quiet and focused, and his mood fluctuated throughout the games.

He seemed to have lost some interest halfway through the game. Although, he always respected

NAO’s turn, and helped when it asked (Table 6.6). Sometimes when NAO was giving him help,

the participant placed the piece, so the robot had to stop that speech. However, in these cases

occurred some delay which was perceived by the child who began to laugh at NAO’s confusion.

He was sad when the robot said he had to go.

Final Session

After that last session with NAO, he played 4 more games with the therapist: two easy puzzles

and two hard ones. The participant started to play in the first two games and the therapist started

the last two. This negotiation was a bit complicated because this child wanted to be always

the first to begin, but then they finally agreed. He did not struggle to know whose turn was.

Sometimes he tried to help the therapist when she could not place the piece.

According to the therapist, it was expected that this participant had a good reception to NAO

and did not have a lot of difficulties due to his good cognitive development. And that’s what was

verified. She referred that was surprising the child smiling to hear NAO.

6.2.6 Participant 7 - M.C.

M.C. has 10 years old and attends special education sessions on CADIn Cascais. The therapist

did not know his autism level, however, his cognitive and linguistic development are below normal



Table 6.6: Participant A.J.: Percentage of times he played in his turn, times he helped the robotand the number of times he played out of turn.

Figure 6.54: Participant M.C.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) of each child turn perpuzzle.

for his age. This child is not capable of focusing on a certain task for a long time. Also, he

has some difficulties taking turns and focusing on communication. He is very interested and

familiarized in tablets, tablet games, and puzzles.

Baseline

In the first session, he played four games with the therapist. He had some difficulties understand-

ing when he should or should not play, but only in the first games (Table 6.7). Sometimes M.C.

said now it’s my turn to play. He smiled and became very excited when finished each puzzle.

When he met the robot he was very surprised and curious.


Figure 6.55: Participant M.C.: Average num-ber per puzzle of: wrong or close placement ofpieces, and drags and rotates of pieces.

Figure 6.56: Participant M.C.: Average numberof external interventions (from the therapist) andhelps (from the robot), per puzzle.

Figure 6.57: Participant M.C.: Percentages ofeye gaze per session.

Figure 6.58: Participant M.C.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.



In the first game with NAO, he had some difficulty in understanding how it worked. So in the first

game, he wanted to return the tablet to the robot so it could play in its turn. He had not realized

that the robot played at a distance. Also, sometimes when the robot asked for help, he realized

what he had to do, but he did not know how. So he asked for help to the therapist. Throughout

the games, he talked a lot with the robot always smiling, and usually replied when it asked him

questions (Figure 6.58).

In the second session, he seemed more intimidated by the robot than in the first time, especially

by the sounds of the engines when it moved. Throughout the game, he was more distracted

and gazed less to the robot (Figure 6.57). He also asked more often the therapist to help him,

sometimes just asking if he could already play, after the robot has said that. It is possible to see

in the graph 6.57 because of that, he looked more to the therapist. However, he continued to

smile and respond to the robot. There were times he was so focused on the robot that he did not

realize that it was his turn to play (Table 6.7).

In the third session, he had some fear to enter the room and to approach the robot. Although the

therapist has said that M.C had confided that he wanted to take it home. He was widely dispersed

throughout the session (may be seen by Gaze Elsewhere values in graph 6.57), especially in the

first game. Therefore, it was necessary more external interventions. In this session he continued

not to realize that in his turn he had to drag the piece, but when he helped the robot he only

had to touch the place where the piece fits. Although this was improved over the games. As in

the previous session, he expressed fear when he heard the NAO’s engines. Also, he spoke less

often and gazed less to the robot. He played sometimes in NAO’s turn but helped whenever it

requested (Table 6.7).

Final Session

In the four games with the therapist, he had better results in terms of performance despite its

difficulty level being similar to the last session with the robot. However, he tried to play more

often out of his turn because he was very distracted.


Table 6.7: Participant M.C.: Percentage of times he played in his turn, times he helped the robotand the number of times he played out of turn.


Figure 6.59: Participant D.B.: Total time (in seconds) to complete each puzzle, the average totaltime (in seconds) spent per puzzle, and the average time (in seconds) of each child turn perpuzzle.

6.2.7 Participant 8 - D.B.

The last participant has five years old and attends early intervention in CADIn Setubal. Because

of his ADOS evaluation was outdated, the therapist did not know the current autism level of

this child. So the skills table (Appendix B) was answered in accordance with the opinion of

the therapist rather than being based on a formal assessment. Cognitive development of D.B.

is slightly below the normal level for his age. His linguistic and motor development are also

below. He has difficulty in concentrating, taking turns and focusing on communication. He is

really interested in robots but had never interacted with any. He also has interest and average

experience on tablets, tablet games, and puzzles. During therapy sessions, this participant is

usually in a good mood. Sometimes he presents grouchy due to sleepiness caused by the

medication.


Figure 6.60: Participant D.B.: Average num-ber per puzzle of: wrong or close placement ofpieces, and drags and rotates of pieces.

Figure 6.61: Participant D.B.: Average numberof external interventions (from the therapist) andhelps (from the robot), per puzzle.

Figure 6.62: Participant D.B.: Percentages ofeye gaze per session.

Figure 6.63: Participant D.B.: Average num-ber per puzzle of gestures and vocalizationstowards the robot or about the game.


Baseline

Although it was probably the first time he played the Tangram, his reaction and performance in the

game was positive. He did not present many difficulties realizing when should play. However for

several times he got distracted, so he tried to play again after having already placed a piece. He

was not concentrated on the turn-taking flow despite being focused on the game. This session

lasted seven minutes (Table 6.9). Later he met the robot, D.B. was very excited and curious. He

asked several questions about the robot (if it had a mouth, and if it could talk and walk), both to

the therapist and the investigator.


On this same day occurred the first interaction with the robot, in which he played 4 games with

him. In the first two puzzles, he was very concentrated doing what the robot told him to do and

following all of its instructions. However, with the advance of the game he was losing focus on

the robot and concentrated only in the game, so he started to trying to play more out of turn and

sometimes he did not help NAO (requiring therapist intervention). According to the therapist, this

was due to the fact that he possibly has not realized how the robot was playing (i.e., NAO did not

need to touch the tablet to play). He spoke a few times with the robot, as we can see in graph

6.62. At the end, the participant asked to give a handshake to the robot.

In the second session with the robot, he was very sleepy and dispersed. Therefore, the response

time to robot’s instructions and the number of attempts out of turn increased significantly (Table

6.8). Sometimes he was so focused on the game that did not hear robot’s speech, other times

was so focused on the robot that he did not see NAO had already played. Although it has

been explained, D.B. still did not realize that the robot played in the distance, so this helped his

confusion on understanding when he could play or not. Throughout the game, he smiled at the

robot but gazed and talked less to it (Figures 6.62 and 6.63), and at the end asked to click the

power button.

In the third session, the participant seemed to be more awake than in the previous. This con-

tributed to a greater focus on the game comparing to the second session. And also to a better

performance, as we can analyze in the graph 6.59, although the average time had increased,

the play time decreased slightly. However, throughout the 4 games, he was slowly losing interest

and getting sleepy. By the middle of the game, D.B. asked questions about the robot and its

functioning, while not listening to NAO instructions. When mediated externally by the therapist,

he was able to focus on what the robot said to him, but in the absence of this mediation he lost

this ability. The number of plays out of turn decreased (Table 6.8).


Final Session

At the end the session with the robot, he seemed less enthusiastic than in the beginning. But

when the final session with the therapist started, he was more focused and interested again in

the game. This was due to the therapist being able to provide more frequent and enthusiastic

feedback. It was easier for the participant to realize when it was his turn to play, this also helped

his performance on this session.


Table 6.8: Participant D.B.: Percentage of times he played in his turn, times he helped the robotand the number of times he played out of turn.

6.2.8 Discussion of the Results

In this section, we summarize all the results. Considering the turn-taking results, for most partic-

ipants (2, 4, 5, 6, and 7) the robot was able to stipulate the turns to play. Clearly the other two

participants who did not have such positive results (3 and 8) are also the youngest participants,

and so had more difficulty on the turn-taking and following instructions. While the Participant

3 (Section 6.2.2) played out of turn, because he wanted to play alone and not respect orders,

the Participant 8 (Section 6.2.7) did it because he got easily distracted and did not listen to the

robot’s instructions. This participant also had trouble understanding that NAO played at a dis-

tance, which contributed to the confusion of turns. Almost all participants processed robot’s help

requests and promptly helped it, few exceptions due to lack of attention. But there were some

children (7 and 8) who confused the two ways of playing - in their turn they had to drag the piece

to the right place, and when helping the robot, only had to touch the spot because it drags them.

Whenever it asked for help, NAO explained this, however, these children sometimes tried to drag

when only had to touch and vice versa. This indicates that they were not aware of the request, or

as soon as they heard the robot ask the question, failed to hear the rest of the sentence.

All participants improved their performance. Over time, they played an increasingly difficult level

games and were proving to be able to solve almost independently in some cases. As expected,

with increasing difficulty, the number of incorrect attempts also increased for most of the par-

ticipants, which contributed to the increment of aid provided by the robot. Most of this aid was

successful, however, one of the therapists said they were few and took too long to emerge. In this

version the main focus of the robot was the turn-taking, so there was a smaller number of help


(as opposed to Tutor version). Furthermore, it is difficult to provide updated feedback because

there is always some delay in the tools associated to the NAO controlling. Whereby there was

one of the participants (Section 6.2.5) that noticed this delay and started laughing. For these

reasons, there was a greater number of external interventions than it should, sometimes the

therapists tried to help children during these difficulties. The therapist of participant 4 (Section

6.2.3) confessed that was surprising this child completed the hard level games since she thought

he would not have cognitive abilities for this. Therefore, this participant at the end of the sessions

played the original Tangram and was able to solve three puzzles almost with no help and always

enthusiastic.

There was a huge interest in the robot by almost all participants (except J.N. - Section 6.2.2) at the

first session. However, this interest noticeably decreased over the sessions due to habituation to

the robot, as can be seen in Gaze graphics. They looked more at the game than at NAO, and this

can also be related to the perception that it is not necessary to look to the robot to understand

its instructions. The only exception was the Participant 4 (Section 6.2.3) that despite having

more sessions with NAO than the other participants, he did not lose interest in it. There was

also the Participant 3 (Section 6.2.2) who did not react positively to the robot, but throughout the

sessions improved his behavior, even smiling and talking to it. Also, it was surprising to see that

all children respond to questions asked by NAO (e.g.: Do you want to play another game?, Did

you like this game?, among others) although this has not been a goal of ours. Some participants

(2, 4, and 5) spontaneously imitated NAO’s lines when they were playing with the therapist in the

final session. Participants 2 and 4 reproduced requests for help conducted by the robot. The first

participant only mimicked the action without really needing help, but the participant 4 used when

he really had trouble. The request for help is a rare action in these children, therefore, this was

so surprising. The number of vocalizations also decreased for some participants which can also

relate to the fact that they do not produce any effect on the robot.

It was really a challenge to transform a monotonous and uninteresting game, into something

appealing that could engage all children with ASD. We think this has been achieved, because al-

though none of the participants particularly like the Tangram, everyone was excited and engaged

while playing. However throughout the sessions, there was a growing disinterest by some of the

participants (3, 5, 6, and 8), this was due in part to the fact that the game always had the same

flow and it became monotonous. As the robot feedback is not as frequent as from a human, the

participants 3 and 8, got distracted and easily lose interest in the game. This is one of the major

differences between human and robot feedback - the human can be more frequent, specific for

each child and excited depending on the occasion. Participant 8 (Section 6.2.7) experienced the

robot as a puppet instead of a game partner, it also caused him to lose the enthusiasm for the


game, making him more interested in the robot instead.

Regarding the proposed hypothesis - it is expected that the performance of the children (concen-

tration in the game and task execution) in the Tangram game with a humanoid robot is similar

than with a therapist, was verified for some children as it was analyzed earlier. The hypothesis

of this condition - the child respects his / her turn to play the much the with the therapist, as has

been said previously, it was verified for some participants (2, 4, 5, 6, and 7) but not for participants

3 and 8. Although the game has worked for most of the children, it does not appear to be suitable

for all.

Participant 1st Session 2nd Session 3rd Session 4th Session

1 Base 1st 2nd 3rd 4th Final29m 3m 14m 15m 14m 12m

2 Base 1st 2nd Final - -8m 10m 10m 7m

3 Base 1st 2nd 3rd7m 23m 9m 13m

4 Base 1st 2nd 3rd 4th Final6m 11m 13m 14m 9m 9m

5 Base 1st 2nd 3rd Final - -15m 11m 11m 16m 20m

6 Base 1st 2nd 3rd Final - -5m 15 10m 11m 11m



Table 6.9: Time spent (in minutes) in each session for every participant

Chapter 7

Conclusions

This chapter summarizes the work described in the present dissertation, as well as pointing

some directions for future research based on the current achievements and results. The

purpose of this M.Sc. thesis was to analyze how engaging a social robot can be to children with

ASD during a therapy session.

To provide a better comprehension of this research, we initiated this report by explaining the

Autism Spectrum Disorder. It was explained that the main characteristics of children with ASD

are the impairment in social communication and social interaction, and repetitive activities. In

addition, were presented some of the treatment approaches most commonly used in therapy

(e.g., ABA, Floortime, TEACCH).

Then, some of the more important works related to this research field were introduced. For

example in The Aurora Project, it was shown that a humanoid robot can encourage children

developing their communication and social behaviors, such as imitation and turn-taking skills

(Dautenhahn et al., 2009; Robins et al., 2009). And the work of Greczek et al. (2014) with

Nao, demonstrated that through the graded cueing feedback, the children could improve their

performance over time. Considering screen-based games, it is important to highlight the work

Battocchi et al. (2009) that presents how a puzzle game can induce collaborative actions between

children with ASD.

Considering numerous studies that show how much children with autism enjoy robots and com-

puters, we planned to exploit this fact to succeed in this study. Therefore, we developed a tablet

game normally played in autistic children therapy sessions. And programmed a social robot to

play with the children as a tutor and as a peer. As a Tutor, the robot should give graded cueing

feedback to help the child throughout the game. And as a Peer, the robot plays the Tangram by

87

88 CHAPTER 7. CONCLUSIONS

taking turns with the child. In this condition, the agent can help the child and ask for helping him

too. In both conditions, the robot provides feedback whenever is necessary, in order to reinforce

child’s engagement in the game.

Two experimental studies were conducted to test the viability and the value of introducing a

humanoid robot as a peer and as a tutor in therapy sessions of children with ASD. Eight children

participated in this study, one in the Tutor Condition and the other seven in the Peer Condition.

The participant in the first condition showed a great interest in the robot and the game. His auton-

omy in the game increased throughout the sessions, as the game difficulty level also increased.

At first, he seemed to ignore robot’s instructions, but during the following sessions, he was more

focused and responded more to NAO’s requests. Given the characteristics of this child, these

results were positive. The results of the game with the original Tangram in the final session were

also unexpected. Compared to baseline, he completed the puzzle more easily and indepen-

dently. It was not expected to obtain results as significant as this, with this participant in a short

period of time.

Overall, the participants in the Peer Condition showed little difficulty in taking turns with the robot.

Except for the two youngest participants who naturally had trouble on the turn-taking (they were

not able to acknowledge NAO as a gaming partner). Although all the participants promptly re-

sponded to its requests for help. Their performance was, in general, improving throughout the

sessions. Our goal in this condition was to study if the child is as or more autonomous with the

robot as with the therapist. Given the decreased number of external interventions and help, this

was verified for several participants.

In all children, a drastic decrement in the interest for the robot was analyzed in the following

sessions. This can be translated by their familiarization with the robot. But this decrease in the

amount of gaze towards the robot can also correspond the perception that it is not necessary to

look towards the robot to understand its instructions. The number of vocalizations also decreased

throughout the study, which can also relate to the fact that these do not produce any effect on

the robot. In this condition, we can say that we have achieved our goals with only some of the

participants.

Given the heterogeneity of the autistic spectrum, it was not expected that a single methodology

would be adequate to all subjects. It is important to emphasize that when using a robot during

ASD therapy is necessary to take into account each child’s needs and to develop robot-child

interactions managed to each individual. As we conclude, our approach was not able to succeed

in all children of the spectrum, as was expected. However, we managed that a monotonous

game became attractive to all children. Again, our aim was never to replace the therapists during

7.1. FUTURE WORK 89

therapy sessions, and as we observed in our study, they are fundamental in order to mediate this

child-robot interaction. Without their presence, possibly the children would not have successfully

interacted with the robot.

7.1 Future Work

With the end of our study, we found a few aspects that can be approached in subsequent work,

as from the game perspective and also in the study itself. Starting with the study, a long-term

experiment should be done with a larger number of participants from both the Tutor and Peer

conditions. These exposures repeated over multiple sessions, would allow us to analyze the

generalizability and repeatability of our observations. The baseline and the final sessions should

also be conducted for several days until results are stable. Children who participated in the

Tutor Mode could, according to its performance, then play in the Peer Mode, so as to be able to

compare the two conditions.

Regarding the game itself, one of the therapists suggested that instead of being the robot to

decide who first gets to play, there should be a negotiation between the child and the robot in

Peer mode. Moreover, other aspects and features could also be added to the game so the

interest in it does not diminish (e.g., more challenges throughout the game).

It should also be developed a web interface targeted to the therapist so that he/she could analyze

the progress of each child, their greatest strengths, and weaknesses. Furthermore, this interface

should also provide a set of services to connect with the game, such as to edit the difficulty for

each user, change the puzzle and to enable or disable certain animations (for those cases where

it is beneficial for some children but not so beneficial for others).

Bibliography

AKSHOOMOFF, N., CORSELLO, C. & SCHMIDT, H. (2006). The role of the autism diagnostic ob-

servation schedule in the assessment of autism spectrum disorders in school and community

settings. The California School Psychologist , 11, 7–19.

ALDEBARAN ROBOTICS (2016). NAO Documentation. http://doc.aldebaran.com/2-1/

index.html, [Last accessed March, 2016].

AMERICAN PSYCHIATRIC ASSOCIATION (2013). Diagnostic and Statistical Manual of Mental Dis-

orders, 5th Ed.. American Psychiatric Association, Arlington, VA.

BAIO, J. (2014). Prevalence of autism spectrum disorder among children aged 8 years-autism

and developmental disabilities monitoring network, 11 sites, united states, 2010. CDC. Morbid-

ity and mortality weekly report. Surveillance Summaries, 63, 1.

BARON-COHEN, S. (1988). Social and pragmatic deficits in autism: cognitive or affective? Journal

of autism and developmental disorders, 18, 379–402.

BARON-COHEN, S., LESLIE, A.M. & FRITH, U. (1985). Does the autistic child have a “theory of

mind”? Cognition, 21, 37–46.

BATTOCCHI, A., PIANESI, F., TOMASINI, D., ZANCANARO, M., ESPOSITO, G., VENUTI, P.,

BEN SASSON, A., GAL, E. & WEISS, P.L. (2009). Collaborative Puzzle Game: a tabletop

interactive game for fostering collaboration in children with autism spectrum disorders (ASD).

In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces.

BEGUM, M., SERNA, R.W., KONKTAK, D., ALLSPAW, J., KUCZYNSKI, J., YANCO, H.A. &

SUAREZ, J. (2015). Measuring the efficacy of robots in autism therapy: How informative are

standard HRI metrics? In Proceedings of the ACM/IEEE International Conference on Human-

Robot Interaction.

91

http://doc.aldebaran.com/2-1/index.html

http://doc.aldebaran.com/2-1/index.html

92 BIBLIOGRAPHY

BERNARDINI, S., PORAYSKA-POMSTA, K. & SMITH, T.J. (2014). ECHOES: An intelligent serious

game for fostering social communication in children with autism. Information Sciences, 264,

41–60.

BILLARD, A., DAUTENHAHN, K. & HAYES, G. (1998). Experiments on human-robot communi-

cation with Robota, an imitative learning and communicating doll robot. In Proceedings of the

Socially Situated Intelligence Workshop, Citeseer.

BILLARD, A., ROBINS, B., NADEL, J. & DAUTENHAHN, K. (2007). Building Robota, a mini-

humanoid robot for the rehabilitation of children with autism. Assistive Technology , 19.

CLEMENTS, D.H. (2000). ‘concrete’ manipulatives, concrete ideas. Contemporary Issues in Early

Childhood , 1, 45–60.

COOK, A.M., ADAMS, K., VOLDEN, J., HARBOTTLE, N. & HARBOTTLE, C. (2011). Using lego

robots to estimate cognitive ability in children who have severe physical disabilities. Disability

and Rehabilitation: Assistive Technology , 6, 338–346.

COSTELLO, M.J. (1988). The greatest puzzles of all time. Courier Corporation.

DAUTENHAHN, K. & BILLARD, A. (2002). Games children with autism can play with Robota, a

humanoid robotic doll. In Universal Access and Assistive Technology , 179–190, Springer.

DAUTENHAHN, K., NEHANIV, C.L., WALTERS, M.L., ROBINS, B., KOSE-BAGCI, H., MIRZA, N.A.

& BLOW, M. (2009). KASPAR – a minimally expressive humanoid robot for human–robot inter-

action research. Applied Bionics and Biomechanics, 6, 369–397.

DAVIS, M., OTERO, N., DAUTENHAHN, K., NEHANIV, C.L. & POWELL, S.D. (2007). Creating

a software to promote understanding about narrative in children with autism: Reflecting on

the design of feedback and opportunities to reason. In Proceedings of the IEEE International

Conference on Development and Learning.

DIAS, J. & PAIVA, A. (2005). Feeling and reasoning: A computational model for emotional char-

acters. In Progress in Artificial Intelligence, 127–140, Springer.

DIETZ, P. & LEIGH, D. (2001). Diamondtouch: a multi-user touch technology. In Proceedings of

the ACM Symposium on User Interface Software and Technology .

DUQUETTE, A., MICHAUD, F. & MERCIER, H. (2008). Exploring the use of a mobile robot as an

imitation agent with children with low-functioning autism. Autonomous Robots, 24.

FEIL-SEIFER, D. & MATARIC, M.J. (2005). Defining socially assistive robotics. In Proceedings of

the International Conference on Rehabilitation Robotics, IEEE.

BIBLIOGRAPHY 93

FEIL-SEIFER, D. & MATARIC, M.J. (2008). Robot-assisted therapy for children with autism spec-

trum disorders. In Proceedings of the ACM Interaction Design and Children.

FEIL-SEIFER, D. & MATARIC, M.J. (2009). Toward socially assistive robotics for augmenting in-

terventions for children with autism spectrum disorders. In Proceedings of the International

Symposium on Experimental Robotics, 201–210, Springer.

FERRARI, E., ROBINS, B. & DAUTENHAHN, K. (2009). Therapeutic and educational objectives in

robot assisted play for children with autism. In Proceedings of the IEEE International Sympo-

sium on Robot and Human Interactive Communication.

GAST, D.L. & LEDFORD, J.R. (2009). Single subject research methodology in behavioral sci-

ences. Routledge.

GIULLIAN, N., RICKS, D., ATHERTON, A., COLTON, M., GOODRICH, M. & BRINTON, B. (2010).

Detailed requirements for robots in autism therapy. In Proceedings of the International Confer-

ence on Systems Man and Cybernetics (SMC), 2595–2602, IEEE.

GOCKLEY, R., BRUCE, A., FORLIZZI, J., MICHALOWSKI, M., MUNDELL, A., ROSENTHAL, S.,

SELLNER, B., SIMMONS, R., SNIPES, K., SCHULTZ, A.C. et al. (2005). Designing robots for

long-term social interaction. In Proceedings of the IEEE International Conference on Intelligent

Robots and Systems, 1338–1343, IEEE.

GRECZEK, J., KASZUBSKI, E., ATRASH, A. & MATARIC, M.J. (2014). Graded cueing feedback in

robot-mediated imitation practice for children with autism spectrum disorders. In Proceedings

of the IEEE International Symposium on Robot and Human Interactive Communication, 561–

566.

HAGEDORN, J., HAILPERN, J. & KARAHALIOS, K.G. (2008). Vcode and vdata: illustrating a

new framework for supporting the video annotation workflow. In Proceedings of the working

conference on Advanced visual interfaces, 317–321, ACM.

HORNER, R.H., CARR, E.G., HALLE, J., MCGEE, G., ODOM, S. & WOLERY, M. (2005). The use

of single-subject research to identify evidence-based practice in special education. Exceptional

children, 71, 165–179.

KANNER, L. (1943). Autistic disturbances of affective contact. Nervous Child , 2, 217–250.

KHAIRIREE, K. (2015). Creative thinking in mathematics with tangrams and the geometer’s

sketchpad. In Proceedings of the Asian Technology Conference in Mathematics, ATCM.

94 BIBLIOGRAPHY

KIM, E.S., BERKOVITS, L.D., BERNIER, E.P., LEYZBERG, D., SHIC, F., PAUL, R. & SCAS-

SELLATI, B. (2013). Social robots as embedded reinforcers of social behavior in children with

autism. Journal of Autism and Developmental Disorders, 43.

KLIN, A. (2006). Autism and Asperger syndrome: an overview. Revista brasileira de psiquiatria,

28, s3–s11.

KOHANOVA, I. & OCHODNICANOVA, I. (2014). Developmente of geometric imagination in lower

secondary education. In Proceedings of the International Conference on Mathematical Confer-

ence in Nitra, 75–80, Department of Mathematics FNS CPU in Nitra.

KOZIMA, H., NAKAGAWA, C., KAWAI, N., KOSUGI, D. & YANO, Y. (2004). A humanoid in company

with children. In Proceedings of the IEEE/RAS International Conference on Humanoid Robots,

vol. 1.

KOZIMA, H., MICHALOWSKI, M.P. & NAKAGAWA, C. (2009). Keepon. International Journal of

Social Robotics, 1, 3–18.

LARSON, M.J., SOUTH, M., KRAUSKOPF, E., CLAWSON, A. & CROWLEY, M.J. (2011). Feedback

and reward processing in high-functioning autism. Psychiatry Research, 187, 198–203.

LEITE, I., MCCOY, M., LOHANI, M., ULLMAN, D., SALOMONS, N., STOKES, C., RIVERS, S.

& SCASSELLATI, B. (2015). Emotional Storytelling in the Classroom: Individual versus group

interaction between children and robots. In Proceedings of the International Conference on

Human-Robot Interaction.

LINDGREN, S. & DOOBAY, A. (2011). Evidence-based interventions for Autism Spectrum Disor-

ders. The University of Iowa.

LO, A.C., GUARINO, P.D., RICHARDS, L.G., HASELKORN, J.K., WITTENBERG, G.F., FEDER-

MAN, D.G., RINGER, R.J., WAGNER, T.H., KREBS, H.I., VOLPE, B.T. et al. (2010). Robot-

assisted therapy for long-term upper-limb impairment after stroke. New England Journal of

Medicine, 362, 1772–1783.

LORD, C., RUTTER, M., DILAVORE, P.C., RISI, S., GOTHAM, K. & BISHOP, S. (2012). Autism

diagnostic observation schedule: ADOS-2. Western Psychological Services Los Angeles, CA.

MESSIAS, J., VENTURA, R., LIMA, P., SEQUEIRA, J., ALVITO, P., MARQUES, C. & CARRICO, P.

(2014). A robotic platform for edutainment activities in a pediatric hospital. In Proceedings of

the International Conference on Autonomous Robot Systems and Competitions, IEEE.

MILES, J. & MCCATHREN, R. (2005). Autism overview. GeneReviews at GeneTests: Medical

Genetics Information Resource.

BIBLIOGRAPHY 95

MOSKOWITZ, A. & HEIM, G. (2011). Eugen bleuler’s dementia praecox or the group of

schizophrenias (1911): A centenary appreciation and reconsideration. Schizophrenia Bulletin,

37, 471–479.

MYERS, S.M. & JOHNSON, C.P. (2007). Management of children with autism spectrum disorders.

Pediatrics, 120, 1162–1182.

NADEL, J. (2004). Early imitation and the emergence of a sense of agency. In Proceedings of the

International Workshop on Epigenetic Robotics, Lund University Cognitive Studies.

OZONOFF, S., ROGERS, S.J. & HENDREN, R.L. (2008). Autism spectrum disorders: A research

review for practitioners. American Psychiatric Publishing.

PIPER, A.M., O’BRIEN, E., MORRIS, M.R. & WINOGRAD, T. (2006). SIDES: a cooperative table-

top computer game for social skills development. In Proceedings of the Conference on Com-

puter Supported Cooperative Work , ACM.

POT, E., MONCEAUX, J., GELIN, R. & MAISONNIER, B. (2009). Choregraphe: a graphical tool for

humanoid robot programming. In Proceedings of the International Symposium on Robot and

Human Interactive Communication, 46–51, IEEE.

PRIZANT, B.M., WETHERBY, A.M., RUBIN, E. & LAURENT, A.C. (2003). The SCERTS model

- a transactional, family-centered approach to enhancing communication and socioemotional

abilities of children with autism spectrum disorder. Infants and young children.

READ, R.C. (1965). Tangrams: 330 puzzles. Courier Corporation.

RIBEIRO, T., VALA, M. & PAIVA, A. (2012). Thalamus: Closing the mind-body loop in interactive

embodied characters. In Intelligent Virtual Agents, 189–195, Springer.

RIBEIRO, T., PEREIRA, A., DI TULLIO, E., ALVES-OLIVEIRA, P. & PAIVA, A. (2014). From tha-

lamus to skene: High-level behaviour planning and managing for mixed-reality characters. In

Proceedings of the IVA 2014 Workshop on Architectures and Standards for IVAs.

ROBINS, B., DAUTENHAHN, K., TE BOEKHORST, R. & BILLARD, A. (2004). Effects of repeated

exposure to a humanoid robot on children with autism. In Proceedings of the Universal Access

and Assistive Technology , 225–236, Springer.

ROBINS, B., DAUTENHAHN, K. & DICKERSON, P. (2009). From isolation to communication: A

case study evaluation of robot assisted play for children with autism with a minimally expressive

humanoid robot. In Proceedings of the International Conference on Advances in Computer-

Human Interactions, IEEE.

96 BIBLIOGRAPHY

ROGERS, L.A. & GRAHAM, S. (2008). A meta-analysis of single subject design writing interven-

tion research. Journal of Educational Psychology , 100, 879.

RUSSELL, D.S. & BOLOGNA, E.M. (1982). Teaching geometry with tangrams. The Arithmetic

Teacher , 30, 34–38.

SACKS, H., SCHEGLOFF, E.A. & JEFFERSON, G. (1974). A simplest systematics for the organi-

zation of turn-taking for conversation. language, 696–735.

SCASSELLATI, B. (2005). Quantitative metrics of social response for autism diagnosis. In Pro-

ceedings of the International Workshop on Robot and Human Interactive Communication, 585–

590, IEEE.

SHAMSUDDIN, S., ISMAIL, L.I., YUSSOF, H., ZAHARI, N.I., BAHARI, S., HASHIM, H. & JAFFAR,

A. (2011). Humanoid robot nao: Review of control and motion exploration. In Proceedings

of the International Conference on Control System, Computing and Engineering (ICCSCE),

511–516, IEEE.

SIEW, N.M. & CHONG, C.L. (2014). Fostering students’ creativity through van hiele’s 5 phase-

based tangram activities. Journal of Education and Learning, 3, 66.

STANTON, C.M., KAHN, P.H., SEVERSON, R.L., RUCKERT, J.H. & GILL, B.T. (2008). Robotic

animals might aid in the social development of children with autism. In Proceedings of the

International Conference on Human-Robot Interaction (HRI), IEEE.

THE NATIONAL AUTISTIC SOCIETY (2015). Autism Speaks. http://www.autismspeaks.

org/, [Last accessed May, 2015].

TRAFTON, J.G., SCHULTZ, A.C., PERZNOWSKI, D., BUGAJSKA, M.D., ADAMS, W., CASSIMATIS,

N.L. & BROCK, D.P. (2006). Children and robots learning to play hide and seek. In Proceedings

of the 1st ACM conference on Human-robot interaction, 242–249, ACM.

WERRY, I. & DAUTENHAHN, K. (1999). Applying mobile robot technology to the rehabilitation of

autistic children. In Proceedings of the Intelligent Robotic Systems.

WING, L. (1997). The history of ideas on autism legends, myths and reality. Autism, 1, 13–23.

WONG, A.W., CHAN, C.C., LI-TSANG, C.W. & LAM, C.S. (2004). Neuropsychological function

for accessibility of computer program for people with mental retardation. Springer.

ZANCANARO, M., GIUSTI, L., GAL, E. & WEISS, P.T. (2011). Three Around a Table: the facilitator

role in a co-located interface for social competence training of children with autism spectrum

disorder. In Human-Computer Interaction, 123–140, Springer.

http://www.autismspeaks.org/

http://www.autismspeaks.org/

Appendix A

Parental Consent Form

97

98 APPENDIX A. PARENTAL CONSENT FORM

Figure A.64: Consent form given to the participants’ parents

Appendix B

Child’s Profile

99

100 APPENDIX B. CHILD’S PROFILE

Figure B.65: Form given to the therapists to fill for each participant

Appendix C

Questionnaires

101

102 APPENDIX C. QUESTIONNAIRES

Figure C.66: Questionnaire to be answered by the therapist regarding the outcome of the inter-action between child, therapist, and tablet

103

Figure C.67: Questionnaire to be answered by the therapist regarding the outcome of the inter-action between child, robot, and tablet

104 APPENDIX C. QUESTIONNAIRES

Figure C.68: Questionnaire to be answered by the therapist regarding child’s first impressions ofthe robot

an interactive tangram game for autistic children · his work explores the usage of a social robot...

Documents