Slide 1

Fairy Tale Directed Training Game Platform with Brainwaves, Motion Sensing Cognitive Technologies for Children with ADHD

November 20, 2015

Kyungeun Park

ICES 2015

The International Conference on Emotion and Sensibility 2015

November 18 (Wed) 21 (Sat), 2015, Jeju, Republic of Korea

Maryland, U.S.A

1

Contents

Research Objective

ADHD Facts

ADHD Treatment Strategy

Serious Game and its Effects

Narrative Therapy

Behavior Training Game Platform

Experiments and Analysis

Demo

Review and Future Research

2

2

Research Objective

Fairy Tale Directed Behavior Training Game Platform

Brain-Computer Interface (BCI) with NeuroSkys MindWave

Skeleton and face tracking with Microsofts Kinect V2

Narrative fairy tale contents with characters and contextual instructions

This Training Game Platform:

Keeps track of users brainwaves and associated behaviors responding to the given narrative contents or implicit contextual instructions

Works as a treatment framework for children with ADHD

to improve the childrens sustained attention

to control their hyperactive behavior

to promote reading comprehension capability

to give them an opportunity of self-examination

to get them treated emotionally

to encourage them to feel self-confidence

3

3

ADHD Facts

ADHD (Attention-Deficit/Hyperactivity Disorder)

5% to 15% children in the U.S. have been diagnosed

American Psychiatric Association (APA): Avg. 5%

Centers for Disease Control and Prevention (CDC) : Avg. 11%

Typical Symptoms include:

Deficits in behavioral inhibition, sustained attention, and resistance to distraction

Impaired behavioral inhibition and self-control needed as a critical foundation for the performance of any task

Weak social skills

Low level reading capabilities

4

4

ADHD by the Numbers

5

http://www.healthline.com/health/adhd/facts-statistics-infographic

5

Types of ADHD

ADD: Attention Deficit Disorder

HD: Hyperactivity Disorder

Mixed ADHD: Attention Deficit/Hyperactivity Disorder

6

Rief, S.F. 2005. How to reach and teach children with ADD/ADHD: practical techniques, strategies and interventions, 2nd ed. San Francisco: Jossey-bass.

6

Attention Deficit Disorder (ADD)

7

https://www.pinterest.com/thebluebunyip/adhd/

Hyperactivity Disorder (HD)

8

http://www.theatlantic.com/health/archive/2012/02/a-western-diet-high-in-sugars-and-fat-could-contribute-to-adhd/252853/

ADHD Affects Children:

9

The more severe, the earlier the diagnosis

8 years old: with mild ADHD

7 years old: with moderate ADHD

5 years old: with severe ADHD

http://www.healthline.com/health/adhd/facts-statistics-infographic

9

ADHD & Co-Existing Conditions

Children might have co-existing conditions

learning disabilities

conduct disorders and difficulties: antisocial behavior, fighting, and oppositional defiant disorder

anxiety disorder

depression

sleep disorders, etc.

Boys and girls display very different ADHD symptoms.

Boys: more likely the attention disorder

Girls: more internal and less obvious

10

10

ADHD Behavioral Treatment Strategy

Develop relatively long-term sustained attention to successfully complete the instruction (ADD)

Keep high-level attention to figure out complicate and boring story or discussion (ADD/HD)

Encourage children to stay motionless for a while and behave appropriately in a certain situation (HD)

11

11

Serious Games

12

Any form of interactive computer-based game software for one or multiple players to be used on any platform and that has been developed with the intention to be more than entertainment

Primary purpose beyond pure entertainment

To train or educate users

To achieve a desired progress

Serious Games: Mechanisms and Effects, by Ritterfield, Cody, & Vorderer, 2009

12

Categories of Serious Games

Educational Games

Social Change & Public Welfare Games

Occupational Games

Healthcare Games

Military Games

Marketing Games

13

13

Expectations via Serious Games

As a training environment,

Draw problem solving strategies

Resolve the problems

Experience the processes

14

Serious Games: Mechanisms and Effects, by Ute Ritterfield, Michael Cody, and Peter Vorderer, 2009

14

Narrative Therapy

Indirect Narrative Experiences

A fairy tale story with pain and frustration

Narrative Experiences

Catharsis, Empathy

Change in perspective, Pair effect,

Self-examination, Generalization

Relief, Problem-solving, Therapy

15

15

Types of Narratives

Dialogue

Personal Experience Narrative

Fairy tale or Fiction

16

16

Dialogue

Do you want to talk about ?

How about school?

17

, !

17

Personal Experience Narrative

Self-Experienced Story

Dreams

Psychoanalytic approach uses interpretation of dreams and presents solution to psychological problems.

18

http://www.seedboxpress.com/the-interpretation-of-dreams/

18

Fairy tale or Fiction

HanselandGretel,

Snow White, Cinderella,

Little Red Riding Hood,

Jack and Beanstalk, etc.

Through painful threats and failure, a childish character changes into a matured character, Hero.

19

http://news.nationalgeographic.com/news/2013/11/131129-little-red-riding-hood-folktale-tehrani-anthropology-science/

19

Structure of Narratives

Story (plain contents): Character, Action, Space, Item, etc.

Discourse (representation): Point of view, Perspective, Focalization, Tone, Style, Order, Duration, Frequency, Plot, etc.

20

Scholes, R., Phelan, J., and Kellogg, R., 2006, The Nature of Narrative, Oxford University Press.

20

Application of Narratives to the Behavioral Treatment

Reading narratives requires complete understanding of the components of a story and its development processing.

Children with ADHD can be effectively treated with well-known, structured narratives (fairy tales) to improve reading skill, control hyperactive behavior, and treat emotional problem.

21

21

Behavior Training Platform

Unity3D

Game Contents Manager

Contents

Kinect V2

MindWave

Logger

Replayer

Screen Capture

MPEG Processor

Middleware

Event Logger

Video Producer

Server

ZooKeeper

Kafka

ElasticSearch DB

Video File Server

Video File Storage

API

Web App Dashboard

User List

Experiment

Analysis

22

Interactive Data Collection

Monitoring Concentration with Brain-Computer Interface (BCI)

Monitoring Behavior with Motion Sensors

Interactive Narratives and Intervention Data

Users Responses

23

23

Brainwaves via MindWave

eSense meter of Brainwaves

Attention

Meditation

Saved to the list of Brainwaves

After finishing game, the data sent to Data Sever

24

Brainwaves

MindWave Interface

Collect and Save Brainwaves

ElasticSearch

Collect Event Logs

Send to

Data Server

24

Kinect Skeleton Structure

25

00_Hip_center10_Wrist_Right01_Spine11_Hand_Right02_Shoulder_Center12_Hip_Left03_Head13_Knee_Left04_Shoulder_Left14_Ankle_Left05_Elbow_Left15_Foot_Left06_Wrist_Left16_Hip_Right07_Hand_Left17_Knee_Right08_Shoulder_Right18_Ankle_Right09_Elbow_Right19_Foot_Right

Joints

Twenty joints

25

Motion and Face Tracking

Skeletal tracking with Kinect:

recognizing 22 different motions

Face tracking with Kinect:

recognizing 6 different motions

26

Skeleton Tracking

Face Tracking

26

Kinect: Motion Recognition

27

27

Interactive Intervention

28

Normal playing with the player sitting in place

Paused upon recognizing the players motion

Controls the game according to the recognized brainwaves, skeleton, and face tracking results.

The main idea of the game is to make the player stay motionless, sitting still in place for a while.

To prohibit a game character from proceeding when the player is not concentrating on the narratives (video clip to show how it works)

28

Interactive Intervention

29

29

Fairy Tale Contents

Dynamically controlled fairy tale contents affect cyberspace with real-world events

While playing the game, a child with his/her own identity influences the story and the discourse of the fairy tale narratives.

30

Fisher man and Genie

3 Characters

Fisher man: main character

Genie: threat then friend character

Child: outside helper

Design Strategy

6 Sequences

Real-time events collection and intervention

Reproduction of past games for post evaluation and self-examination

31

Reading Flow

32

Processes

33

Sequence 1Sequence 2Sequence 3Sequence 4Sequence 5Sequence 6Reading Comprehension Fisher man is preparing for fishing.Fisher man is heading for fishing place.Fisher man is fishing a bottle.Genie is threatening fisher man to kill.Fisher man is escaping out of the life-threatening situation using wisdom.Fisher man and reader (child) are saving Genie together.Contextual InstructionGreetingKeep high-level attention without hyper actionKeep high-level attention according to instructionsComprehend complex dialogue Comprehend complex dialogue according to instructionsConsider others situation

Sequence 1

34

OrderStoryInterfaceComprehension and Contextual Instruction InterventionEventCharacterSpace1.GreetingFisher manIndoorTextGreetingKinectMutual interaction2.Preparation for fishing outFisher manIndoorTextPreparing fishingKinectResponse according to the instruction

35

OrderStoryInterfaceComprehension and Contextual Instruction InterventionEventCharacterSpace1.Go to sea shoreFisher manPath to the fishing placeTextGo to the Fishing placeFisherman is falling down.KinectStay motionlessMindWaveSustained attention is needed2. QuizFisher manPath to the fishing placeTextReading comprehensionRepetitionKinectAnswering3.Go to sea shoreFisher manPath to the fishing placeTextFiguring out the reason why fisher man is continuously walking to the fishing place.Fisherman is falling down.KinectMaintain the sustained attention with the understanding of the boring situationMindWaveSustained attention is needed.

Sequence 2

Sequence 3

36

OrderStoryInterfaceComprehension and Contextual Instruction InterventionEventCharacterSpace1.FishingFisher manSea shoreTextCatching a donkeyRecognize this is not wat the fisher man wantedFisherman is falling down.KinectStay motionlessMindWaveSustained attention according to the instruction

Sequence 4

37

OrderStoryInterfaceComprehension and Contextual Instruction InterventionEventCharacterSpace1.Opening the gourd bottleFisher manSea shoreTextGenie is out of the gourd bottleGame temporarily stops and waits for the child follow the instructionKinectKeep motionless to perform the instructionMindWaveSustained attention according to the instruction2.Dialogue between Genie and the fisher manGenieFisher manSea shoreTextGenie threats the fisher man to killGame temporarily stops and waits for the child follow the instructionKinectKeep motionless and read the text

Sequence 5

38

OrderStoryInterfaceComprehension and Contextual Instruction InterventionEventCharacterSpace1.Dialogue between Genie and the fisher manGenieFisher manSea shoreTextThe fisher man is seeking a way to escape from the life threating crisis by Genie.Game temporarily stops and waits for the child follow the instructionKinectKeep motionless and read the text2.Closing the gourd bottleGenieFisher manSea shoreTextShutting up Genie in the bottle Game temporarily stops and waits for the child follow the instructionKinectKeep motionless to perform the instructionMindWaveSustained attention according to the instruction3.Dialogue between Genie and the fisher manGenieFisher manSea shoreTextGenie asks for savingGame temporarily stops and waits for the child follow the instructionKinectKeep motionless and read the text

Sequence 6

39

OrderStoryInterfaceComprehension and Contextual Instruction InterventionEventCharacterSpace1.Dialogue between the fisher man and the childChildFisher manSea shoreTextThe fisher man asks the child if he saves Genie.Game temporarily stops and waits for the child follow the instructionKinectKeep motionless and read the text2.Opening the gourd bottleGenieFisher manSea shoreTextSaving GenieGame temporarily stops and waits for the child follow the instructionKinectKeep motionless to perform the instructionMindWaveSustained attention according to the instruction3.Dialogue among Genie, the fisher man, and the childGenieFisher manSea shoreTextGenie thanks to the fisher man and becomes friends each other. Game temporarily stops and waits for the child follow the instructionKinectKeep motionless and read the text

Load Balancing for Reading and Behavior Requirements

40

contentSequence 1Sequence 2Sequence 3Sequence 4Sequence 5Sequence 6Text (words)1099718622920357BCI Attention Req. Duration (sec)010590303030

Text (words)

BCI Attention Req. Duration (sec)

text(/word)sequence1sequence2sequence3sequence4sequence5sequence61099718622920357interaction(/sec)sequence1sequence2sequence3sequence4sequence5sequence6010590303030

Expected Effects via Complicate Reading with Intervention Scheme

Reading Skill Improvement: By figuring out complicate dialogue in Sequence 4 and 5 with sustained high- level attention

Behavior Treatment: By training high-level attention, keeping motionless in each scene, and behaving accordingly in a certain situation

Emotional Treatment: By helping the child experience feeling of sympathy for Genie especially in Scene 6.

41

Treatment Effects via Self-Examination with Reproduction of the Game

Visualization of the Childs Experience with the Game for Post Evaluation and Self-Examination

The child recognizes how Cyberspace was affected by his/her behavior.

As an Object Watcher, self-examination may stimulate changes in his/her behavior

42

H/W Configuration

Windows 8.1, 64 bit

Ram: 8GB

USB 3.0

CPU: Intel Core i5-4590m

HDD: 500GB SATA3

GPU: GeForce 705

43

43

Experiments and Analysis

1) Motion Tracking Test

To see if brainwaves and motion tracking data are successfully collected while playing the narrative game

The result shows the motion tracking capabilities recognize 22 different motions successfully

2) Game Tracking Test

Skeletal and face tracking have been tested while playing the game.

At the same time, the testers brainwaves were measured and collected.

44

44

Game Tracking Results

45

Body CategoryMotionNo. Motions Tester 1No. MotionsTester 2No. Motions Tester 3No. Motions Tester 4No. Motions Tester 5UpperBodyUBLeft24130UBRight23030UBFront05211UBBack02100UBStand02011UBSit02011Left HandLHLeft32130LHRight21130LHFront01101LHBack01000LHUp23341LHDown00001Right HandRHLeft22430RHRight11130RHFront01110RHBack01000RHUp21331RHDown00001Left FootLFLeft11100LFRight11100LFFront11251LFBack01000LFUp01000Right FootRFLeft11000RFRight11000RFFront11251RFBack01010RFUp01101

45

Game Tracking Results (cont.)

46

Body CategoryMotionNo. Motions Tester 1No. Motions Tester 2No. MotionsTester 3No. Motions Tester 4No. Motions Tester 5HeadHDLeft12121HDRight12021HDUp02010HDDown12010HDLeftTilt02111HDRightTilt01011Brainwaves Avg. eSenseAttention75.875962.782951.426353.782929.0852Meditation67.255863.092161.252355.228234.1012Clear Time (sec)Total651126710161Scene12840265124Scene23759415037TotalNo. Motion2553284815

46

Web App Dashboard (Statistics Data)

User List

Experiment List

Players Interaction Log:

Motion Sensing Data Log: Body and face tracking

Brainwaves: Attention and meditation tracking

Web App Dashboard Demo

47

User List

48

Click admin User

48

Experiment List

49

Experiment List of admin User

Click this event

Kinect Motion Tracking and Interaction

50

50

MindWave Brainwaves Tracking

51

Attention Tracking

Meditation Tracking

51

Demo

Fisher man and Genie

52

The main contribution is the implementation of the behavior training platform which combines BCI and motion sensing technology with fairy tale narrative contents and treat children with ADHD in terms of the behavioral treatment and reading comprehension.

We are expecting this training platform to help the children improve their attention and control hyperactive behavior based on narrative therapy strategy.

In addition to the current narrative content, Fisher man and Genie, we hope to provide diverse contents with wide-range of fairy tales.

We would like to test this platform as the treatment framework for children with ADHD by collaborating with related fields and medical professionals.

Review and Future Research

53

Q&A

Thank you for your attention!

54

Kyungeun Park, D.Sc.

Dept. of Computer and Information Sciences

Towson University

Towson, Maryland, USA

kpark@towson.edu

kepark@gmail.com

Sequence 1agreerequestYesObject searchNocomplimentrequestNoYesLeaving more than 5 minutesFinishNoYesLogin

Sequence 1agreerequestYesObject searchNocomplimentrequestNoYesLeaving more than 5 minutesFinishNoYesLogin

Sequence 2No movement for 15 secondsRemember detailstestrequestYesNorequestNocomplimentYesLeaving more than 5 minutesFinishYesNoLogin

Sequence 2No movement for 15 secondsRemember detailstestrequestYesNorequestNocomplimentYesLeaving more than 5 minutesFinishYesNoLogin

Sequence 3No movement for 10 secondsHelp cyclerequestNoYesNo movement for 5 secondsYesConcentration more than 40%Object acquisitionemptyYesNoLeaving more than 5 minutesFinishNoYesLoginNo

Sequence 3No movement for 10 secondsHelp cyclerequestNoYesNo movement for 5 secondsYesConcentration more than 40%Object acquisitionemptyYesNoLeaving more than 5 minutesFinishNoYesLoginNo

Sequence 4Concentration more than 40%requestGenie appearsNoYesLeaving more than 5 minutesFinishYesNoLogin

Sequence 4Concentration more than 40%requestGenie appearsNoYesLeaving more than 5 minutesFinishYesNoLogin

Sequence 5Concentration more than 40%requestGenie disappearsNoYesLeaving more than 5 minutesFinishNoYesLogin

Sequence 5Concentration more than 40%requestGenie disappearsNoYesLeaving more than 5 minutesFinishNoYesLogin

Sequence 6Concentration more than 40%requestGenie appearsLeaving more than 5 minutesFinishNoYesLogin

Sequence 6Concentration more than 40%requestGenie appearsLeaving more than 5 minutesFinishNoYesLogin