It’s About Time – Analyzing Temporal Micro‐It s About Time  Analyzing Temporal MicroLevel Behavioral Patterns

Chen YuIndiana University

Temporal Bands of Human Computation (B ll d H h P k & R 1997)(Ballard, Hayhoe, Pook, & Rao, 1997) 

micro level

macro-level behavior

tern

alna

l

micro-level behavior ex

tin

tern

Dual Eye Tracking in Child-Parent Interaction (with Linda Smith, Damian Fricker, & Linger Xu) ( , , g )

child’s first person view parent’s first person view

from eye camera

from head camera

child eye tracker

from head camera

adult eye trackertracker

Dual Eye Tracking in Child-Parent InteractionInteraction

Multimodal data• Vision: 720*480, 30 frames/second, 3 cameras, 18,000

frames per dyads.

• Motion tracking: 250 Hz/second, 6 sensors with six dimensions (x,y,z,h,p,r) on each.900 000 data points per dyads900,000 data points per dyads.

Eye tracking: 30Hz

• Speech: 44.1Hz

• Eye tracking: 30Hz

• 12GB per participant

Dealing with data

• Synchronization of multiple data streams

• Data annotation (automatically or manually)

• Data management

• Data Mining and Knowledge Discovery

Multi‐Streaming Multimodal Data

speech

0 500 1000 1500 2000 2500 3000 350

5

1

5

2

5x 104

x 104

0 500 1000 1500 2000 2500 3000 30

2

4

6

8x 10

x 104

vision

…0 500 1000 1500 2000 2500 3000

…

motion

Data Mining• Our goal of analyzing gaze data is to find new patterns• Our goal of analyzing gaze data is to find new patterns

and gain new knowledge from such data.

• With micro level data even if we have some predictions• With micro-level data, even if we have some predictions from our experimental designs, we nonetheless lack precise predictions about the structure and patterns of

• But how can we discover new and meaningful patterns if d k h l k f

data at the micro-level.

we do not know what we are looking for?

• Discovering new knowledge requires the ability to detect unknown, surprising, novel, and unexpected patterns.

• A particular challenge is not just from the amount of data but p g jfrom how to extract, select and interpret meaningful patterns from a sea of complex data.

childchildgaze

parentparentgaze

Interactive Data Analysis

top‐down knowledge

data

knowledge

patterndata visualization

pattern extraction

• This solution relies on both computational techniques and human domain knowledge. Data visualization and pattern extraction techniques provide candidate patterns through a bottom-up waycandidate patterns through a bottom up way.

• Compared with “blind” data mining, what we suggest is that researchers with top-down theoretical knowledge need to be in the loop of data mining andtop down theoretical knowledge need to be in the loop of data mining and data analysis.

childchildgaze

parentparentgaze

face three objects

hild

red blue red green red blue

childgaze

parentparentgaze

face gaze and mutual gaze joint attention

Joint Attention StateJoint Attention State60%

40%

50%

20%

30%

0%

10%

t l hild f / t bj t t f / hild bj t bj t diff t bj t

‐10%

mutual gaze child‐face/parent‐object parent‐face/child‐object same object different objects

child followingchild followingface three objects

childgaze

parentgaze

parent followingchildgaze

parent following

g

parentgaze

What is joint attention made of?

hild t

when baby looks mom’s face, mom looks at baby’s face 54% of the time; when baby looks at an object mom 

looks at 32% of time

when MOM looks baby’s face, baby looks at mom’s face 13% of the 

time; when mom looks at an object baby looks at 14% of timechild parent

54%(face)32%(object)

13%(face)

baby looks at 14% of time

gaze gaze

13%(face)14%(object)

56%55%

handhand

17%

21%

probabilities of following an attended object/face

Sequential Patterns From Multimodal Data Stream

face-to-active

Multimodal Data Stream

1 1learner’s gaze 1

facejoint attention following

1 1gaze

teacher’s gaze

1

3 33 222

4 4

teacher’s hand action 4

face three objects

Segmentation and AlignmentSegmentation and Alignment

Discovering Statistically Reliable Sequential Patterns

(Fricker, Zhang, & Yu, 2011)

multiple instancesmultiple instancesnaming utterance

describing utterance

face look

sequential prototype

durations and timings within and across multiple events

More complex patterns from lti l d t tmultiple data streams

From Data To Patterns

From Patterns To Knowledge

……

Interactive Data Mining

top‐down knowledge

data visualization

pattern extractionvisualization extraction

• Human-in-the-Loop data analysis involves active and iterative examination and exploration of visually displayed patterns to select useful information and guide further knowledge discoveryselect useful information and guide further knowledge discovery.

• “Discovery takes place between the ears” – Ben Shneiderman

Visual Data Mining (Yu, Zhong, Smith, Park, & Huang, 2008, 2009)

Acknowledgement

Data collection and pre-processing : Damian Fricker, Amanda p p g ,Favata, Char Wozniak, Melissa Elston.

Data Processing: Tian Xu Damian Fricker Thomas Smith HenryData Processing: Tian Xu, Damian Fricker, Thomas Smith, Henry Shen.

This esea ch as s ppo ted b NSF BCS 0924248 and NIH T32This research was supported by NSF BCS 0924248 and NIH T32 HD 07475

Thanks!