feature analysis for affect recognition supporting task sequencing in adaptive intelligent tutoring...

Feature Analysis for Affect RecognitionSupporting Task Sequencing

in Adaptive Intelligent Tutoring Systems

Ruth Janning, Carlotta Schatten, Lars Schmidt-Thieme

Information Systems and Machine Learning Lab,University of Hildesheim, Germany

EC-TEL, 19 September 2014

Feature Analysis for Affect Recognition

Contents

1. Introduction

2. Features, Classes, Instances, Methods

3. Real Data Set

4. Feature Analysis

5. Supporting Task Sequencing

6. Conclusions

R. Janning et al. EC-TEL, 19 September 2014 1 / 18

Feature Analysis for Affect Recognition 1. Introduction

Contents

1. Introduction


3. Real Data Set

4. Feature Analysis


6. Conclusions



(Adaptive) intelligent tutoring systems:

I Important tool for supporting education of students(e.g. in learning fractional arithmetic)

I Advantages:I Possibility for a student to practice any timeI Possibility of adaptivity and individualization for a single

student

I Possesses internal model of the student & a task sequencerI Originally, task sequencing needs expert and domain knowledgeI Former work1: new efficient task sequencer only using former

performance information→ uses performance prediction (by matrix factorization)to sequence tasks according to the theory of Vygotskys Zoneof Proximal Development

1Schatten, C. and Schmidt-Thieme, L. 2014. Adaptive Content Sequencingwithout Domain Information. In Proceedings of the Conference on computer supportededucation (CSEDU 2014).



Proposal:Support the task sequencer and performance prediction systemusing Vygotskys theory by affect recognition on features extractedfrom speech input

→ Indicates, if the last task wastoo easy,too hard orappropriatefor the student (according to the theory of Vygotskys Zone ofProximal Development)



Contributions:

I Proposal of a new approach for supporting task sequencing byaffect recognition

I Proposal and analysis of features for affect recognitionextracted from students speech input

I Verification of the possibility to use the proposed features foraffect recognition for supporting task sequencing in adaptiveintelligent tutoring systems


Feature Analysis for Affect Recognition 2. Features, Classes, Instances, Methods

Contents

1. Introduction


3. Real Data Set

4. Feature Analysis


6. Conclusions



To clarify:

I What kind of features shall be used?

I What kind of classes shall be used?

I Which instances shall be mapped to features and labelled withthe class labels?

I Which methods shall be used?



Features:Instead of linguistic features:→ Disfluencies features (speech pauses)

I Do not inherit error of speech recognition

I Independent from need that students use words related toaffects

Classes:

I Goal: neither bore the student with too easy tasks norfrustrate him with too hard tasks, but keep him in the Zone ofProximal Development

I → Get an answer to the question “Was this task too easy, toohard or appropriate for the student?”

I → Classes: Perceived Task-Difficulty(Labels: under-challenged, over-challenged, flow)



Instances:Which instances shall be mapped to features and labeled with theclass labels?

I We need at the end of a task the information, if the taskoverall was too easy, too hard or appropriate for the student.→ Instances: whole speech input of a student for one task

Methods:I Extracting speech pause information:

I Energy threshold on the decibel scale

I Affect recognition:I Support vector machine (classification method)


Feature Analysis for Affect Recognition 3. Real Data Set

Contents

1. Introduction


3. Real Data Set

4. Feature Analysis


6. Conclusions


Feature Analysis for Affect Recognition 3. Real Data Set

Study:I 10 German students

I Age: 10 to 12 years

I Labelling (concurrently by tutor, retrospectively by secondreviewer) of perceived task-difficulties per task and per student

I Shown to students: paper sheet with fraction tasks

I Speech and screen recordings


Feature Analysis for Affect Recognition 4. Feature Analysis

Contents

1. Introduction


3. Real Data Set

4. Feature Analysis


6. Conclusions



Measurements

Symbol Explanation

m Number of students

pi Total length of pauses of student i

si Total length of speech of student i

npi Number of pause segments of student i

nsi Number of speech segments of student i

p(x)i xth pause segment of student i

s(y)i y th speech segment of student i

nti Number of tasks shown to student i

nci Number of correctly solved tasks by student i

score Overall score for student i (ncinti

)



Speech Features

Feature Formula

Ratio between pauses and speech pisi

Frequency of speech pause changesnpi +nsi

maxj (npj +nsj )

Percentage of pauses of input speech data pi(pi+si )

Length of maximal pause segment maxx(p(x)i )

Length of average pause segment∑

x p(x)i

npi

Length of maximal speech segment maxy (s(y)i )

Length of average speech segment∑

y s(y)i

nsi

Average number of seconds needed per task (pi+si )nti



Speech Features

Speech pauses carry useful information about students behaviour:

I Study showed e.g. tendency of over-challenged students tostay longer in silent thinking phases than students in flow.

Encoding of the perceived task-difficulty labels for statistical tests:

I 0 = over-challenged

I 1 = over-challenged/flow

I 2 = flow

I 3 = flow/under-challenged

I 4 = under-challenged



Single Speech Features

Investigating relevance of features:

1. Mapping the single feature values to the labels

2. Doing a linear regression

3. Measuring p-value: indicating the statistical significance

4. Measuring (Adjusted) R2 value: indicating how well theregression line can approximate the real data points

Best single speech features:

Feature p-value R2 Adjusted R2

Maximal pause 0.0678 0.3577 0.2774

Average length of speech 0.0873 0.3217 0.2369

Percentage of pauses 0.0923 0.3136 0.2278



Speech Feature Combinations

Investigating relevance of feature combinations:

1. Mapping all considered feature values to the labels

2. Doing a multivariate linear regression

3. Measuring p-value: indicating the statistical significance

4. Measuring (Adjusted) R2 value: indicating how well theregression hyper-plane can approximate the real data points



Speech Feature Combinations# Features p-value R2 Adjusted R2

Ratio pause speech, frequency of changes,5 seconds per task, average length of pause, 0.0284 0.9158 0.8106

average length of speechFrequency of changes, seconds per task,

5 average length of pause, maximal speech, 0.0305 0.9127 0.8035average length of speech

4 Ratio pause speech, frequency of changes,average length of pause, average length of speech 0.0154 0.8818 0.7872

4 Frequency of changes, average length of pause,maximal speech, average length of speech 0.0272 0.8501 0.7302

4 Ratio pause speech, frequency of changes,seconds per task, average length of speech 0.0288 0.8465 0.7236

4 Frequency of changes, seconds per task,maximal speech, average length of speech 0.0354 0.8324 0.6984

4 Frequency of changes, seconds per task,average length of pause, average length of speech 0.0420 0.8199 0.6759

3 Ratio pause speech,frequency of changes, average length of speech 0.0117 0.8207 0.7311

3 Frequency of changes,average length of pause, average length of speech 0.0175 0.7944 0.6916

3 Frequency of changes,maximal speech, average length of speech 0.0242 0.7699 0.6549

2 Frequency of changes,average length of speech 0.0327 0.6238 0.5163

→ There are statistically significant feature combinations.→ Features are able to describe perceived task-difficulty.



Histograms as Features

Over-challengedstudent:

0 20 40 60 80 100 120

05

1015

2025

Pause length

Fre

quen

cy

0 20 40 60 80 100

02

46

810

Speech length

Fre

quen

cy

Studentin flow:

0 20 40 60 80 100 120

05

1015

2025

Pause length

Fre

quen

cy

0 20 40 60 80 100

02

46

810

Speech length

Fre

quen

cy


Feature Analysis for Affect Recognition 5. Supporting Task Sequencing

Contents

1. Introduction


3. Real Data Set

4. Feature Analysis


6. Conclusions


Feature Analysis for Affect Recognition 5. Supporting Task Sequencing

(1) Use perceived task-difficultyfor rules:too easy → harderappropriate → similar difficulttoo hard → easier

(2) Feed feature values or affectinto performance predictor.

I E.g. difference between realperformance and scorecomputed by featuresindicates outliers (like if astudent felt to be in a flowbut his score is worse).

(3) Learn personalised value forthreshold based on VygotskysZone of Proximal Developmentused by the sequencer.

Task sequencer

Performanceprediction

Affectrecognition

Featureextraction

Next task

(1)

(1)

(2a)

(2b)

Formerperformances

Speechinput


Feature Analysis for Affect Recognition 6. Conclusions

Contents

1. Introduction


3. Real Data Set

4. Feature Analysis


6. Conclusions


Feature Analysis for Affect Recognition 6. Conclusions

Proposed:

I Approach for supporting task sequencing by affect recognition

I Appropriate speech features

Shown:

I There are statistically significant feature combinations forperceived task-difficulty

I → The proposed features are applicable for affect recognitionsupporting task sequencing in intelligent tutoring systems

Next steps:

I Training of an appropriate affect recognition method


Feature Analysis for Affect Recognition

Thank youfor your attention!

This work is co-funded by the EU project iTalk2Learnwith grant agreement no. 318051.

Project webpage: http://www.italk2learn.eu


feature analysis for affect recognition supporting task sequencing in adaptive intelligent tutoring...

Data & Analytics