adapting to student uncertainty improves tutoring dialogues

29
Adapting to Student Uncertainty Improves Tutoring Dialogues Kate Forbes-Riley and Diane Litman University of Pittsburgh Pittsburgh, PA USA

Upload: tokala

Post on 12-Jan-2016

27 views

Category:

Documents


0 download

DESCRIPTION

Adapting to Student Uncertainty Improves Tutoring Dialogues. Kate Forbes-Riley and Diane Litman University of Pittsburgh Pittsburgh, PA USA. Outline. Overview System: Original and Adaptive Versions Evaluation of Uncertainty Adaptations Conclusions, Future Work. Background. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Adapting to Student Uncertainty Improves Tutoring Dialogues

Adapting to Student Uncertainty Improves Tutoring Dialogues

Kate Forbes-Riley and Diane Litman

University of PittsburghPittsburgh, PA USA

Page 2: Adapting to Student Uncertainty Improves Tutoring Dialogues

Outline

Overview

System: Original and Adaptive Versions

Evaluation of Uncertainty Adaptations

Conclusions, Future Work

Page 3: Adapting to Student Uncertainty Improves Tutoring Dialogues

Background Student uncertainty is of interest in tutoring community

Correlates with learning (Craig et al., 2004) Co-occurs with incorrectness (Bhatt et al., 2003) Annotated and detected (D’Mello et al., 2008)

Few computer tutors have evaluated substantive responses to uncertainty Human-based positive feedback responses improved

satisfaction (Tsukahara and Ward, 2001) and persistence (Aist et al., 2002)

Human-based substantive responses didn’t improve learning, but limited detection scheme (Pon-Barry et al., 2006)

Page 4: Adapting to Student Uncertainty Improves Tutoring Dialogues

This Paper

We show responding to uncertainty with additional content can significantly improve computer tutoring performance

2 uncertainty adaptations evaluated in Wizard of Oz experiment

Performance gains measured for learning efficiency and user satisfaction

Page 5: Adapting to Student Uncertainty Improves Tutoring Dialogues

Normal (non-adaptive) Computer Tutor ITSPOKE (Intelligent Tutoring Spoken Dialogue System)

Back-end: text-based Why2-Atlas (VanLehn, Jordan, Rosé et al., 2002)

Tutors 5 qualitative physics problems Dialogue Format: Question – Student Answer – Response

Response Types: to Corrects (C): positive feedback (e.g. “Fine”) to Incorrects (I): negative feedback (e.g. “Well…”) and

Bottom Out: correct answer with reasoning (easier) Subdialogue: questions walk through reasoning (harder)

Page 6: Adapting to Student Uncertainty Improves Tutoring Dialogues

Tutoring Theory: Uncertainty and Incorrectness both signal Learning Impasses (opportunities to better learn concepts (VanLehn et

al., 2003))

Our Prior Work: Rank correctness (C, I) + uncertainty (U, nonU) states in terms of impasse severity

State: I+nonU I+U C+U C+nonU

Severity: most less least none

Adaptation Hypothesis:

ITSPOKE already provides content to resolve I impasses (I+U, I+nonU), but it ignores one type of U impasse (C+U)

Performance improvement if ITSPOKE provides additional content to resolve all impasses

Adaptive Computer Tutor(s)

Page 7: Adapting to Student Uncertainty Improves Tutoring Dialogues

Simple Adaptation Provide same substantive response for all 3 impasses

Complex Adaptation Provide different substantive responses for the 3

impasses

Acknowledge student uncertainty in feedback

Two Uncertainty Adaptations

Page 8: Adapting to Student Uncertainty Improves Tutoring Dialogues

For C+U, I+U, I+nonU answers

ITSPOKE gives same content with same dialogue act

ITSPOKE gives feedback on (in)correctness

Simple Adaptation to Uncertainty

Page 9: Adapting to Student Uncertainty Improves Tutoring Dialogues

Simple Adaptation ExampleTUTOR1: By the same reasoning that we used for the car, what’s the

overall net force on the truck equal to?

STUDENT1: The force of the car hitting it?? [C+U]

TUTOR2: Fine. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we

did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Same TUTOR2 subdialogue if student was I+U or I+nonU

Page 10: Adapting to Student Uncertainty Improves Tutoring Dialogues

Depending on if answer is C+U, I+U, I+nonU:

ITSPOKE gives same content but varies dialogue act

Based on human tutor responses significantly associated with C+U, I+U, I+nonU answers

ITSPOKE gives complex feedback on uncertainty and (in)correctness

Based on empathetic computer tutor literature (Wang et al., 2005; Hall et al., 2004; Burleson et al., 2004)

Complex Adaptation to Uncertainty

Page 11: Adapting to Student Uncertainty Improves Tutoring Dialogues

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: The impact force of the car on the truck?? [C+U]

TUTOR2: That’s exactly right, but you seem unsure, so let’s sum up. [FEEDBACK] The net force on the truck is equal to the impact force on it. We can prove this just like we did for the car. First, we know that gravity and the normal force on the truck must cancel each other, otherwise the truck would not be at rest vertically. Second, we know that the impact force is the only horizontal force exerted on the truck. [LONG BOTTOM OUT]

Complex Adaptation Example: C+U

Page 12: Adapting to Student Uncertainty Improves Tutoring Dialogues

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: Zero?? [I+U]

TUTOR2: That’s not correct, but don’t worry. You seem to be aware of your mistake. Let’s resolve it. [FEEDBACK] The net force on the truck is equal to the impact force on it. Let’s walk through this answer step by step. [SHORT BOTTOM OUT] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Complex Adaptation Example: I+U

Page 13: Adapting to Student Uncertainty Improves Tutoring Dialogues

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: The force of gravity. [I+nonU]

TUTOR2: I’m sorry, but I see an error in your answer. Let’s fix it. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Complex Adaptation Example: I+nonU

Page 14: Adapting to Student Uncertainty Improves Tutoring Dialogues

Experimental Design Wizard of Oz version of ITSPOKE

Human wizard performed speech recognition, natural language understanding, and annotated correctness and uncertainty

4 Conditions Simple Adaptation: used same response for all impasses Complex Adaptation: used different responses for each impasse Normal Control: used original system (no adaptation) Random Control: gave Simple Adaptation to random 20% of

correct answers (to control for additional tutoring)

Prediction: Complex Adaptation > Simple Adaptation > Random Control > Normal Control

Procedure: reading, pretest, 5 problems, survey, posttest

Page 15: Adapting to Student Uncertainty Improves Tutoring Dialogues

Learning efficiency: Amount of learning achieved in a given amount of tutoring (e.g., Ringenberg and VanLehn, 2006)

Learning gain / total tutoring time in minutes

User Satisfaction: subjective student perceptions of system performance as measured by survey (e.g., Baylor et al., 2003; Walker et al., 2001)

Total survey score

Score for each survey question

For each metric:

1-way ANOVA with condition as between-subjects factor

Paired contrasts tests for each pair of conditions

Evaluation Metrics

Page 16: Adapting to Student Uncertainty Improves Tutoring Dialogues

Survey

Tutoring

Uncertainty

Spoken Dialogue

Page 17: Adapting to Student Uncertainty Improves Tutoring Dialogues

Learning Efficiency Results

Metric Condition N Mean Diff p

Normalized learning gain / total tutoring

time in minutes

Normal Control 21 .010 < Simple Adapt .004

Random Control 20 .014 -

Simple Adaptation 20 .016 -

Complex Adaptation 20 .011 < Simple Adapt .013

Given same amount of tutoring time, Simple Adaptation yields more student learning than either Normal Control or Complex Adaptation

Results also hold using raw learning gain, and total number of student turns

F(3, 77) = 3.56, p = 0.02

Page 18: Adapting to Student Uncertainty Improves Tutoring Dialogues

Survey Results

Metric Condition N Mean Diff p

Spoken Dialogue Question 13

Normal Control 21 3.90 -

Random Control 20 4.15 > Simple Adapt .016

Simple Adaptation 20 3.50 -

Complex Adaptation 20 4.15 > Simple Adapt .016

Spoken Dialogue Question 13: “It was easy to understand the tutor”

Students perceive tutor in Simple Adaptation as hard to understand

May reflect student confusion as to why Simple Adaptation was treating C+U answers as incorrect – students already uncertain at this point

F(3, 77) = 2.69, p = 0.05

Page 19: Adapting to Student Uncertainty Improves Tutoring Dialogues

Satisfaction-Learning Correlations Survey results suggest no strong student preference for either

uncertainty-adaptive ITSPOKE tutoring system

Is there a relationship between student preferences and learning?

E.g., subjects who prefer Complex Adaptation may learn more from it than those who don’t prefer it

Mixed prior results (e.g., Moreno et al., 2002; Rotaru, 2008)

Pearson’s correlation between each user satisfaction metric and posttest (controlled for pretest) over all ITSPOKE tutors (conditions) and for each tutor

Page 20: Adapting to Student Uncertainty Improves Tutoring Dialogues

Satisfaction-Learning Correlations:Simple Adaptation

• Tutoring Question 7: “The tutor helped me concentrate.” (R = 0.482, p = 0.037) Those who perceived more concentration learned more

• Uncertainty Question 12: “The tutor’s responses decreased my uncertainty about my understanding of the content.” (R = 0.432, p = 0.065) Simple Adaptation “works”: even if not most preferred overall,

it is decreasing uncertainty while increasing learning

Page 21: Adapting to Student Uncertainty Improves Tutoring Dialogues

Discussion

Why didn’t Simple Adaptation and Complex Adaptation outperform Random Control? Random Control adapted to some C+U, diminishing differences Adapting to C+nonU may increase certainty

Why didn’t Complex Adaptation outperform Simple Adaptation? Complex Adaptation’s feedback and content elements may differ

in effectiveness Complex Adaptation’s human-based content responses were based

on frequency, not effectiveness

Page 22: Adapting to Student Uncertainty Improves Tutoring Dialogues

Conclusions Adapting to student uncertainty during wizarded

computer tutoring improves learning efficiency and user satisfaction Simple Adaptation improved learning efficiency,

had positive correlation between learning and student perception of decreased uncertainty

Complex Adaptation showed trend for improvement on user perception of tutor response quality

Page 23: Adapting to Student Uncertainty Improves Tutoring Dialogues

Current and Future Work User Modeling (Interspeech 2009) and Metacognitive

data analysis

Investigate other approaches for developing complex uncertainty adaptations

reinforcement learning

dialogue act-learning correlations

Replicate analysis using recently collected data from fully automated ITSPOKE

Page 24: Adapting to Student Uncertainty Improves Tutoring Dialogues

Questions?

Further Information?

web search: ITSPOKE

Thank You!

Page 25: Adapting to Student Uncertainty Improves Tutoring Dialogues
Page 26: Adapting to Student Uncertainty Improves Tutoring Dialogues

Simple Adaptation: For CU, IU, InonU answers:

ITSPOKE gives same content with same dialogue act

ITSPOKE gives feedback on (in)correctness

Complex Adaptation: Depending on if answer is CU, IU, InonU:

ITSPOKE gives same content but varies dialogue act

Based on human tutor responses significantly associated with CU, IU, InonU answers

ITSPOKE gives complex feedback on affect and (in)correctness

Based on empathetic computer tutor literature (Wang et al., 2005; Hall et al., 2004; Burleson et al., 2004)

Two Uncertainty Adaptations

Page 27: Adapting to Student Uncertainty Improves Tutoring Dialogues

Tutoring Theory: Uncertainty and Incorrectness both signal a Learning Impasse: opportunity to better learn concept (VanLehn et al., 2003)

Uncertainty indicates impasse perceived, so rank correctness (C,I) + uncertainty (U, nonU) states in terms of impasse severity:

State: InonU IU CU CnonU

Severity: most less least none

Adaptation Hypothesis: ITSPOKE already provides additional content to resolve I impasses (IU,

InonU), but it ignores one type of U impasse (CU) Performance improvement if ITSPOKE provides additional content to

resolve all impasses

Two Uncertainty Adaptations

Page 28: Adapting to Student Uncertainty Improves Tutoring Dialogues

Satisfaction-Learning Correlations Normal:

“The tutor worked the way I expected it to.” (R = -0.382, p = 0.096)

Those who perceived a hard time using system learned more

Random:

“It was easy to learn from the tutor.” (R = 0.401, p = 0.089)

Those who perceived an easy time using system learned more

Simple:

“The tutor helped me to concentrate.” (R = 0.482, p = 0.037)

Those who perceived more concentration learned more

“The tutor’s responses decreased my uncertainty about my understanding of the content.” (R = 0.432, p = 0.065)

Simple “works”: even if not most preferred overall, it is decreasing uncertainty while increasing learning

Page 29: Adapting to Student Uncertainty Improves Tutoring Dialogues

Efficiency (TOT) DifferencesMetric Condition N Mean Diff p

Time On Task (min)

Normal Control 21 40.92 - -

Random Control 20 43.79 - -

Simple Adaptation 20 39.49 - -

Complex Adaptation 20 40.66 - -

F(3, 77) = 0.774, p = 0.51