adapting to student uncertainty improves tutoring dialogues

Adapting to Student Uncertainty Improves Tutoring Dialogues

Kate Forbes-Riley and Diane Litman

University of PittsburghPittsburgh, PA USA

Outline

Overview

System: Original and Adaptive Versions

Evaluation of Uncertainty Adaptations

Conclusions, Future Work

Background Student uncertainty is of interest in tutoring community

Correlates with learning (Craig et al., 2004) Co-occurs with incorrectness (Bhatt et al., 2003) Annotated and detected (D’Mello et al., 2008)

Few computer tutors have evaluated substantive responses to uncertainty Human-based positive feedback responses improved

satisfaction (Tsukahara and Ward, 2001) and persistence (Aist et al., 2002)

Human-based substantive responses didn’t improve learning, but limited detection scheme (Pon-Barry et al., 2006)

This Paper

We show responding to uncertainty with additional content can significantly improve computer tutoring performance

2 uncertainty adaptations evaluated in Wizard of Oz experiment

Performance gains measured for learning efficiency and user satisfaction

Normal (non-adaptive) Computer Tutor ITSPOKE (Intelligent Tutoring Spoken Dialogue System)

Back-end: text-based Why2-Atlas (VanLehn, Jordan, Rosé et al., 2002)

Tutors 5 qualitative physics problems Dialogue Format: Question – Student Answer – Response

Response Types: to Corrects (C): positive feedback (e.g. “Fine”) to Incorrects (I): negative feedback (e.g. “Well…”) and

Bottom Out: correct answer with reasoning (easier) Subdialogue: questions walk through reasoning (harder)

Tutoring Theory: Uncertainty and Incorrectness both signal Learning Impasses (opportunities to better learn concepts (VanLehn et

al., 2003))

Our Prior Work: Rank correctness (C, I) + uncertainty (U, nonU) states in terms of impasse severity

State: I+nonU I+U C+U C+nonU

Severity: most less least none

Adaptation Hypothesis:

ITSPOKE already provides content to resolve I impasses (I+U, I+nonU), but it ignores one type of U impasse (C+U)

Performance improvement if ITSPOKE provides additional content to resolve all impasses

Adaptive Computer Tutor(s)

Simple Adaptation Provide same substantive response for all 3 impasses

Complex Adaptation Provide different substantive responses for the 3

impasses

Acknowledge student uncertainty in feedback

Two Uncertainty Adaptations

For C+U, I+U, I+nonU answers

ITSPOKE gives same content with same dialogue act

ITSPOKE gives feedback on (in)correctness

Simple Adaptation to Uncertainty

Simple Adaptation ExampleTUTOR1: By the same reasoning that we used for the car, what’s the

overall net force on the truck equal to?

STUDENT1: The force of the car hitting it?? [C+U]

TUTOR2: Fine. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we

did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Same TUTOR2 subdialogue if student was I+U or I+nonU

Depending on if answer is C+U, I+U, I+nonU:

ITSPOKE gives same content but varies dialogue act

Based on human tutor responses significantly associated with C+U, I+U, I+nonU answers

ITSPOKE gives complex feedback on uncertainty and (in)correctness

Based on empathetic computer tutor literature (Wang et al., 2005; Hall et al., 2004; Burleson et al., 2004)

Complex Adaptation to Uncertainty

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: The impact force of the car on the truck?? [C+U]

TUTOR2: That’s exactly right, but you seem unsure, so let’s sum up. [FEEDBACK] The net force on the truck is equal to the impact force on it. We can prove this just like we did for the car. First, we know that gravity and the normal force on the truck must cancel each other, otherwise the truck would not be at rest vertically. Second, we know that the impact force is the only horizontal force exerted on the truck. [LONG BOTTOM OUT]

Complex Adaptation Example: C+U

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: Zero?? [I+U]

TUTOR2: That’s not correct, but don’t worry. You seem to be aware of your mistake. Let’s resolve it. [FEEDBACK] The net force on the truck is equal to the impact force on it. Let’s walk through this answer step by step. [SHORT BOTTOM OUT] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Complex Adaptation Example: I+U

TUTOR1: By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?

STUDENT1: The force of gravity. [I+nonU]

TUTOR2: I’m sorry, but I see an error in your answer. Let’s fix it. [FEEDBACK] We can derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [SUBDIALOGUE]

Complex Adaptation Example: I+nonU

Experimental Design Wizard of Oz version of ITSPOKE

Human wizard performed speech recognition, natural language understanding, and annotated correctness and uncertainty

4 Conditions Simple Adaptation: used same response for all impasses Complex Adaptation: used different responses for each impasse Normal Control: used original system (no adaptation) Random Control: gave Simple Adaptation to random 20% of

correct answers (to control for additional tutoring)

Prediction: Complex Adaptation > Simple Adaptation > Random Control > Normal Control

Procedure: reading, pretest, 5 problems, survey, posttest

Learning efficiency: Amount of learning achieved in a given amount of tutoring (e.g., Ringenberg and VanLehn, 2006)

Learning gain / total tutoring time in minutes

User Satisfaction: subjective student perceptions of system performance as measured by survey (e.g., Baylor et al., 2003; Walker et al., 2001)

Total survey score

Score for each survey question

For each metric:

1-way ANOVA with condition as between-subjects factor

Paired contrasts tests for each pair of conditions

Evaluation Metrics

Survey

Tutoring

Uncertainty

Spoken Dialogue

Learning Efficiency Results

Metric Condition N Mean Diff p

Normalized learning gain / total tutoring

time in minutes

Normal Control 21 .010 < Simple Adapt .004

Random Control 20 .014 -

Simple Adaptation 20 .016 -

Complex Adaptation 20 .011 < Simple Adapt .013

Given same amount of tutoring time, Simple Adaptation yields more student learning than either Normal Control or Complex Adaptation

Results also hold using raw learning gain, and total number of student turns

F(3, 77) = 3.56, p = 0.02

Survey Results

Metric Condition N Mean Diff p

Spoken Dialogue Question 13

Normal Control 21 3.90 -

Random Control 20 4.15 > Simple Adapt .016

Simple Adaptation 20 3.50 -

Complex Adaptation 20 4.15 > Simple Adapt .016

Spoken Dialogue Question 13: “It was easy to understand the tutor”

Students perceive tutor in Simple Adaptation as hard to understand

May reflect student confusion as to why Simple Adaptation was treating C+U answers as incorrect – students already uncertain at this point

F(3, 77) = 2.69, p = 0.05

Satisfaction-Learning Correlations Survey results suggest no strong student preference for either

uncertainty-adaptive ITSPOKE tutoring system

Is there a relationship between student preferences and learning?

E.g., subjects who prefer Complex Adaptation may learn more from it than those who don’t prefer it

Mixed prior results (e.g., Moreno et al., 2002; Rotaru, 2008)

Pearson’s correlation between each user satisfaction metric and posttest (controlled for pretest) over all ITSPOKE tutors (conditions) and for each tutor

Satisfaction-Learning Correlations:Simple Adaptation

• Tutoring Question 7: “The tutor helped me concentrate.” (R = 0.482, p = 0.037) Those who perceived more concentration learned more

• Uncertainty Question 12: “The tutor’s responses decreased my uncertainty about my understanding of the content.” (R = 0.432, p = 0.065) Simple Adaptation “works”: even if not most preferred overall,

it is decreasing uncertainty while increasing learning

Discussion

Why didn’t Simple Adaptation and Complex Adaptation outperform Random Control? Random Control adapted to some C+U, diminishing differences Adapting to C+nonU may increase certainty

Why didn’t Complex Adaptation outperform Simple Adaptation? Complex Adaptation’s feedback and content elements may differ

in effectiveness Complex Adaptation’s human-based content responses were based

on frequency, not effectiveness

Conclusions Adapting to student uncertainty during wizarded

computer tutoring improves learning efficiency and user satisfaction Simple Adaptation improved learning efficiency,

had positive correlation between learning and student perception of decreased uncertainty

Complex Adaptation showed trend for improvement on user perception of tutor response quality

Current and Future Work User Modeling (Interspeech 2009) and Metacognitive

data analysis

Investigate other approaches for developing complex uncertainty adaptations

reinforcement learning

dialogue act-learning correlations

Replicate analysis using recently collected data from fully automated ITSPOKE

Questions?

Further Information?

web search: ITSPOKE

Thank You!

Simple Adaptation: For CU, IU, InonU answers:

ITSPOKE gives same content with same dialogue act

ITSPOKE gives feedback on (in)correctness

Complex Adaptation: Depending on if answer is CU, IU, InonU:

ITSPOKE gives same content but varies dialogue act

Based on human tutor responses significantly associated with CU, IU, InonU answers

ITSPOKE gives complex feedback on affect and (in)correctness

Based on empathetic computer tutor literature (Wang et al., 2005; Hall et al., 2004; Burleson et al., 2004)

Two Uncertainty Adaptations

Tutoring Theory: Uncertainty and Incorrectness both signal a Learning Impasse: opportunity to better learn concept (VanLehn et al., 2003)

Uncertainty indicates impasse perceived, so rank correctness (C,I) + uncertainty (U, nonU) states in terms of impasse severity:

State: InonU IU CU CnonU

Severity: most less least none

Adaptation Hypothesis: ITSPOKE already provides additional content to resolve I impasses (IU,

InonU), but it ignores one type of U impasse (CU) Performance improvement if ITSPOKE provides additional content to

resolve all impasses

Two Uncertainty Adaptations

Satisfaction-Learning Correlations Normal:

“The tutor worked the way I expected it to.” (R = -0.382, p = 0.096)

Those who perceived a hard time using system learned more

Random:

“It was easy to learn from the tutor.” (R = 0.401, p = 0.089)

Those who perceived an easy time using system learned more

Simple:

“The tutor helped me to concentrate.” (R = 0.482, p = 0.037)

Those who perceived more concentration learned more

“The tutor’s responses decreased my uncertainty about my understanding of the content.” (R = 0.432, p = 0.065)

Simple “works”: even if not most preferred overall, it is decreasing uncertainty while increasing learning

Efficiency (TOT) DifferencesMetric Condition N Mean Diff p

Time On Task (min)

Normal Control 21 40.92 - -

Random Control 20 43.79 - -

Simple Adaptation 20 39.49 - -

Complex Adaptation 20 40.66 - -

F(3, 77) = 0.774, p = 0.51