measuring the effectiveness of error messages designed for novice programmers
DESCRIPTION
Measuring the Effectiveness of Error Messages Designed for Novice Programmers . Guillaume Marceau Kathi Fisler (WPI) Shriram Krishnamurthi (Brown) Danny Yoo (WPI/Brown , Utah). define: expected only one expression for the function body, but found at least one extra part. - PowerPoint PPT PresentationTRANSCRIPT
1
Measuring the Effectiveness of Error Messages Designed for Novice Programmers Guillaume MarceauKathi Fisler (WPI)Shriram Krishnamurthi (Brown)Danny Yoo (WPI/Brown, Utah)
2
define: expected only one expression for the function body, but found at least one extra part
How well do error messages support learning(or fail to?)
When errors fail to teach,in which ways do they fail?
What makes a good error message?What is a valid metric of quality?
Can we make recommendations to the creators of pedagogical IDEs/compilers/ languages?
Prior Approaches to Message Effectiveness
“Were the messages helpful?”
No alternative
Did grades improve?
Too coarse
5
6
= execution
= receives an error message
The Student’s Response to the Error Message
= keystroke
[DEL] Deletes the problematic code wholesale.
[UNR]Unrelated to the error message, anddoes not help.
[DIFF]Unrelated to the error message, but it correctly addresses a different error or makes progress in some other way
[PART] Evidence that the student has understood the error message (though perhaps not wholly,) and is trying to take an appropriate action (though perhaps not well.)
[FIX] Fixes the proximate error (though other cringing errors might remain.)
8
Rater #1
goodbad
goodgoodbad
Rater #2goodbad
goodgoodgood
9
Rater #2Rater #1
Inter-rater agreement : κ0.80 0.58
0.580.38
Expected Agreement = 0.702 + 0.302
= 0.58
0.0 1.0slight fair moderate substantial excellent
0.20
0.40
0.60
0.80
0.84
0.38
eq. to chance
perfect agreement
worse than chance
10
Lab number #1 #2 #3# of errors a studentreceived during this lab(on average)
8.5 16.3 14.4
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 5% 48% 0.22 17% 27% 0.74 14% 17% 0.33parens matching 28% 24% 0.58 12% 14% 0.27 17% 0% 0.00runtime cond 3% 0% 0.00 3% 100% 0.49 4% 20% 0.12runtime type 2% 100% 0.15 8% 73% 0.91 16% 40% 0.93syntax cond 14% 51% 0.59 4% 50% 0.31 6% 26% 0.24syntax defi ne 16% 50% 0.68 14% 50% 1.14 6% 15% 0.14syntax func. ca l l 14% 64% 0.74 14% 17% 0.37 12% 14% 0.26syntax s truct 0% 0% 0.00 8% 32% 0.43 5% 92% 0.73unbound id. 16% 16% 0.21 13% 40% 0.85 16% 14% 0.32Total: 3.16 5.51 3.07
#error × %error × %bad = #bad
11
Lab number #1 #2 #3# of errors a studentreceived during this lab(on average)
8.5 16.3 14.4
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 5% 48% 0.22 17% 27% 0.74 14% 17% 0.33parens matching 28% 24% 0.58 12% 14% 0.27 17% 0% 0.00runtime cond 3% 0% 0.00 3% 100% 0.49 4% 20% 0.12runtime type 2% 100% 0.15 8% 73% 0.91 16% 40% 0.93syntax cond 14% 51% 0.59 4% 50% 0.31 6% 26% 0.24syntax defi ne 16% 50% 0.68 14% 50% 1.14 6% 15% 0.14syntax func. ca l l 14% 64% 0.74 14% 17% 0.37 12% 14% 0.26syntax s truct 0% 0% 0.00 8% 32% 0.43 5% 92% 0.73unbound id. 16% 16% 0.21 13% 40% 0.85 16% 14% 0.32Total: 3.16 5.51 3.07
Lab number #4 #5 #6# of errors a studentreceived during this lab(on average)
9.0 9.8 9.8
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 13% 20% 0.24 35% 21% 0.74 12% 31% 0.36parens matching 14% 0% 0.00 13% 0% 0.00 10% 15% 0.15runtime cond 6% 72% 0.40 8% 78% 0.62 1% 100% 0.06runtime type 8% 22% 0.17 6% 44% 0.26 3% 38% 0.13syntax cond 10% 28% 0.25 9% 20% 0.17 11% 11% 0.12syntax defi ne 7% 24% 0.14 2% 17% 0.03 3% 38% 0.10syntax func. ca l l 23% 27% 0.55 4% 29% 0.12 13% 38% 0.48syntax s truct 0% 0% 0.00 1% 0% 0.00 0% 0% 0.00unbound id. 16% 0% 0.00 20% 7% 0.14 34% 13% 0.44Total: 1.75 2.07 1.84
Vocabulary difficulties
n-ary tree
Difficult structure accessor names
Reflection
Rubric identifies pain points
Identifying fix needs deeper evaluationCurriculum Error Messages
12
13
For the Community
Look at fine-grained editsGather around a common rubric
14
Interviews
Four interviewsOne hour long eachDone around the midtermAverage-to-good students
15
Observation From Interviews
1 of 2
16
Interviewer: The error message says “the function body.”Do you know what “function body” means?
Student: Nah, the input, everything that serves
as a piece of input? Interviewer: Actually, it's this. When DrScheme
says “function body” it means this part. Student: Oh man! I didn't… [The student proceeds to fix the error successfully]
17
What DrRacket Says:
define: expected only one expression for the function body, but found at least one extra part.
18
What the Student Sees:
define: expected only one thingamabob for the geewhiz’s whosiewhatsit, but found one extra whatchamacallit.
19
Circle an identifier in this program
(define (celsius->fahrenheit temp) (+ (* temp 9/5) 32))
RIGHT WRONG
20
Circle one instance of each vocabulary term in the code below
Vocabulary term Sample usage
Q1. Argument >: expects at least 2 arguments, given 1Q2. Selector this selector expects 1 argument, here it is provided 0 argumentsQ3. Procedure this procedure expects 2 arguments, here it is provided 0 argumentsQ4. Expression expected at least two expressions after `and', but found only one expressionQ5. Predicate this predicate expects 1 argument, here it is provided 2 arguments
;; (make-book number string string number number bst bst) (define-struct book (isbn title author year copies left right))
;; this-edition?: bst number number -> boolean;; Consumes a binary search tree, an ISBN number, and a year, and produces true ;; if the book with the given ISBN number was published in the given year(define (this-edition? a-bst isbn-num year) (cond [(symbol? a-bst) false] [(book? a-bst) (cond [(= isbn-num (book-isbn a-bst)) (= year (book-year a-bst))] [(< isbn-num (book-isbn a-bst)) (this-edition? (book-left a-bst) isbn-num year)] [else (this-edition? (book-right a-bst) isbn-num year)])]))
21
< 50%
22
Serendipitous Controlled Trials
BrownNEU WPI
Primitive name Procedure Primitive operator Field name Procedure application Predicate Defined name Type name Identifier Function body Function header
Argument Clause Expression Selector
= USED IN CLASS
Number of times the word was presented to a student in an error message (WPI)
Quiz score residual(after factoring out word difficulty and university strength)
Wordsused in class
Words NOTused in class
Wordsused in class
Words NOTused in class
p = 0.034
(which is statistically significant at the
0.05 level)
For words used in class:more exposure in error messages better quiz scores
For words not used in class:more exposure in error messages worse quiz scores
25
Old term New termProcedurePrimitive namePrimitive operatorPredicateSelectorConstructor
Function
NameIdentifier ArgumentDefined name
Variable, argument(“argument” is reserved for actualarguments in function calls)
Sequence At least oneStructure type name Structure nameQuestion—answer clause A clause is expected to have a question and an answerFunction header KeywordType< > ‘ ’
These words and notations are removed entirely and reworded in terms of other vocabulary words.
Function bodyExpressionField nameType nameTop levelBindingClausePart
These words stay unchanged
26
Observation From Interviews
2 of 2
27
Interviewer: When you get these highlights, what do they mean to you?
Student #1: The problem is between here and here, fix the problem between these two bars.
28
1. This expression contains the error2. The parser didn’t expect to find
this3. The parser expected to see
something after this, but nothing is there
4. This parenthesis is unmatched5. This expression is inconsistent
with another part of the code
DrRacket’s Highlight Semantics
29
Summary of Findings1. Error messages need to explicate the
meaning of the highlight.
2. Students need an avenue through which they will learn the vocabulary.
3. Error messages are hard to get right;user studies are important.
30
Read Understand Formulate
goodbad
goodgoodbad
La b number #4 #5 #69.0 9.8 9.8
% error % bad # bad % error % bad # bad % error % bad # bad
arg. count 13% 20% 0.24 35% 21% 0.74 12% 31% 0.36parens matching 14% 0% 0.00 13% 0% 0.00 10% 15% 0.15runti me cond 6% 72% 0.40 8% 78% 0.62 1% 100% 0.06runti me type 8% 22% 0.17 6% 44% 0.26 3% 38% 0.13synta x cond 10% 28% 0.25 9% 20% 0.17 11% 11% 0.12synta x defi ne 7% 24% 0.14 2% 17% 0.03 3% 38% 0.10synta x func. ca l l 23% 27% 0.55 4% 29% 0.12 13% 38% 0.48synta x struct 0% 0% 0.00 1% 0% 0.00 0% 0% 0.00unbound id. 16% 0% 0.00 20% 7% 0.14 34% 13% 0.44Total: 1.75 2.07 1.84
n-ary tree
[FIX] Fixes the proximate error (though other cringing errors might remain.)[PART] Evidence that the student has understood the error message (though perhaps not wholly,) and is trying to take an appropriate action (though perhaps not well.)[UNR] Edit unrelated to the error message, and does not help.[DEL] Deletes the problematic code wholesale.[DIFF] Edit unrelated to the error message, but it correctly addresses a different error or makes progress in some other way.
Kathi Fisler (WPI)Shriram Krishnamurthi (Brown)[email protected]@cs.brown.edu