d esigning an i nteractive t eaching t ool with abml k nowledge r efinement l oop enabling arguing...
Post on 01-Apr-2015
217 Views
Preview:
TRANSCRIPT
DESIGNING AN INTERACTIVE TEACHING TOOL WITH ABML
KNOWLEDGE REFINEMENT LOOPenabling arguing to learn
1 Faculty of Education, University of Ljubljana, Slovenia
2 Faculty of Computer and Information Science, University of Ljubljana, Slovenia
Matej Zapušek1, Martin Možina2, Ivan Bratko2, Jože Rugelj2, Matej Guid2
12th International Conference on Intelligent Tutoring SystemsITS 2014: Honolulu, Hawaii 2014
SOME CONCEPTS ARE DIFFICULT TO EXPLAIN...
How to distinguish edible from toxic mushrooms?
INTRODUCING DOMAIN EXPERTS
Even for domain experts it is hard to articulate their knowledge!
MACHINE LEARNING: HOW TO INVOLVE A DOMAIN EXPERT?
The expert can state constraints and the domain knowledge in advance...
… verify, evaluate, and correct results of machine learning…
… or the expert and the computer iteratively improve the model.
ABMLargument-based machine learning
ARGUMENT-BASED MACHINE LEARNING
given
set of labeled learning examples ei
described with attribute values Di
where Ci is classification of learning example ei goal
learn prediction model (hypoteshis) H
IF ... THEN ...IF ... THEN ......
HCiei: Di
ai
example ei may have argument ai
IT IS MUCH EASIER TO EXPLAIN INDIVIDUAL CASES!
Is this mushroom toxic? Why?
ABML KNOWLEDGE REFINEMENT LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Expert explains the example
Return to step 1
critical example
learn data set
ArgumentABML
ABML KNOWLEDGE REFINEMENT LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Expert explains the example
Return to step 1 Step 3a: Explaining a critical example (in a natural language)
Step 3b: Adding arguments to the example
Step 3c: Discovering counter examples
Step 3d: Improving arguments
Return to step 3c if counter example found
ILLUSTRATIVE EXAMPLE: LEARNING TO DIAGNOSE FLU
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
The current model:
IF Temperature < very high THEN Flu = no
cannot explain well Pacient 2.
The question to the expert:
„What is the reason for Pacient 2 having the flu?“
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
EXPERT‘S EXPLANATION
Expert‘s explanation:
„Pacient #2 has the flue because of a high temperature.“
Expert‘s argument
is attached to learning example #2. New model is built.
Flu = yes BECAUSE Temperature > Normal
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
WHAT IF THE EXPERT‘S ARGUMENT IS NOT GOOD ENOUGH?
ML method now induced a rule consistent with argument:
IF Temperature > Normal THEN Flu = yes
The rule is inconsistent with data! Expert is presented with counter example:
„Compare pacients #2 and #4. Why Pacient #4 doesn‘t have the flu?“
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
THE EXPERT MAY IMPROVE THE ARGUMENT
Expert finds the crucial difference between Pacients #2 and #4:
„Pacient 2 didn‘t get vaccinated against the flu.“
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
IMPROVED RULES MAY EXPLAIN UNSEEN EXAMPLES AS WELL
ML method induces a new rule:
IF Temperature > Normal AND Vaccination = no
THEN Flue = yes
The new rule also explains diagnosis for Pacient #3:
„Has flu because of a high temperature and didn‘t get vaccinated against it.“
ABML KNOWLEDGE REFINEMENT LOOP: THE INNER LOOP
Step 3a: Explaining a critical example (in a natural language)
„Pacient #2 has the flue because of a high temperature.“
Step 3b: Adding arguments to the example
Step 3c: Discovering counter examples
Step 3d: Improving arguments with counter examples
IF TEMPERATURE > NORMAL AND VACCINATION = NO
Temperature > Normal
ABML REFINEMENT LOOP & KNOWLEDGE ELICITATION
IF ... THEN ...IF ... THEN ......
ABMLargument-based machine learning
explain single example easier for experts to articulate knowledge
“critical” examples expert provides only relevant knowledge
“counter” examples detect deficiencies in explanations
arguments
critical examplescounter examples
EXPERT CAN INTRODUCE NEW CONCEPTS (ATTRIBUTES)
Pacient ... Headache Fatigue SoreThroat Appetite Flu
1 ... no no yes normal no
2 ... no yes yes low yes
3 ... yes yes no low yes
4 ... no no no normal no
... ... ... ... ... ... ...
FluSymptoms
no
yes
yes
no
Possible rule with the new attribute:
IF Temperature > Normal AND FluSymptoms = yes
THEN Flu = yes
...
EXPERT CAN CORRECT CLASSIFICATION OF LEARNING EXAMPLE
... Headache Fatigue SoreThroat Appetite FluSymptoms Flu
... no no no normal no yes
Pacient Temperature Vaccination Coughing ...
37 normal no no ...
The question to the expert:
„What is the reason for Pacient 37 having the
flu?“
Expert corrects the classification of Pacient 37:
„Pacient 37 doesn‘t have the flu.“
no
KNOWLEDGE ELICITATION WITH ABML
IF ... THEN ...IF ... THEN ......
ABMLargument-based machine learning
inconsistencies in labels
are detected automatically
misclassificated examples are
easily recognized and corrected
arguments
critical examplescounter examples
experts introduce
new attributes
human-understandable models
suitable for teaching
IF ... THEN ...IF ... THEN ......
ABMLargument-based machine learning
arguments
critical examplescounter examples
How to use ABML in educational setting?
INTRODUCING STUDENTS
THE OUTER LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Student explains the example
Return to step 1
THE OUTER LOOP & THE INNER LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Student explains the example
Return to step 1 Step 3a: Explaining a critical example (in a natural language)
Step 3b: Adding arguments to the example
Step 3c: Discovering counter examples
Step 3d: Improving arguments with counter examples
Return to step 3c if counter example found
USING TEACHER‘S ATTRIBUTES!
ARGUING TO LEARN
Argumentation involves elaboration, reasoning, and reflection. These activities have been shown to contribute to deeper conceptual learning (Bransford, Brown, & Cocking, 1999)
Participating in argumentation helps students learn about argumentative structures (Kuhn, 2001)
A NEW PARADIGM
arguing to learn
with
argument-based machine learning
Let‘s see how this works in practice
BASIC OR ADVANCED?
„basic“
„advanced“
LEARNING DATA SET
121 solutions of 62 different exercises
teacher labeled each solution
as „basic“ or „advanced“
learn data: 91 examples
test data: 30 examples
KNOWLEDGE ELICITATION FROM THE TEACHER
1. relevant description language: new attributes
2. consistently labeled examples
TEACHER‘S GOALS:
RESULTS OF KNOWLEDGE ELICITATION FROM THE TEACHER
9 iterations
9 new attributes
9 rules
only 1 out of 5 initial attributes remained
A STUDENT-COMPUTER INTERACTIVE LEARNING SESSION
STUDENT‘S TASK
obtain rules for distinguishing „basic“ and „advanced“ solutions rules must consist of attributes in teacher‘s final model
use teacher‘s descriptive language
A STUDENT-COMPUTER INTERACTIVE LEARNING SESSION
RECOMMENDATIONS TO THE STUDENT
use the most important features for explanations
use the smallest possible number of features in a single argument
try not to repeat the same arguments
THE FIRST „CRITICAL“ EXAMPLE
The question to the student:
„Why is this solution advanced?“
Student‘s argument:
„Because function zip is present and the number of rows is low.“
Solution = advanced BECAUSE Zip = True AND cRows = low
COUNTER EXAMPLE
IF Zip = True AND cRows = low THEN Solution = advanced
The rule is inconsistent with data! Student is presented with counter example:
„Compare these two solutions. Why is the second solution a basic one?“
IMPROVING ARGUMENT
IF Zip = True AND cRows = low AND LiCom = True
THEN Solution = advanced
Student‘s extended argument:
„Because function zip is present, the number of rows is low,
and a list comprehension occurs. “
no more counter examples next iteration
AT THE END OF THE INTERACTIVE SESSION
5 iterations
half an hour
90% accuracy on (previously unseen) testing data
several suggestions of new descriptive features
ASSESMENT
Results: experiment with 7 students
7.1 iterations
87.1% classification accuracy of obtained rule model
86.7% correctly „manually“ classified (previously unseen) examples
very positive qualitative feedback from the students
CONCLUSIONS
New paradigm:
arguing to learn with argument-based machine learning
Future work:
• applications in several domains
• assessing argument‘s quality for improved immediate feedback
• goal-oriented extension (see our ITS 2012 paper)
QUESTIONS & DISCUSSION
Thank you!
slides: ailab.si/matej
enabling arguing to learn
top related