an intelligent discussion-bot for guiding student interactions in threaded...
TRANSCRIPT
Kim et al.PedDiscourse
An Intelligent Discussion-Botfor Guiding Student Interactions in
Threaded Discussions
Jihie Kim Erin Shaw
Grace ChernDonghui Feng
University of Southern CaliforniaInformation Sciences Institute
?
Kim et al.PedDiscourse
Outline
• Discussion-Bot Framework• Modeling Student Interactions in On-Line
Discussions• Modeling Student Interactions with Speech Act
Classifiers• Current Results• Summary and Future Work
Kim et al.PedDiscourse
Courses InvolvedPast• Five semesters of Undergraduate CS Operating System, USC• Two semesters of Graduate CS Advanced Operating Systems, USC• One semester of Psychology of Women course at the University of
Massachusetts• One semester of Engineering Negotiation for Collaborative Product
Development, USC
Ongoing• Undergraduate CS Operating System, USC• Graduate Security Systems, USC • Formal Languages and Automata, UC Irvine • Undergraduate CS Operating System, Michigan Technological Univ.
~500 past students, ~150 current studentsOver 7000 messages
Kim et al.PedDiscourse
Discussion-Bot Framework
• Modeling and assessing student interactions in on-line discussions
• Handling many student queries• Guiding/scaffolding student interactions
Kim et al.PedDiscourse
Discussion-Bot: Responding to Student Queries
Discussion-bot refers questioner to mostsimilar discussion
or document segment(as an explanation)
(Feng, Shaw, Kim, Hovy IUI-2006)
Kim et al.PedDiscourse
Modeling Discussion
Individual Messages
Response/Replies
Discussion threads
. . .
Kim et al.PedDiscourse
An Example Discussion Thread
S1
S2
S1 I am still confused. I understand it is in the same address space as the parent process, where do we allocate the 8 pages of mem for it? And how do we keep track of .....? … I am sure it is a simple concept that I am just missing.
S3
read the student documentation for the Fork syscall
…
…
The Professor gave us 2 methods for forking threads from the main program. One was ....... The other was to ......... When you fork a thread where does it get created and take its 8 pages from? Do you have to calculate ......? If so how? Where does it store its PCReg .......? Any suggestions would be helpful.
If you use the first implementation...., then you'll have a hardlimit on the number of threads....If you use the second implementation, you need to....Either way, you'll need to implement the AddrSpace::NewStack() function and make sure that there is memory available.
…
Highly incoherent and noisy!
Kim et al.PedDiscourse
Modeling and Assessing Student Interactions
• Contribution content• Topic of the discussion, topic coherence• Quality of the content (e.g. technical term uses)
• Role of each participant and his/her contributione.g. person who asks many questions on a particular topic
• Interaction patterns in threadse.g. long vs. short discussionse.g. threads that reach an agreement on a topic versus threads
that have unanswered queries e.g. effect of instructor intervention
• Interaction changes over timee.g. topic changes over a semester
Kim et al.PedDiscourse
Thread Lengths
data from an undergraduate CS Course
0
100
200
300
400
500
600# of threads
1 3 5 7 9 11 13 15 18 20 31 # of posts
data from a graduate CS Course
0
2
4
6
8
10
12
14
16
18
1 2 3 4 5 6 8 9 10 12 16
# of threads
# of posts
• Discussion threads are often very short, many consisting of only one or two messages
• Student jump into programming details without understanding what is to be programmed or related technical concepts
• TA and instructors are not always available to fully guide interactions
Kim et al.PedDiscourse
Discussion Topic Analysis
• Automatically classify discussion threads topics and model topic shifts within each thread.
T:4830
T:4711
T:4716
0
10
20
30
40
50
60
70
Period 1 Period 2 Period 3 Period 4 Period 5 Period 6
Tim e
Num
ber o
f Mes
sage
s 6
5
4
3
2
1
Topic Shifts within Threads Topic Distribution over Time
(Feng, Kim, Shaw, Hovy, AAAI-2006)
Kim et al.PedDiscourse
Modeling Interactions: Speech Act Categories
Inspired by(Austin, 1962;Searle, 1969)
Give advice or suggest a solutionSuggestSUG
Answer with a short phrase or few wordsSimple AnswerSANS
Ask question about a specific problemQuestionQUES
Object to an argument or suggestionObjectOBJ
Elaborate on a previous argument or questionElaborateELAB
Support an argument or suggestionSupportSUP
Criticize an argumentCriticizeCRT
Correct a wrong answer or solutionCorrectCORR
Praise an argument or suggestionComplimentCOMP
Command or announceCommandANNO
Give answer requiring a full description of procedures, reasons, etc.Complex AnswerCANS
Confirm or acknowledgeAcknowledgeACK
DescriptionNameSpeech Act
(Feng, Shaw, Kim, Hovy, HLT/NAACL 2006)
Kim et al.PedDiscourse
AcknowledgeACK
ComplementCOMP
CorrectCORR
ObjectOBJ
SuggestSUG
ElaborateELAB
SupportSUP
CriticizeCRT
Simple AnswerSANS
Complex AnswerCANS
AnnouncementANNO
QuestionQUES
NameCode 1
ACK/SUP
CORR/OBJ
ELAB
ANS/SUG
ANNO
QUES
Code 3
NEG
NEUT
POS
Code 2
Code 1
Code 2Code 3
% agreement: 81Kappa: 0.70
% agreement: 63Kappa: 0.54
% agreement: 92Kappa: 0.58
Speech Act Categories Explored
Observed agreemnt - Chance agreemntKappa = ---------------------------------------------------------
Total observed - Chance agreemnt
Kim et al.PedDiscourse
Speech Acts in a Discussion Thread
ANS-SUG
ANS-SUG/QUES
S1
S2
S1 I am still confused. I understand it is in the same address space as the parent process, where do we allocate the 8 pages of mem for it? And how do we keep track of .....? … I am sure it is a simple concept that I am just missing.
S3
ANS-SUG
read the student documentation for the Fork syscall
…
…
The Professor gave us 2 methods for forking threads from the main program. One was ....... The other was to ......... When you fork a thread where does it get created and take its 8 pages from? Do you have to calculate ......? If so how? Where does it store its PCReg .......? Any suggestions would be helpful.
If you use the first implementation...., then you'll have a hardlimit on the number of threads....If you use the second implementation, you need to....Either way, you'll need to implement the AddrSpace::NewStack() function and make sure that there is memory available.
QUES
Kim et al.PedDiscourse
Statistics of Speech Acts
29.2A question about a problem, including question about a previous messageQUES
8.1An elaboration (of a previous message) or description, includingelaboration of a question or an answerELAB
9.7A correction or objection (or complaint) to/on a previous messageCORR-OBJ
37.8A simple or complex answer to a previous question. Suggestion oradviceANS-SUG
6.7Information, Command or AnnouncementINFORM
8.5An acknowledgement, compliment or support in response to a previous message
ACK-SUP-COMP
%DescriptionSpeech Act
Kim et al.PedDiscourse
Automatic SA Classifiers
[categ_person] am a [tech_term]
do [categ_person] have tolook at the [tech_term]in the same [tech_term]
look at thefor example ,. [categ_person] shouldlet [me/him/her/us] knownot seem to
look at[or/and] doseems likein [tech_term]stated in
yesam
helpsbut
depends
Answer Classifier(AC)
do [categ_person] have todo [categ_person] need to[tech_term] [tech_term]
[tech..] ?is there a betterdoes this mean that
[categ_wh] should
[categ_person][categ_person] was
wondering[or/and] do [categ_person]is there a [tech_term] ?
do [categ_person][tech_term] ?can [categ_person]is there? thanks
?[categ_wh
]willdo
confused
Question Classifier (QC)
4-grams3-grams2-grams1-gramCategory
• Cleaning/preprocessing/transformation of raw data• N-gram features and Linear SVM
• Accuracy: QC – 88% and AC – 73%
Kim et al.PedDiscourse
Thread Classification with SA Classifiers
• 70-75% of the predictions from the system were consistent with human answers (Ravi & Kim, AIED 2007)
QC=yesAC=no
QC=yesAC=no
M1
QC = yesAC = no
M2QC=noAC=yes
QC=noAC=yes
M2
M1
M3
QC=yesAC=no
M1
QC=noAC=yes
QC=yesAC=no
QC=noAC=yes
M2
QC=yesAC=no
M1
QC=noAC=yes
QC=yesAC=no
M2
M3
M3
M4
1) whether the given thread contains questions2) whether the questions were answered or not.
Kim et al.PedDiscourse
Related Work• Email Speech Act analysis (Carvalho and Cohen 2005)• Dialogue analysis for intelligent tutoring systems (Graesser et al.,
2001)• Dialogue act analysis, surface cues (Samuel 2000; Hirschberg and
Litman 1993)• Topic analysis (Joachims, 1997; Liu et al., 2004; Yang et al., 2005)• Improving Questions Answering with Speech Act Classifiers (Feng,
Shaw, Kim, Hovy HLT-NAACL 2006)• Thread summarization (Zhou and Hovy 2005)• Predicting the likelihood of a message receiving a reply (Arguello et
al., 2006)• Computer supported collaborative argumentation (Shum 2000)• Collaborative interaction in learning (Soller and Lesgold 2003)
Kim et al.PedDiscourse
Summary• Modeling and Guiding Student Interactions in On-
Line Discussions• Modeling student interactions with SA classifiers• Finding discussion threads that may need instruction
attention
• Ongoing Work• Classifiers for other speech act types• Integration of interaction modeling and question
answering: when to intervene• Developing scaffolding techniques