evaluation of multimodal input for entering mathematical equations on the computer lisa anthony, jie...
Post on 22-Dec-2015
219 views
TRANSCRIPT
Evaluation of Multimodal Input for Entering Mathematical
Equations on the Computer
Lisa Anthony,Jie Yang, Kenneth R. Koedinger
Human-Computer Interaction Institute
Carnegie Mellon University, Pittsburgh, PA
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Programming constructs or special syntax
Linearization of mental representation
Difficult to revise structure in template-based editors
Computer-Based Math Tools
MATHEMATICA
MICROSOFT EQUATION EDITOR
Maple V, Mathematica, Matlab; Microsoft Equation Editor, …
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Problems:►Large learning curve►Tied to keyboard and
mouse input
Computer-Based Math Tools
MATHEMATICA
MICROSOFT EQUATION EDITOR
Maple V, Mathematica, Matlab; Microsoft Equation Editor, …
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Notations already exist for paper-based math
Affordance for spatial representation
Especially good for students learning math
Problem: recognition accuracy
Pen-Based Computing
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Solution: Multimodal Input
Increased robustness►Better recognition accuracy with multiple input
streams (Oviatt, 1999; pen gestures + speech)
Consider both repair-only and simultaneous input in both streams
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Motivation for User Study
Literature says typing is faster (Brown, 1988) ►Compared inputting paragraphs of English text►Math is different domain
Little evaluation done of pen-based equation input►Systems constrained by recognition accuracy
(Smithies et al, 2001)
Human-Computer Interaction Institute Anthony, Yang, Koedinger
User Study: DesignInput method
Equation complexity► Number of characters in equation► Number of “complex” symbols (e.g., √ and ∑)
Keyboard-and-mouse Used keyboard and mouse with Microsoft Equation Editor (MSEE).
Handwriting-only Wrote in Windows Journal program; without recognition.
Speech-only Dictated into microphone; without recognition or visual feedback.
Handwriting-plus-speech
Both spoke and wrote in series or in parallel (user choice); without recognition.
“Recognition” means system tries to interpret user input.
Human-Computer Interaction Institute Anthony, Yang, Koedinger
User Study: Participants
48 paid participants (27 male, 21 female)
Undergraduate/graduate, full-time/part-time students at Carnegie Mellon
Native English speakers only
Most (33 of 48) had no experience with MSEE before study
Human-Computer Interaction Institute Anthony, Yang, Koedinger
User Study: Procedure
Entered math equations on TabletPC►36 equations (7 + 2 practice per condition)►Conditions counterbalanced across participants
Instructions for each condition►No prompting for specific ways of expressing
equations►5 min “explore time” for MSEE
Human-Computer Interaction Institute Anthony, Yang, Koedinger
User Study: Measures
Time per equation
Number of errors per equation (corrected and uncorrected)
User preferences before and after session
Equation entry screen inhandwriting condition.
Human-Computer Interaction Institute Anthony, Yang, Koedinger
User Study: Results
Average time in seconds per equation by condition. Error bars
show 95% confidence interval (CI).
05
101520253035404550
Keyboard-and-mouse
Handwritingonly
Speech only Handwriting-plus-speech
Condition
Tim
e (s
eco
nd
s)
Human-Computer Interaction Institute Anthony, Yang, Koedinger
User Study: Results
0
0.5
1
1.5
2
2.5
Keyboard-and-mouse
Handwritingonly
Speech only Handwriting-plus-speech
Condition
Nu
mb
er o
f E
rro
rs
Mean number of user errors made per equation by condition. Error bars
show 95% CI.
Human-Computer Interaction Institute Anthony, Yang, Koedinger
User Study: Results
00.5
11.5
22.5
33.5
44.5
5
Keyboard-and-mouse
Handwriting only Speech only
Condition
Sco
re (
1 to
5)
Preference questionnaire rankings of each condition on a 5-point Likert scale. Error bars show 95% CI.
Pre-test Post-test
00.5
11.5
22.5
33.5
44.5
5
Keyboard-and-mouse
Handwritingonly
Speech only Handwriting-plus-speech
Condition
Sco
re (
1 to
5)
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Conclusions
Handwriting faster, more efficient, and more enjoyable to novice users than standard keyboard-and-mouse
Handwriting-plus-speech faster and better liked than keyboard-and-mouse
Handwriting-plus-speech not much worse than handwriting alone, so multimodal may be a winner for technology reasons
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Further Analyses
Transcription of spoken input as corpus for generation of language model
Consistency across and within users in handwriting and speech► Ambiguity resolution► Self correction► Pausing and synchronization in multimodal input
Greater variability within speech condition than within handwriting condition
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Further Analyses
Transcription of spoken input as corpus for generation of language model
Consistency across and within users in handwriting and speech► Ambiguity resolution► Self correction► Pausing and synchronization in multimodal input
Greater variability within speech condition than within handwriting condition
(Supported by Oviatt et al, 2005)
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Questions?
Project Webpage:
http://www.cs.cmu.edu/~lanthony/research/multimodal/
Pittsburgh Science of Learning Center:
http://www.learnlab.org/index.php
Human-Computer Interaction Institute Anthony, Yang, Koedinger
ReferencesBrown, C.M.L.: Comparison of Typing and Handwriting in “Two-Finger
Typists.” Proceedings of the Human Factors Society (1988) 381–385.
Oviatt, S.: Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. Proceedings of the CHI Conference (1999) 576–583.
Smithies, S., Novins, K., and Arvo, J.: Equation Entry and Editing via Handwriting and Gesture Recognition. Behaviour and Information Technology 20 (2001) 53–67.
Hausmann, R.G.M. and Chi, M.T.H.: Can a Computer Interface Support Self-explaining? Cognitive Technology 7 (2002) 4–14.
Human-Computer Interaction Institute Anthony, Yang, Koedinger
Other References in PaperAnderson, J.R., Corbett, A.T., Koedinger, K.R., and Pelletier, R.:
Cognitive Tutors: Lessons Learned. The Journal of the Learning Sciences 4 (1995) 167–207.
Blostein, D. and Grbavec, A.: Recognition of Mathematical Notation. In Handbook on Optical Character Recognition and Document Analysis, Wang, P.S.P. and Bunke, H. (eds) (1996) 557–582.
Locke, J.L. and Fehr, F.S.: Subvocalization of Heard or Seen Words Prior to Spoken or Written Recall. American Journal of Psychology 85 (1972) 63–68.
Microsoft.: Microsoft Word User’s Guide Version 6.0 (1993), Microsoft Press.
Sweller, J.: Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12 (1988) 257–285.