Kingston University

Creating and Editing Mathematical Content using Natural Language

CommandsA Progress Report of the TalkMaths

Project

Eckhard Pflügel

Faculty of Science, Engineering and Computing

Kingston University

Kingston University

Overview

• Motivation and Objective• Existing Solutions• The Technical Challenges• Our Answer: TalkMaths• Current Progress• Future Steps• Conclusion

Introduction

• Our Motivation:– Not everyone can use a computer with

traditional input devices– Mathematical content is notoriously difficult to

be accessed with alternative technologies

• Our Objective: – To carry out research and development

that will lead to the creation of a user-friendly tool/system, for accessing mathematical content using speech

Kingston University

Speech Recognition Technology

• Continuous speech recognition has been around for a while

• Recently, experienced tremendous improvements and attraction in the media (iPhone, Google)

• Two different architectures: client-based, server-based

• Why would people want to use it?– Out of commodity?– Out of necessity!

Kingston University

Desktop-based Speech Recognition Tools

• Currently, most reliable (commercial) tool available is Dragon NaturallySpeaking (DNS)

• An alternative is Windows Speech Recognition (free since Vista, much improved in Windows 7)

• An open source (free) solution exists –Sphinx

• Other solutions might appear in the near future

Kingston University

Dragon NaturallySpeaking

• Popular commercial speech recognition product

• Excellent continuous speech recognition rates

• Effectively allows for hands-free creation and editing of text

• Additional features:– Commands for typical computer-based tasks– Supports creation of (application-specific or

global) macros

Kingston University

Dragon NaturallySpeaking – Drawbacks

• No support for mathematics (all symbols need to be spelled out)

• Does not integrate in any mathematical editor• Cannot “translate” speech input into specific

mathematical markup (MathML, LaTeX)• Development of more sophisticated macros

needs (expensive) SDK• Anecdotal evidence of mediocre user support

Kingston University

Standards for Spoken Mathematics

• Motivated by different contexts:– Dictating to other human beings– Input for computer systems (parsing)– Audio output for text-to-speech systems

• Probably first documented source: Chang’s booklet [1]

• Fateman [2,3] gives fairly detailed rules for spoken mathematics

• Other approach: [4]• Raman [5] is motivated by TTS• Our contribution, based on [2,3]: Wigmore [6]

Kingston University

Existing Tools and Systems

• MathTalk– Collection of Dragon NaturallySpeaking macros– Only usable as input for specific computational maths

interface– Commercial product (expensive)– Limited scope

• Maths Speak & Write– Uses Windows speech technology– Desktop application with GUI– Supports multi-modal input– Research Project– Mainly experimental relevance

Kingston University

Related Tool

• Speed– Goal: programming by voice– Speech-plugin for Eclipse IDE– Uses Java interface for DNS– Speech input with keyboard and mouse– Exports spoken version of Java– This includes navigation– Also research project

• Overall conclusion: none of the existing tools are fit for purpose.

Kingston University

Kingston University

Technical Challenges

• Research aspect: Need tools and techniques from natural language processing, compiler construction and HCI

• Standards for spoken maths– Need flexible and powerful grammar– Have to deal with ambiguity• Parsing Algorithms – Difficulties: incomplete or incorrect input• Editing Paradigms– Need novel strategies for speech-driven UIs

Kingston University

TalkMaths Background

• Use speech recognition for my own work• Frequently needed specialist tasks:

enter/modify mathematical equations• Could use Equation Editor combined with

DNS macros• More problematic if using LaTeX • Idea: write more sophisticated commands• Turned this into research project

Our Answer: TalkMaths

• Web-based User Interface/System• Separate the application from speech-

front-end• Devise special class of speech commands

(“speech templates”)• Insight: spoken mathematics can and

should be process similarly to spoken structured content (i.e. markup languages, programming code)

Kingston University

TalkMaths UI

Kingston University

TalkMaths.org Web UI

Kingston University

http://talkmaths.org/editor.php

TalkMaths.org Web UI – Future Version

Kingston University

Kingston University

How can TalkMaths Help with Accessibility of Mathematics?

• TalkMaths aims at people for whom it is– Difficult to use keyboard/mouse– Difficult to decipher equations on screen

• TalkMaths can help by– Speaking mathematical input– Equation rendering: arbitrary big font sizes– Voice-activated zooming function– Editing mathematical expressions by voice– Planned features:

• Playback of formulae• Importing/exporting existing documents• Maintaining documents on server

Progress Report

• Are developing speech front-end prototype, using Windows speech recognition

• Still have issues with the DNS interface

• About to release new version of parser with more robust error recovery

• New website design in progress

Kingston University

Kingston University

Future Work

• Extending the range of covered mathematics• Allowing for multi-line equations/multiple/embedded

expressions• Potentially, allows sharing of maths input• Improved speech editing (“select and say”)• Higher robustness of code/ease of use/installation• Improved documentation

Ensuring Continuation

• Ph.D. student support (full-time until March 2014, part-time until 2017)

• Industrial funding?

• EU funding?

• Domain/hosting not that expensive

Kingston University

Kingston University

Conclusion

• TalkMaths seems to be a novel application

• TalkMaths can be very helpful for members of Higher Education with computer access problems

• The current prototype will improve significantly over the next years

• Keep up to date: www.TalkMaths.org

Acknowledgements

• This work would have been impossible without my colleagues and students:– James Denholm-Price– Gordon Hunter– (and others)

• Funded by – EPSRC Doctoral Training Award– MSOR Mini Project grant

Kingston University

References

1. Lawrence A. Chang. Handbook for spoken mathematics (Larry's speakeasy). Lawrence Livermore National Laboratory, University of California, USA, 1983.

2. R. Fateman. How can we speak math? University of California at Berkeley, 2009.

3. R. Fateman. 2-D Display of Incomplete Mathematical Expressions. University of California at Berkeley, 2006.

4. Cameron Elliott and Je A. Bilmes. Computer based mathematics using continuous speech recognition. CHI 2007 Workshop on Striking a C[h]ord: Vocal Interaction in Assistive Technologies, Games and More, 2007.

5. T. V. Raman. Audio system for technical readings. Springer Verlag, Berlin, 1998.

6. A. Wigmore. Speech-Based Creation and Editing of Mathematical Content. Ph.D. Thesis, Kingston University, U.K., 2011.

Kingston University