Using Mobile Phones To Write In AirChris Coykendall – ODU CS495

Introduction Researchers from Duke University and University

of Illinois collaborated to develop a prototype system called PhonePoint Pen on the Nokia N95 platform

The PhonePoint Pen system aims to allow humans to write messages or diagrams in the air by using a phone as a pen.

The Nokia N95 deviceSource: Engadget

Some Motivations Make “post-it” notes on the fly Assistive communications for disabled persons

and/or the elderly Sketching complex ideas, equations and non-

standard information Emergency operations and first responders

So Why A New System? User experience on existing mobile technologies

leaves much to be desired Mobile keyboards difficult to use for motor

impaired individuals Voice recording technologies are cumbersome to

search through (and speech-to-text software can be hit or miss)

Primary Goals To explore the viability of using mobile phone

accelerometers to write in the air. To understand and overcome limitations with

character recognition. To develop a prototype on the Nokia N95 platform

and perform a test study.

Challenges Lack of a gyroscope

Many systems such as the Wii, Kinect and others are more resourceful in hardware and have a gyroscope for filtering rotation. (This study was done shortly before many phones with gyroscopes, such as the iPhone 4, were released.)

In this study, users were constrained to using the phone by holding it in a steady, non-rotating grip like a pen. They were also asked to write in slow, deliberate motions and make each character around 12 inches square.

Challenges Suppressing background vibration “noise”

Jitter is caused by natural hand vibrations and the accelerometer error itself.

The researchers approached this issue by implementing a moving ∆velocity over last 7 readings. Any acceleration samples < .05m/s2 were ignored.

Challenges Computing displacement of device

The physical position in the air is necessary to know the size of the characters and their relative positions.

This was overcome by looking at the velocities measured suppressing the background vibrations. If n readings were measured as all background noise, it is likely a pause and the velocity is reset to zero.

(From source article)

Challenges Multi-stroke characters and transitions

Letters such as the letter ‘A’ require two distinct strokes to write. The letter ‘B’ looks remarkably similar to writing ‘13’. With no frame of reference for position, this is difficult to accommodate.

The study exploited user natural motion of “picking” up the pen on the Z-axis.

They also relied on a combination of heuristics such as delimiters and grammar algorithms they developed to predict what the next stroke was intended to be.

(From source article)

Implementation Prototypedon Nokia N95 phone platform. A server-side implementation was developed in

MATLAB, which they were able to use basic libraries for the signal processing.

The on-phone processing version was implemented in Python, but stripped-down to only one character at a time and simpler signal processing methods.

User Evaluation Tests were conducted with 10 average users to write the

English alphabet: mainly CS and Engineering students. 4 subjects were trained on the system, while the other 6

“novice” subjects had writing less than ten characters. Another 5 individuals took part in 8-character tests at Duke

University Hospital under supervision. These patients either suffered from cognitive disorders or motion impairments.

Result Highlights Human readability accuracy (HRA) of the characters was

around 83-85% for the student test subjects. Character recognition accuracy (CRA) was 91.9% for trained users and 78.2% for novices

The hospital patient 8-character HRA test results were:User 1 – 1/8User 2 – 1/8User 3 – 1/8User 4 – 5/8User 5 – 0/8 (could not operate button)

(From source article)

Patient Barriers The PhonePoint Pen required a button to be

pressed to begin and end the writing. Shoulder, elbow, and wrist coordination for large

12in characters can be difficult. Familiarity with mobile devices. IRB restrictions make it difficult to exhaustively test

patient use.

Improvements To Be Made Faster writing (only 3.02s per letter on average) Writing longer words/drawings Cursive handwriting Writing while moving More diverse test subjects (CS/Enginering majors

likely more technologically-inclined.) More advanced algorithms (Bayesian Networks,

Hdden Markov Models, etc.)

Conclusions and Related Research PhonePoint Pen compares well to other research

in this area considering its limited processing power and sensor hardware (just the accelerometer)

Air-gestures with 3D accelerometers (uWave) Vision-based gestures (Microsoft write-in-the-air) Stylus-based sketch recognition (SketchREAD) Wiimote, Logitech Air-Mouse, Nokia NiiMe Smart Pen and SmartQuill

Source S. Agarwal

et. al., "Using Mobile Phones to Write in Air", MobiSys'11