Natural Input On Mobile PC Systems

Bodin DresevicDevelopment ManagerTablet PC Group, Mobile PCs Divisionbodind @ microsoft.comMicrosoft Corporation

Session OutlineSession Outline

Applications of Natural Input

Speech Input

Pen Input

Touch Input

Summary

Call to ActionCall to Action

Microsoft will continue investing in making Natural Input more ubiquitous and easier to use

OEMs and IHVs should take advantage of the software innovations in Natural Input

It takes both to bring the best experience to our customers and to succeed in today’s competitive marketplace!

Applications of Natural InputApplications of Natural Input

Text InputDictation & handwriting recognition

Natural annotation and capture Ink or audio annotations on text

Ink or audio notes, recordings or emails

Natural Navigation and ActivationSpeech commands

Pen & Touch used as mouse and for gestures

AuthenticationBiometric: signature, fingerprint etc.

Hardware Buttons to provide access tosystem features

Ctrl-Alt-Delete, change resolution, switch orientation, scroll wheel etc

SpeechSpeech

User benefits

Software support in Windows

Hardware considerations

Recommended solutions

Call to Action

Speech – End User BenefitsSpeech – End User Benefits

Speech Input offers distinct advantages in certain scenarios / situations

For users with repetitive stress injuries

For devices without a keyboard: slate Tablet PCs, PDAs, smart phones

Solutions exist for vision impaired

Audio capture: voice as voice record a meeting, attach voice note to email message, audio sticky

Text to SpeechAccessibility applications: only US and CHS engines offeredin Windows codenamed “Longhorn”

Speech Input or Text to Speech on the serverMultiple applications in telephony, e.g. caller ID

Software Support for Speech in Windows XPSoftware Support for Speech in Windows XP

Support built in Tablet PC Edition 2005 (Windows XP SP2)

Speech input integrated in Tablet Input Panel

Mode switch required between Dictation and Command & Control

Accuracy in Commanding mode superior to that in Dictation mode

Training the speech recognition engine highly recommended

Planned Software Support for SpeechPlanned Software Support for Speechin Longhornin Longhorn

Focused improvement on user experience

Dedicated UI for speech input on desktop

Integrated Dictation and Commanding experience

On Tablet PC the new UI lives side by side with Pen TIP

Engine training is being replaced by interactive tutorial

25 minutes in training and learning speech commands greatly improves overall experience

Accuracy of the speech engines is greatly improved

More languages: US, UK, German, French, Spanish, Chinese Traditional, Chinese Simplified and Japanese

Software support for microphone array technology

Audio Hardware ConsiderationsAudio Hardware Considerations

Command & Control: constrained recognition outputWorks well with almost any reasonable microphone

Dictation requires high-quality audio stream 16KHz / 16-bit, better than 20db signal-to-noise ratio

Usually headset is recommendedAnalog or USB microphones meet the dictation requirement

Bluetooth 2.0 headsets are likely to meet the dictation requirement

Microphone arrays can meet dictation requirementsFor < 2 feet distance and normal noise conditions comparable to close-talk microphone

Use three or more in non linear arrangement at uneven distances

Inexpensive: $5 manufacturing cost (processing in software)

Today’s hardware limited with stereo input paths(2 input channels)

Speech Input DemoSpeech Input Demo

Tablet PC running Longhorn

Email dictation with correction and editing commands

Deleting word, making it bold or italic, etc.

Call to ActionCall to Action

Improve the sound capturing quality: integrate microphone arrays in your devices

For more info please attend the following session:

Microphone Array Support in Longhorn

Pen InputPen Input

User benefits

Software support in Windows

Hardware Considerations

Recommended solutions

Call to Action

Pen Input – User BenefitsPen Input – User Benefits

Handwriting capture as Ink for note takingand drawing

Handwriting recognition for text entry through TIP

Simple Pen gestures for basic navigationTap, double-tap, drag

press & hold for right-click

Gestures for advanced navigation shortcutsFlicks for scrolling up and down, browser backand forward

Additional flicks for copy, paste, delete, undo

Scratch out in TIP, etc.

Pen Input DemosPen Input Demos

Note taking in OneNoteink “flows” from the pen in real time

Search ink notes, apply bullets to text outline

Text Input via recognition: Longhorn TIPEdge targeting: bring TIP from the edge of the screen

Auto-complete in the Run dialog

Auto-complete in IE address bar

Auto-complete stock quotes in IE

Scratch out improvements: x gesture, horizontal line over a word, old scratch out

For password input fields TIP switches to soft keyboard and back to lined mode for user name (Hotmail)

Pen Demo - Shell ImprovementsPen Demo - Shell Improvements

New visual feedback for basic Pen gesturesTap: a drop in the water, concentric circles

Barrel Button: blue circle around cursor that flashes upon leaving the surface

Press and Hold: circle grows and flashes whenleaving surface

Cursors switch between new pen cursor and traditional mouse cursor

Checkboxes allow us to do multiple selection without doing ctl + (multiple mouse clicks)

Pen Demos – FlicksPen Demos – Flicks

Flicks can be configured in 2 modes:Basic: N = page down, S = page up, E = forward, W = back

Advanced: NE = paste, SE = delete, NW = copy, SW = undo

Control Panel applet allows the flicks to be remapped, their sensitivity adjusted

Notepad demo: open new document, delete it (SE) and undo (SW)

Outlook demos: Flick up and down through a long email

Flick emails out of the inbox (SE), occasional undo (SW)

Copy (NW) / paste (NE) emails between two folders

Flick back / forth between Inbox and Calendar

Pen Input – Software Support in WindowsPen Input – Software Support in Windows

In Windows XPWindows XP Tablet PC Edition 2005

is a superset of Windows XP SP2 Professional

Contains support for pen input, inking and handwriting recognition

In Longhorn, planning to take Tablet functionality mainstream

Pen Input and inking available on hardware units with digitizer

Handwriting recognition and ink viewing available even on desktop

Improved end user and developer experience

Pen Input – Hardware ConsiderationsPen Input – Hardware Considerations

For best recognition accuracy use a high-resolutionEM digitizer

Recommended resolution: > 9K x 6K, sampling rate > 133Hz

User’s top pain points when using penAccurate targeting of small objects: check boxes, icons or buttons

Accidental right click activation when stylus button is depressed

Today’s solutions require a user to calibrate the digitizerThe calibration utility is hard to discover

Calibration has to be done in landscape and portrait modes

Calibration does not remove the problem entirely

Calibration is required becauseDigitizer’s EM field is non-linear especially near the edges

Parallax caused by:the high position of the coil inside the stylus

Glass thickness and optical properties

Pen Input – Call to ActionPen Input – Call to Action

Continue innovating and improving digitizers built into Tablet PC units

Suggested ideas to reduce parallax and need for end user calibration

Better OEM calibration

“Tuck” the digitizer under the screen edges

Lower the position of the coil in the stylus

Or look for alternative technology to enable Hover and Stylus tracking above the digitizer surface

Use thinner glass with better optical properties

Recommendations (con’t)Recommendations (con’t)

Provide innovative stylus designs – remove possibility of accidental button activation

Adjust screen surface smoothness to provide paper-like writing experience

The Input experience is inextricably tied to the good screen readability

Build great LCD screens…With wide viewing angles

That enable outdoor readability – use low reflection glass and have auto-adjustable display brightness

TouchTouch

User benefits

Software support in Windows

Hardware Considerations

Recommended solutions

Call to Action

Touch – End User BenefitsTouch – End User Benefits

In Longhorn we are planning to add Touch as a new mode for Navigation and Control

Great complement to Pen and Speech Input

All navigation with Pen also works with Touch but is more efficient and intuitive - no pen required

Mouse operations: Tap and Double tap with finger

Advanced gestures: Finger flicks

Longhorn plans to enable easy precision targeting with finger

Stylus still preferred for ink input

Touch Input DemoTouch Input DemoUI is Under Construction – It May Change!UI is Under Construction – It May Change!

Demo on retrofitted Tablet PC with N-Trig digitizer

Left click/double click on larger targets is performed simply by tapping on the target

For small targets Longhorn provides intuitive Targeting Ring UI

To right click on a targeteither depress the Right Click button on the Ring when the pointer is above the desired target

Or simply press & hold the target with a finger

Finger visual metaphors are very similar to the pen ones

Touch is disabled when Stylus is in proximity (<1cm)

Palm rejection – important when pen is not in proximityBlunt, low-pressure contact is ignored

Existing Pen and Touch Digitizer TechnologiesExisting Pen and Touch Digitizer Technologies

Digitizer Type

EM(Tablet V1,V2) Capacitive Resistive

Input Device

“pointy stick” Gloved finger

Finger EM stylus “”

Longhorn Software Architecture for TouchLonghorn Software Architecture for Touch

For touch to work correctly on Longhorn system

HID Layer (HIDCLASS.SYS)

EM digitizer mini-driver Touch digitizer mini-driverEM-Touch digitizer mini-driver

If the unit has an integrated dual mode digitizer (such as N-Trig), HID layer still needs to enumerate Touch and EM digitizers as two virtual independent HID devices

If a unit has two independent digitizer devices (Touch screen and EM digitizer), each one needs an independent HID mini-driver (could be the same but listed twice in HID layer)

Software Support (con’t) OEM / IHV GuidelinesSoftware Support (con’t) OEM / IHV Guidelines

Technical Requirements for Touch DigitizerHID usage page is 0X0D (misc. devices) and the Usage is 4 (touch screen)

Packet Properties – still evolving requirementsRequired: x, y

Optional: area of contact, pressure

Highly recommended: confidence (that this is a finger as opposed to palm)

Confidence is very important for correct mode auto-switching decisions.

Longhorn does the final arbitration when to turn Touch on or off

The OS can take advantage of the context knowledge (the active application, buttons, scroll bar etc positions)

Start button can always be touched even when pen is close

Application Behavior and TouchApplication Behavior and Touch

Current applications – pre-LonghornPen aware: treat Finger and Stylus as same – show OneNote

Pen non-aware (ordinary windows apps): Touch generates mouse messages and facilitates navigation

New applications can take advantage of TouchTablet Platform APIs (Ink Collector and RTS) are extended:

New method to inquire about the types of digitizer devices Stylus, Touch or mouse

propagate digitizer information (x,y, area) with pen events

Enables building of modal apps, Stylus vs. TouchDemo: app that uses stylus for inking and finger for erasing

Avalon Inking Platform supports Touch at the same level as COM / Winforms Tablet PC Platform

Shipping Digitizer Mini-Driver with LonghornShipping Digitizer Mini-Driver with Longhorn

Digitizer device manufacturer has a responsibility to provide the HID mini-driver(s) for its device(s)

Microsoft is building Windows Logo Program Test Suite for (EM and touch) digitizer mini-drivers

Examples of what the Test Suite may checkScreen/digitizer bounds aligned

Linearity / calibration (edges and hover)

Noise: Hold the pen still and ensure x, y change minimally

Minimal lag / distance between pen and cursor

Sample rate and digitizer resolutions sufficient

Button state permutations register correctly

Basic system gestures tap, double tap, and press & hold work

To ship “in the box” must pass the Test Suite

The Test Suite spec not available yet, stay tuned

Call to ActionCall to Action

Add Touch support to your Tablet PC units and greatly enhance user’s navigation experience

Build in a single integrated dual mode digitizer into your Tablet PC

Or build a traditional Touch digitizer and a traditional EM digitizer into your Tablet PC unit

In either case Longhorn will do the arbitration for you – so long as the touch mini-driver meetsthe specifications

Community ResourcesCommunity Resources

Windows Hardware & Driver Central (WHDC)www.microsoft.com/whdc/default.mspx

Technical Communitieswww.microsoft.com/communities/products/default.mspx

Non-Microsoft Community Siteswww.microsoft.com/communities/related/default.mspx

Microsoft Public Newsgroupswww.microsoft.com/communities/newsgroups

Technical Chats and Webcastswww.microsoft.com/communities/chats/default.mspx

www.microsoft.com/webcasts

Microsoft Blogswww.microsoft.com/communities/blogs

Additional ResourcesAdditional Resources

Related SessionsMicrophone Array Support in Windows LonghornFull Client Usability for the Highly Mobile Market

Building digitizer mini-driver for LonghornWindows Logo Program Test Suite for digitizers

Email: burtp @ microsoft.com

N-Trig Web site: http://www.n-trig.comTouch support in Tablet API for Longhorn

Longhorn Platform SDK (at the moment Touch not represented yet)http://msdn.microsoft.com/longhorn

Support for Tablet functionality and Touch in Avalon API (WAP1 spec)

http://www.microsoft.com/downloads/details.aspx?FamilyID=c8f904e1-b4ca-402b-accf-aaa2bd60da74&displaylang=enhttp://msdn.microsoft.com/longhorn/understanding/pillars/avalon/default.aspx

Summary Summary

Continue innovating and improving hardware that enables Natural Input

Microsoft is working on complementing your advances with software advances in the ease of use of Natural Input and and core improvements in the recognition engines

Only together we can bring the best Natural Input experience to our customers

© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.