Handheld Multimodal InterfacesTechnology, Markets, and

Opportunities

Jordan CohenCTO Voice SignalFebruary 2005

Baltimore, Maryland

Multimodal Communications Thru the Ages

Characteristics of Multimodal Interfaces

Transparent Easy to Learn Easy to use Robust

Flexible More than one input mode Mode Choice Adaptability over Time (Mobile Requirement)

Efficient Accommodate Adverse Conditions Stable Combined Modes often Quicker Superior Error Handling Simplified Language Mode Switching Facilitates Error Recovery Mutual Disambiguation Possible for Simultaneous Inputs

– Sharon Oviatt, “Multimodal Interfaces”, 2002

The First Computer

The BabbageDifference Engine(1832)

25,000 partscost: £17,470

The Mainframe - IBM 3090

Famous Predictions

“I think there is a world market for maybe five computers”

(Thomas Watson, Chairman of IBM, 1943)

The Easy-To-Use PC

Down to the Pocket PC

The Shape of Multimodal Interfaces

Famous Predictions

“This ‘telephone’ has too many shortcomings to be seriously considered as a means of communication. The device is inherently of no value to us.”

Western Union internal memo, 1876.

The Early Car Phone

And There Were Cell Phones

Motorola Made the Phone Portable! Martin Cooper, April 3, 1973 First Public Cell Phone Call (non-car phone)

The Ubiquitous Computing Platform

Digital Cellular Market(Phones Shipped)

1996 1998 2000 2004*

Units 48M 162M 435M 600M* Analog Baseband

Digital Baseband

(DSP + MCU)

PowerManagement

Small Signal RF

PowerRF

data from Texas Instrumentsdata from Texas Instruments* - Projection* - Projection

CellPhone

MP3 players – what are they?

Name comes from type of file (MPEG), audio layer three – MP3

Key to popularity is file compression Some phones have MP3 play back

capability Usually separate units

TODAY’S 3D MOBILE GAME HARDWARE

Things to Think About: Features

Text (SMS) Wireless Internet (3G) Voice Dialing MP3 Ringtones Push-to-talk Games Photo/Video Headsets (Bluetooth) Chargers

A Sign of the Times

“ I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone.”Bjarne Stroustrop (originator of C++)

Enhanced Personal Information Managers Calendars Address Books Application Launchers

Time management tools

Advanced calculators Statistical tools Graphics design tools Audio design tools Thousands of ebooks

www.palmone.com/us/software

CRM Education Finance Government Health care Law Real estate Reference

books SFA U.S. Navy

Business and Professional

Over 23,000 available Palm OS® titles!

Killer app?

Obvious strong sellers: Games, especially casual PIM and calendar tools

Less obvious: Financial services Health care Emergency services Chat and dating services

Average price of mobile applications: $16 on Handango (mostly PDA) $5 on carriers (mostly Java and BREW)

Strategy Analytics Survey (Kerr, 2004)

Evolution of Mobile Services

Basic text messaging

Hi, I’ll see you at 6 PM Regards, Bob

Messaging

Black and white screens,simple graphics

BrowsingBrowsing

Instant connectivity,colour screen,WAP Push

xHTML for WAP and web integration, animated content Ringing tones,

icons,screensavers,business cards

DownloadingDownloading

Downloadable applications,MIDI sounds

Video content(MMS, streaming)

Multimedia messaging with video

Dial-up connections

Accelerated connections,voice and data multitasking

Corporate accessCorporate access

Always-on,performance and securityenhancements

Multimedia messaging with pictures and audio

Games

- Other predictions to year 2006: $3,6 billion and $4,4 billion [21] - Mobile gamers in USA 7M in 2002 (IDC estimate), 2007 112,4M [21]- New mobile devices (big color displays, joystick, voices..)- Different run-time environments: Java key technology [20]- Licenced formats e.g. Codetoys ”Who Wants to be a Millionaire?”- Mobility enabling positioning based and other on-line interactive games - Entertainment in home, bus... Affordable device nearly all (in future)

Mobile gaming revenue prediction [19], Play Away: The future of mobile entertainment (2002)

Smart phones

Smart phones as hybrid products Cell phone features (esp.

voice) PDA features (esp. PIM)

Indirect market entry by incumbents Nokia palmOne Microsoft

Industry ambiguity New suppliers, competitors,

customers Competition from separate

products Cross-industry alliances to

share capabilities

PDA industry

Cell phone industry

Smart phone

PC, PDA, Cell Phone sales (2000-2002)

Source: Assinform / NetConsulting

Millions of Units

128

123

130

11.7

12.8

11

405

390

410

PC's

PDA's

Cell Phones

2000

2001

2002

Operating Systems

Smartphone Windows Powered environment using familiar

Microsoft software Outlook capabilities, Windows Media Player,

infrared beaming, and MSN Messenger Programmable using Microsoft Mobile

Development Toolkit (incl. in The Smartphone Developer Kit)

www.microsoft.com/mobile/smartphone/default.asp

Operating Systems

Symbian OS Version 7.0s

•No user training: Unlike old voice dialing, VAP works right out of the box, and voice-activates every name in the phonebook, no matter how large.

•Multimodal: User can dial by voice, or key press. Dialogs are presented on the display, and played through the earpiece for total eyes free dialing

•Eyes-free, helpful voice guide: Intuitive dialog is easy to use the very first time and doesn’t require looking at the display.

•Listening indicators: Users can see visual (ear icon) and hear (beep) cues when to speak.

•Dial from the entire phonebook: When multiple numbers are stored, the system asks which one to dial.

Voice Activated Phonebook (VAP): Dialing with Discrete Commands

Say a Command:

Settings HelpSettings Help

Digit Dial

Name Dial

Quick Dial

Voice Memo

Phone Book

Browser

Say a Command:Say the Name:

Settings Help

Say the Name:

Settings Help

Which Number?

John Smith

HelpSettings

Which Number?

John Smith

HelpSettings

Home

Mobile

Work

Connecting…

John SmithMobile555-1212

Cancel

Click toStart VSuite

VSuite Core Feature Index

Comments from Time.com

"...Now there's something infinitely better: it's called "speaker-independent" voice recognition, and the best one out by far is from Voice Signal...this million-dollar technology might already be in your phone. Have a look. If not, trade it in, because this stuff is awesome!..."

WILSON ROTHMAN

Time Magazine, “Gadget of the Week”, Feb. 4, 2004

Voice Activated Phonebook (VAP): Dialing with Natural Commands

Say a Command:

Settings HelpSettings Help

Call [number]

Call [name]

Go To [Application]

Voice Message [Name]

Lookup [Name]

Send [Message] to [ ]

Say a Command:Connecting…

John SmithMobile555-1212

Cancel

•Freedom, Flexibility, Convenience: Natural Commands enhance the VSuite 1.x experience by allowing users to accomplish tasks with single, natural voice commands.

•What – Who – Where Paradigm

•What ? Call, Send Message?

•To whom? Person or Number?

•To which device?Mobile? Email? Pager?

•Smart Dialog: When VSuite isn’t sure what you said, it will ask, “Did you say?” allowing for instant correction.

Click toStart VSuite

VSuite Upgrade Index

Voice Activated Message Setup

Say a Command:

Settings HelpSettings Help

Call [number]

Call [name]

Go To [Application]

Voice Message [Name]

Lookup [Name]

Send [Message] to [ ]

Say a Command:

Click toStart VSuite

•Fastest and Simplest Method to Set Up and Address Messages:

•Single key press

•Compatible with any message client and protocol (SMS, EMS, MMS, Email, Instant Messenger,)

•In just a few seconds, the messaging client is open, addressed, and waiting for text input.

•Any name or number in phonebook can be addressed by voice

VSuite Upgrade Index

John Smith

Send ABC

A Messaging Application

Demonstration

Customers? - Nokia prognosis for 3G mobile communication

Monthly income pr. user in euro (1 euro = 1 USD ca.)

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

0

10

20

30

40

50

60

70

80

90

100

Location based services

Commercials

Entertainment

Information servicesPayment transactions

Music and video

Internet surfingDownload from internetChat on internetMultimedia messages

Text messages

Vide conferencing

Normal speech

Fixed subscription fees

Div. telecomm.

Photo messages

Number of people online by language spoken (http://www.glreach.com/globstats/)

British Student Cell Phone Penetration

A quarter of 7 to 10-year-olds own mobile phones, according to a survey of 2,000 children and their parents by the market analysts Mintel. The figures show that the proportion of primary school pupils with mobiles has almost doubled from 13% in 2001. (Telegraph, 28 April 2004)

There are as many cell phone in Britain as there are people 10 years old or more.

96% of 15 to 24-year-olds own a cellphone and cannot function normally without one

Messaging

Short Message Service (SMS) Up to 160 characters of text in length: words

or numbers or an alphanumeric combination Non-text based short messages (for example,

in binary format) are also supported. These are used for ringtones and logos services for instance

SMS is a store and forward service: short messages are not sent directly from sender to recipient, but always via an SMS Center (SMSC)

Short messages can be sent and received simultaneously with GSM voice, Data and Fax calls

Smileys – from the SMS world!

:) Original smiley

:-) Classic smiley

:-( Frown

:@ Ouch!

:-& Tongue-tied

:-# My lips are sealed

~#:-( Bad hair day

(-: Sender is left-handed

{} No comment

<:-) Dumb question

#-) Partied all night

:-$ Put your money where your mouth is

Messaging

Multimedia Message Service (MMS) The data speed in the networks must be at

least 14.4 kbit/s for carrying MMS messages. MMS messages are currently between 30 kB

and 100 kB in size They can contain still images (JPEG, GIF,

WBMP), text, voice (AMR voice), audioclips or videoclips 

MMS transport is done using WAP over GPRS (or CSD) transport. WAP push features are used to deliver the message from server to receiving phone.

British Commentary on Messaging:

Texts overtook phone calls as the favoured means of contacting someone by mobile in 2003. Research found more than eight out of 10 under-25s are more likely to text than call. At New Year revellers smashed the record for the most messages in a day, tapping out 111 million in the first 24 hours of 2004

Nearly a quarter of British teenagers text-message their friends more often than they talk to their families according to a study of more than 400 girls carried out on behalf of the Guide Association. A fifth claim never to talk to parents and siblings.

From www.didyouknow.cd/sms

If the universe did start with a bangwhen God loved and the angels sangone of the sparks that flewchased time to become you

if kisses wer rain id send u showers, if fun was time id send u hrs, if u needed a frnd id send u me!

As i lie on my bed and look at the stars, and the moon, and the sky... i wonder...WHAT HAPPENED TO MY ROOF?!?!?!?

here's my halomade of glassmess with me and i'll kick ur ass

when God gave us friendships he tried to be fair, but when i got you, i got more than my share!

Ek dwaal doeloos verby Jupitor. Waar is jy? Ek verdwaal iewers in die Amasone maar waar is jy? In my drome soek ek jou maar jy is waar jy moet wees... in my hart

From Tech Central Station

A survey of Italian children between the ages of nine and ten found that, incredibly, 56 percent owned cell phones. This was the finding of a study published on 8 July 2003, coordinated by Francesco Pira, a lecturer at the University of Trieste.

New research released by the research group, mobileYouth has highlighted the growing number of young children with mobile phones in Britain . The study conducted across the 5-24 year old market highlights that there are now 400,000 children in the UK under the age of 10 owning a mobile phone, as against only 80,000 in 2000. (Cellular News, 12 Aug, 2003)

From IT Facts (www.itfacts.biz)

Taiwan and Hong Kong lead the world in terms of mobile penetration - there are 110 mobile phones for every 100 inhabitants of Taiwan, and 105.75 mobile phones for every 100 people in Hong Kong. Other countries ranked lower in International Telecommunications Union annual survey, with US reaching 57% mobile penetration by the end of 2004 Dec 18, 04

Gartner said that it expected 20 mln smartphones to ship in 2006 compared to about 13 mln PDAs

More on phone handsets

By 2006, more ‘smart’ phones than PCs.

80% of new phones will be color. 80% of new phones will have

Internet. 74% handsets sold this year will be

GSM. Tip: Use GSM handsets with shared

plans.

Internationalization esp. imp. in mobile market: By 2006, Asia Pacific will have 45% of

mobile subscribers. 433 million handsets this year, 35%

sold in China alone.

Thank You

The End

Jordan Cohen

Jrc@voicesignal.com