1 ken hinckley microsoft research collaborators: mike sinclair, erik hanson, rick szeliski, matt...

1

Ken HinckleyMicrosoft ResearchCollaborators: Mike Sinclair, Erik Hanson, Rick

Szeliski, Matt Conway

Disclaimer: • The devices demonstrated are research prototypes only and do not

represent commitments by Microsoft to future products.• Opinions are Ken’s only and may not reflect the views of Microsoft, or

anyone else for that matter

The Future of the Mouse: An Illustrated Tour of Commercial & Research Mouse-like Input Devices

2

The First Mouse

• 1968• Douglas Englebart & colleagues• Stanford Research Institute• Separate rollers for x, y

• Why has it endured for so long?

3

• Approx. as good as pointing with hand itself (Card)

• from Balakrishnan et al. CHI ’97:• Flexibility of grip – both power & precision

grasp• Stability – damps tremor, little overhead to

acquire • vs. stylus; slightly better for high-precision

tasks• Relative mode: limits footprint, no nulling

problem• Buttons integrated & orthogonal to sensed

plane• Dragging works well, no motion when you

click• Familiar, entrenched device; +10% not

enough

Some Desirable Properties of the Mouse

4

Illustrated Tour: Some Axes of Innovation in Computer Mice

• Strictly mouse-like form factors • other devices (tablet/pen, trackball, touchpad,

joystick) have advantageous properties, but not my focus today

• What has been explored for desktop mice?• What strategies are there to improve user

performance?• What has succeeded in the marketplace?• What are some interesting areas for the future?• What are some really silly ideas?

5

• Scrolling: save time to grab scrollbar; maintain visual focus on primary task• Wheel: MS IntelliMouse, Mouse Systems

• Isometric Joystick: IBM ScrollPoint I & II

• Touchpad: Fujitsu ScrollPad Mouse

• Better w/ left hand? (Buxton & Myers)• …

• PadMouse (Balakrishnan)• Gesture-based

command selection

Multi-Channel Inputfor Secondary Tasks

6

New Sensor & Connection Technologies

GyroPoint Microsoft IntelliEye /IntelliMouse Explorer

Wacom Intuos Pen ToolID

• Motion sensing via gyros, CCD camera• Cordless, unique ID • USB, higher DPI, sampling rate, …

Logitech Cordless MouseMan Wheel

7

Better Support for the Human Senses

Logitech WingMan Force Feedback

Tractile (Cambell, Zhai et. al)

Tactile nib in mouse (MacKenzie)

• Force, Tactile feedback• 10-20% faster pointing speed?• But, errors, multiple targets,

dependent on visual feedback• Imposes constraints on device (esp. force)

8

Better Support for the Human Senses

• Force, Tactile feedback• Visual feedback on device?

• Indicators / displays• (Static Labels, Industrial Design, …)

• Audio output on mouse?• “clicking” named after the sound it makes

9

Better Support for the Human Senses

• Force, Tactile feedback• Visual feedback on device• Audio output on mouse?• Smell: powerful cue for memory

• Sensorama “VR” experience

10

Better Support for the Human Senses

• Force, Tactile feedback• Visual feedback on device• Audio output on mouse?• Smell: powerful cue for memory

• Sensorama “VR” experience Intelli Smelli TM

11

Better Support for the Human Senses

• Force, Tactile feedback• Visual feedback on device• Audio output on mouse?• Smell: powerful cue for memory• Taste: Edible Mice?

• Candy Mouse (Amurol Confections Company)

12

Additional Sensed Degrees-of-Freedom

• Rotation• Two-ball mouse, MacKenzie• 4D Mouse, Wacom tablet (A|W)

• Tilting (4DOF)• Rockin’Mouse,

Balakrishnan

• Hand Proximity in X, Y, Z• FieldMouse, (Josh Smith, MIT)

13

• 2D camera mice already here!

• Expand software capabilities?

• Mouse-like 6dof device atconsumer price points?

• Built proto using grid• Gridless design feasible?

VideoMouse

IntelliMouse Explorer

14

Lightweight Scanning

• Camera has small fov, but useful for quick scanning tasks

• e.g. scan paper title, bring up electronic document

• Vision algorithm stitches together video images

• Camera senses if grid is present Scanning mode

• Registration mark on mouse, line up w/ paper doc

• OCR not impl. – typically assumes full page of text

15

Biometric Sensing

• Fingerprint scan for user ID / login• uMatch biolink mouse

• biolinkusa.com

• GSR, heart rate, temperature, …• EmotionMouse (IBM BlueEyes project)

• Pressure sensing• Tablet stylus for art software• Area-of-contact in touchpads• Mice: not as well-defined

point(s) of contact

16

• Sense hand contact via capacitance • New events: Touch, Release

• For Palm area • for button, wheel, thumb, …

• Goal: use touch as passive input modality for awareness

• Emulation from software...• Did user release mouse? Or just holding still?• Cannot emulate multiple sensors at all.• Cannot emulate on most non-mouse devices.

Touch-Sensing Devices

17

Integration with Keyboard

• IBM Trackpoint• Reduces device acquisition times• Tiny footprint• Less intuitive cursor control• Dragging problematic

• Other pointing devices in kbd• touchpad, trackball• e.g. Acer Ergo61 Ergonomic Keyboard

18

• Just having 2 cursors is a poor design• Mouse / tablet for scrolling +15%

(Buxton/Myers)• Camera Control & Docking in 3D

(Balakrishnan)• Camera control +20% faster• Supports epistemic actions

• Map Navigation (panning + zooming)

Use Both Hands: Two Mice are Better Than One?

19

Don’t Use Your Hands at All?

• Feet for input (Hunter Digital, I/O Test Inc.)

• Head mice, Eye gaze (Zhai: MAGIC pointing)• All of these have lower bandwidth than hand• Some utility for secondary tasks, disabled use

• Head tracking in games• Hands-busy situations

20

• “optimal” or nonlinear C:D ratio • Gain theoretically does not affect

performance by Fitts’ Law (Jellinek & Card)

• Reduces footprint of device & reclutching• Visual feedback (cursor, selection highlight,

dragging rectangle, …)• Snapping the cursor• Inferring intent by cursor location?

Better Software?

21

Other Strange Axes of Innovation

• “Interfaces should be transparent to the user”

22

• “Interfaces should be transparent to the user”

• The endless N-button debate

Other Strange Axes of Innovation

1: Too Few 5: Busy 10+: ?!?2-3: About Right

23

Other Strange Axes of Innovation

• “Interfaces should be transparent to the user”

• The endless N-button debate• Nail consumer product XYZ

to the mouse• TeleMouse

(Altra Technology Inc.)

24

Other Strange Axes of Innovation

• “Interfaces should be transparent to the user”

• The endless N-button debate• Nail consumer product XYZ

to the mouse• Gelatinous (soft / squishy)

• Fellowes Corp: GelMouse, SoftGrip Mouse

25

• “Interfaces should be transparent to the user”

• The endless N-button debate• Nail consumer product XYZ

to the mouse• Gelatinous (soft / squishy)

• Less Is More?

Other Strange Axes of Innovation

26

no-button mouse (0% error rate)

27

• A lot has been tried!• Some very clever ideas -- some very bad ones

too!

• There are still some promising areas that are yet unproven for consumer devices. Ken’s bets:• New sensors, multi-channel, kbd

integrated ()

• Added DOF, 2 hands beneficial; in research stage

• Biometric sensing: some obvious wins; others?

• Tactile feedback, maybe force feedback (games)

• There are still some undiscovered gold nuggets!

Illustrated Tour: Summary