earpod : eyes-free menu selection with touch input and reactive audio feedback
DESCRIPTION
earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback. Shengdong Zhao. Pierre Dragicevic. Mark Chignell. Ravin Balakrishnan. Patrick Baudisch. Menu Selection on Mobile Devices Often Requires Visual Feedback. Why Eyes-free?. Why Eyes-free?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/1.jpg)
University of Toronto
earPod:
Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback
Shengdong Zhao
Pierre Dragicevic
Mark Chignell
Ravin Balakrishnan
Patrick Baudisch
![Page 2: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/2.jpg)
![Page 3: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/3.jpg)
![Page 4: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/4.jpg)
Menu Selection
on
Mobile Devices Often Requires
Visual Feedback
![Page 5: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/5.jpg)
Why Eyes-free?
![Page 6: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/6.jpg)
Why Eyes-free?
![Page 7: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/7.jpg)
Eyes-free capable
![Page 8: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/8.jpg)
Eyes-free capable
Marking Menus
![Page 9: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/9.jpg)
Eyes-free capable
Marking Menus (Expert Use)
![Page 10: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/10.jpg)
Eyes-free capable
Marking Menus (Expert Use)
![Page 11: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/11.jpg)
True eyes-free
• TouchPlayer (Pirhonen et al, 2002)
![Page 12: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/12.jpg)
True eyes-free
• TouchPlayer (Pirhonen et al, 2002)
• Head gesture-based auditory pie menu (Brewster et al, 2003)
![Page 13: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/13.jpg)
earPod
13
![Page 14: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/14.jpg)
earPod
14
![Page 15: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/15.jpg)
earPod Design
15
![Page 16: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/16.jpg)
earPod Walkthrough
16
![Page 17: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/17.jpg)
Browsing
![Page 18: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/18.jpg)
Fast browsing
18
![Page 19: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/19.jpg)
Fast browsing
![Page 20: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/20.jpg)
Very fast browsing and cancel
20
![Page 21: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/21.jpg)
Very fast browsing and cancel
![Page 22: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/22.jpg)
Expert use
![Page 23: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/23.jpg)
Expert use
![Page 24: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/24.jpg)
Interactive Voice Response System
24
![Page 25: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/25.jpg)
Interactive Voice Response System
25
Slow
Painful
“Touchtone Hell”
(Yin and Zhai, 2006)
![Page 26: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/26.jpg)
Characteristics of IVR
• Inpatient user – Urgent; want to talk to real person
• Lack of user control– It’s both bad and good
• Typically narrow but deep phone tree, resulting long sentences
![Page 27: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/27.jpg)
Visual vs. auditory menu
27
Visual Linear Menu
![Page 28: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/28.jpg)
Visual vs. auditory menu
28
Visual Linear Menu IVR System
![Page 29: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/29.jpg)
Problems of IVR system
• Speech is serial and slow (Slowiaczek and Nusbaum 1985)
![Page 30: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/30.jpg)
Problems of IVR system
• Synchronous vs. asynchronous communication
Synchronous
Visual Outputs
Mouse Input
time
User
System
![Page 31: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/31.jpg)
Problems of IVR system
• Synchronous vs. asynchronous communication
Synchronous
Visual Outputs
Mouse Input
time
User
System
Asynchronous
time
Audio
PressKey
User
System
WaitingPressKey
Waiting
Audio
![Page 32: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/32.jpg)
Problems of IVR system
• Transition to expert use
Skill development
Ski
ll le
vel
Novice(Voice menu: recognition)
![Page 33: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/33.jpg)
Problems of IVR system
• Transition to expert use
Skill development
Ski
ll le
vel
Expert(direct code access: recall)
Novice(Voice menu: recognition)
![Page 34: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/34.jpg)
Problems of IVR system
• Transition to expert use
Skill development
Ski
ll le
vel
Expert(direct code access: recall)
Novice(Voice menu: recognition)
![Page 35: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/35.jpg)
earPod’s approach
• Problem: speech is serial and slow
![Page 36: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/36.jpg)
earPod’s approach
• Problem: speech is serial and slow– By-pass audio
![Page 37: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/37.jpg)
earPod’s approach
• Problem: speech is serial and slow– By-pass audio – Use interruptible audio feedback
![Page 38: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/38.jpg)
earPod’s approach
• Problem: speech is serial and slow– By-pass audio – Use interruptible audio feedback– Use non-speech audio
![Page 39: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/39.jpg)
earPod’s approach
• Problem: asynchronous communication
![Page 40: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/40.jpg)
earPod’s approach
• Problem: asynchronous communication– Synchronous communication (reactive
audio feedback)
Audio Outputs
Touchpad Input
time
User
![Page 41: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/41.jpg)
earPod’s approach
• Problem: steep novice to expert transition
![Page 42: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/42.jpg)
• Learning menu structure
Smooth transition
![Page 43: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/43.jpg)
• Learning menu structure
• Learning expert gesture
Smooth transition
Target Item
![Page 44: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/44.jpg)
• Learning menu structure
• Learning expert gesture
Smooth transition
Target Item Novice
![Page 45: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/45.jpg)
• Learning menu structure
• Learning expert gesture
Smooth transition
Target Item Novice Novice/Expert
![Page 46: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/46.jpg)
• Learning menu structure
• Learning expert gesture
Smooth transition
Target Item Novice Novice/Expert Expert
![Page 47: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/47.jpg)
47Skill development
Ski
ll le
vel
Expert(direct tapping: recall)
Novice(glide browsing: recognition)
![Page 48: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/48.jpg)
Support Single-hand Usage
48
(Karlson et al, 2007)
![Page 49: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/49.jpg)
Haptic guidance
![Page 50: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/50.jpg)
Haptic guidance
![Page 51: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/51.jpg)
earPod Design Summary
• Touch Input
51
![Page 52: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/52.jpg)
earPod Design Summary
• Touch Input• Reactive & Interruptible Audio
52
![Page 53: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/53.jpg)
earPod Design Summary
• Touch Input• Reactive & Interruptible Audio• Non-speech Audio
53
![Page 54: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/54.jpg)
earPod Design Summary
• Touch Input• Reactive & Interruptible Audio• Non-speech Audio• Input/Output Mapping
54
![Page 55: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/55.jpg)
earPod Design Summary
• Touch Input• Reactive & Interruptible Audio• Non-speech Audio• Input/Output Mapping• Direct Item Access
55
![Page 56: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/56.jpg)
earPod Design Summary
• Touch Input• Reactive & Interruptible Audio• Non-speech Audio• Input/Output Mapping• Direct Item Access• Transition to Expert Use
56
![Page 57: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/57.jpg)
Experiment
57
• earPod vs. iPod-like linear menu
![Page 58: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/58.jpg)
Experiment – visual condition
58
![Page 59: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/59.jpg)
Experiment – audio condition
59
![Page 60: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/60.jpg)
Experimental design
• Within-subject, counter-balanced• 12 participants x• 2 techniques (audio and visual) x• (40+80) items for the 2 menu
configurations (8 and 8x8) x • 4 blocks• = 11,520 menu selections in total.
60
![Page 61: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/61.jpg)
Results
61
Block number Block number
![Page 62: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/62.jpg)
Results
62
Block number Block number
![Page 63: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/63.jpg)
Results
63
Block number Block number
![Page 64: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/64.jpg)
Results
64
Block number Block number
![Page 65: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/65.jpg)
Results
65
Block number Block number
![Page 66: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/66.jpg)
Results
• Detailed Log Analysis– 9/12 reached expert use (1 level menu)– 8/12 reached expert use (2 level menu)
66
![Page 67: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/67.jpg)
Results
• Detailed Log Analysis– 9/12 reached expert use (1 level menu)– 8/12 reached expert use (2 level menu)
• Subjective Feedback – 4 preferred earPod– 4 preferred iPod-like visual menu– 4 had no preferences– 11/12 suggest combine audio & visual
67
![Page 68: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/68.jpg)
earPod limitations
• Work with structured menu of limited breadth
68
![Page 69: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/69.jpg)
earPod limitations
• Work with structured menu of limited breadth
69
![Page 70: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/70.jpg)
Conclusion
• earPod: eyes-free menu selection for structured menu of limited breadth
• Performance comparable to iPod-like linear menus
• Support single-handed interaction• Design for mobile use
70
![Page 71: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/71.jpg)
Future Work
• Long/dynamic menus
71
![Page 72: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/72.jpg)
Future Work
• Long/dynamic menus
• Dual channel menu
72
![Page 73: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/73.jpg)
Future Work
• Long/dynamic menus
• Dual channel menu
• Dichotic listening (Ranjan et al., 2006)
73
![Page 74: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/74.jpg)
Future Work
• Long/dynamic menus
• Dual channel menu
• Dichotic listening (Ranjan et al., 2006)
• IVR on the phone
74
![Page 75: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/75.jpg)
AcknowledgementDanielle Lottridge, Tomer Moscovich, Mike McGuffin,
Jingnan Yang, Heather Thorne, Alvin Chin,
Derek Nowrouzezahrai, Yongchang Zhao, Lisa Min,
Andrew Chignell, Paula Gilverson, Michelle Qu,
Andrew McGutman, Noah Lockwood, John Chattoe,
Annie Xu, Silvia Gonzaloz, Anna Malandrino
Anonymous reviewers
![Page 76: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/76.jpg)
Mobile Usage Design
• Stability issues
76
![Page 77: earPod : Eyes-free Menu Selection with Touch Input and Reactive Audio Feedback](https://reader035.vdocuments.us/reader035/viewer/2022062314/56813fdb550346895daac221/html5/thumbnails/77.jpg)
earPod Novice to Expert Transition
2 61 7
3 5
0
4