skype for business usability study
TRANSCRIPT
Usability Study Nathan Hedin | Clara Jauquet | Andre Wyatt
Skype for Business
Table of Contents
Introduction 4
Method and Structure 5
Participant Profiles 7
Severity Ratings 8
Executive Summary 9
Scorecard Results 12
Detailed Findings 15
Conclusions 101
Appendix 105
2
About the Study
3
About the Study
What
Skype for Business on Android
Where
Microsoft Usability Labs, Redmond, WA on 2/20, 2/22, 2/26
Why
Ongoing series of usability tests to evaluate the current release of Skype for Business
How
10 participants worked through 10 scenarios mapped to experience outcomes (tasks). Task success rates, subjective
ratings, and qualitative data about the participant’s experience were collected.
Research Objectives
Has the newest release of SfB improved from a usability and customer satisfaction standpoint?
Can users use multiple features at once?
How valuable do users think the meetings feature is?
Which key features are easily discoverable and learnable by users, and which features do they struggle with?
4
Method and Structure
Pre-test questions
Open questions
about workplace
culture and comm
tools
Ovo Software
Android screen and
participant recorded
Scenarios and tasks
10 scenarios, each
with multiple tasks
within SfB, Success
rates collected
Experience Ratings
Collected after each
scenario with probing
questions
Qualitative Data
About participant task
experience
Post-test questions
Open questions
about experience
with Skype for
Business app
Test Time
90-120 minutes
1 2 3
5
Sample Scenario
Scenario 2b: Once you figured out what the
lunch plan is with each colleague, combine them
into one group conversation to finalize a place to
go for lunch. At the end of the task, return to
main screen view.
Key Tasks:
1. Add a contact to the ongoing chat to create
group chat
2. Have a group chat
3. Return to the main view
Experience Rating:
6
Participant Profiles
P1 P2 P3 P4 P5 P6 P7 P8 P10 P9
Occupation Communication Tools Age Range
7
Severity Ratings
Each finding was given a severity rating based on how it impacts the application usability
Low
Minor usability problem: fixing this should be given
low priority
High
Usability catastrophe: imperative to fix before
product can be released
Medium
Major usability problem: important to fix, should be
given high priority
8
Executive Summary
9
Positive Findings
Overall look and feel of the application
Participants mentioned that it was “familiar” [to Skype desktop], “easy to
navigate,” and liked the “app arrangement” and “UI”
Individual IMs and Notifications
Participants encountered no issues searching for a contact, IMing them,
acknowledging notifications, and responding to IMs
Voice Calls
Participants encountered no issues making a voice call, muting/unmuting
the call, and ending the call
Video Calls
Participants encountered no issues receiving a video call
Meetings
Participants encountered no issues joining meetings
Availability Management
Participants encountered no issues changing their availability status
...clear and intuitive UI, I
liked the look, clean,
colors and circular icon,
look and feel and
arrangement of things.
10
Opportunities
Multiple video windows during group video calls
● Most users independently mentioned they would like to see, or expected to see, video windows
of each of the participants on a video call
Calls
● Error messages were persistent and obscured notifications
● Incoming IM notifications were frequently unseen during phone and video calls (each have their
own issues)
● It was easy for users to accidently drop a call when attempting to add participants to it
Meetings
● Several tasks did not receive top scores from users
● User confusion was observed with in-call buttons, video switching during active speaker mode,
and figuring out who was and was not in the meeting
● Half of all users expressed interest in seeing a shared desktop feature made available for the
Android app
11
Scorecard (% of Top-Two Box Scores*)
12
*Top-two box score: % of satisfaction scores rated a 4 or 5 by participants
Unable to share historical ratings and top box scores
Scorecard (% of Top-Two Box Scores) cont.
13
Unable to share historical ratings and top box scores
Scorecard (% of Top-Two Box Scores) cont.
14
Unable to share historical ratings and top box scores
Detailed Findings
15
IM Conversations Scenario 2a: You want to chat with your colleagues. Search for your colleague Nathan and send him an IM conversation
about his lunch plans for today. Then IM Andre to see if he is free for lunch today.
Run a search for your colleague
Have an IM conversation
Have two IM conversations
Be aware of messages from another conversation
Be able to respond to messages from another conversation
16
Search for Contact
17
Search for Contact
Perfectly easy. It was direct to see
how to get there.
- P9
18
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• Participants encountered no major issues searching for a colleague
• The search bar was the first place all users went to when searching for a
colleague
• The magnifying glass was appreciated by users
• The search result clears and reloads every time the user types a new letter,
which one user complained about
• Searched colleague name showed up twice in one participant's search results
Consider:
• Removing the clear state after each letter entry
• Looking into why a contact might show up twice
Search for Contact
19
Have an IM Conversation
20
Have an IM Conversation
Very simple and intuitive…
-P5
21
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• Participants encountered no major issues when having an IM conversation
• When exiting out of a chat with a partially-typed message, the app says
“Message saved as draft” but the message is not there when the user returns
• One user mentioned she likes to type messages before she sends them when
she multitasks
Consider:
• Saving user’s partially drafted messages for when the user returns to app or
conversation
Have an IM Conversation
22
Have two IM Conversations
23
Have two IM Conversations
...didn’t even have to think about it…
-P2
24
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• Met expectations
• Users liked the notifications for incoming IMs
• The back button was appreciated by many users
Have two IM Conversations
25
Be aware of Messages from Another Conversation
26
Be aware of Messages from Another Conversation
...was looking to see if someone answered
and just saw ‘available’...
-P1
27
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• For new messages, most users encountered no major issues noticing
messages
• Sometimes the first line of the most recent IM shows up below a user’s name,
and sometimes it just says “Available”
Consider:
• Consistently showing the first line of the last message sent or received
Be aware of Messages from Another Conversation
28
Respond to Messages from Another Conversation
29
Respond to Messages from Another Conversation
I want to be able to respond to a message and
not leave the application I was in…
- P6
30
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Respond to Messages from Another Conversation
Findings
• Within the app, no users had issues responding to a message
• The only user who gave a 4 had attempted to respond from the Android pull-
down notification bar (outside of the Skype app)
Consider:
• Allowing users to respond from the Android pull-down notification bar
31
IM Conversations
Scenario 2b: Once you figured out what the lunch plan is with each colleague, combine them into one group conversation to
finalize a place to go for lunch. At the end of the task, return to main screen view.
Add a contact to the ongoing chat to create group chat
Have a group chat
Return to the main view
32
Create a Group Chat
33
Create a Group Chat
Create a Group Chat
34
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Create a Group Chat
Findings
• Group chat was easy to set up - click a name and you’re returned to the chat
with them in it
• “Impressively simple”
• The tradeoff is it would be hard to set up group chats with many people
• “If I wanted to add multiple people he noted he would have to go in and out to
add people”
Consider:
• Allowing users to select several people to chat at once
Create a Group Chat
35
Voice Calls
Scenario 3: You want to have a voice call with your colleague Andre. Ask him about the status of Project X. While Andre is
responding, mute the call, cough, and then unmute the call. When you finish your conversation, end the call.
Make a voice call
Mute and unmute the call
End the call
36
Make/End Call
37
Make/End Call
Make/End Voice Call
So similar to my own phone that it didn’t take much thought… -
- P3 38
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• Making and ending calls were very easy
• Two users complained that they couldn’t find speakerphone
Consider:
• Changing “audio” to “speaker”
Make/End Call
39
Mute/Unmute Call
40
Mute/Unmute Call
*n=9 (one score not captured)
Icon was big and clear and
right there, very well labeled…
- P7
41
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• Every user was able to mute and unmute with ease
Mute/Unmute Call
42
Voice Calls
Scenario 4: You are going to receive a call from your colleague Andre. During the call, you might get an IM message from
Nathan; ignore his IM message. At the end of this task, return to the main screen view.
Receive a call from the first screen
Receive an IM message during the call
End the call
43
Receive a Call from the Homescreen
44
Receive a Call from the Homescreen
That was fine, just like a regular call…
-P6
45
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• No users had any issue answering a call
• Based on the feedback we received, it seems like the call receiving notification
is close enough to a regular conventions that it is easy to tell users are
receiving a call, but different enough to make it clear that they are answering a
Skype call and not a system phone call
Receive a Call from the Homescreen
46
Receive and Ignore an IM In-Call
47
Receive and Ignore an IM In-Call
I didn't know I got an IM during the call, until after
I was off the phone.
- P6
48
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• The notification was missed by half of the users
• Many had the phone up to their ear and never saw the banner that came down
on screen. Since there is no audio or vibration cue, it was impossible to tell
something had occurred
Consider:
• Making IM notifications more obvious
• Adding an audio cue
• Make notifications appear over errors
• Visually differentiating notifications from the app
Receive and Ignore an IM In-Call
49
Video Calls Scenario 6a: Start a video call with Andre. Make sure your face is centered and the lighting is good for the video call.
During the call, your colleague Nathan might contact you. Ignore him.
Make a video call
Make sure your video looks professional
While in a call, ignore an incoming video call
Stay on the call
50
Make a Video Call and Check Your Video
51
Make a Video Call
Very satisfied with that…
- P5
52
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Check Your Video
53
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• No users had any issue making a video call
• No users had any issue checking their video
• Many users expressed discomfort looking at their own face for such a long time
Consider:
Easing user’s discomfort while waiting for the other party to start their video
Check Your Video
54
Ignore an Incoming Call
55
Ignore an Incoming Call
I didn’t see an “x” right away...I wanted to
look at the person I was talking to rather
than the top of the screen.
- P10
56
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Findings
• The phone didn’t stop vibrating for 3 users
• “The vibration is very distracting while trying to hold a conversation”
• The notification wasn’t seen by 5 of the users
• Two users requested a polite decline option
• Users wanted to focus on the ongoing call, not the incoming call
Consider:
• Having the phone automatically stop vibrating after a short amount of time
• Increasing the length of time the notifications were on the screen for and make
it simpler to understand
• Fewer buttons, clearer affordances, or button labels
• Adding a polite decline option
Ignore an Incoming Call
57
Video Calls
Scenario 6b: During your call with Andre, you decide you need to include Nathan in the discussion. Please add him to the
call. At the end of the task, return to the main screen view.
Add person to 1:1 call
Have a group video call
End the call
58
Add a Person to the Call
59
Did not see add participant button right
away…maybe have more icons, so it is
more obvious.
-P10
Add a Person to the Call
60
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Add a Person to the Call
61
Unable to share usability
videos
Findings
• Three users dropped the original call while trying to add the second person
Consider:
• Unnestling “add a participant” from the submenu
• When making a new call while in another call, adding a “Merge” or “End
Ongoing Call” option post click
Add a Person to the Call
62
Have a Group Video Call
63
...switching back and forth between the users… I think if
you have too many participants and if the video keeps
switching back and forth it's going to drive people mad or
give them a seizure.
- P7
Have a Group Video Call
64
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Multiple People Video Highlight
Unable to share usability
videos
Findings
• Eight users independently requested having either split screen or thumbnail
video options
Consider:
• Adding a way to see everyone’s video at once
Have a Group Video Call
66
Meetings
Scenario 7a: Confirm if you have a meeting. Join the meeting and get your bearings by confirming who is in the meeting
and who hasn’t joined yet. View the contact card details of one of the attendees.
Check for and join a scheduled meeting
Know who is in the meeting (and who hasn’t joined yet)
Know who is speaking in the meeting
View contact card details of one of the attendees
Stay in the meeting
67
Check for and Join a Scheduled Meeting
68
Check for and Join a Scheduled Meeting
*n=9 (technical issue, P7)
Even if it’s a bad day before
coffee, you aren’t going to
miss that [join] button...
- P2
69
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Check for and Join a Scheduled Meeting
Findings
• All participants saw scheduled meeting right away and experienced no
problems joining the meeting
• Participants gave this task a top-box score, 8/9 rated the experience a 5
• 9/9 participants succeeded in completing the task
70
Know Who is in the Meeting (and who hasn’t joined yet)
71
Know Who is in the Meeting (and who hasn’t joined yet)
*n=9 (technical issue, P7)
I know who is in the meeting but I wasn't
sure who hadn't joined yet that was part
of meeting. I'd probably want to see their
name on the list, but I didn't see that.
- P5 72
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Know Who is in the Meeting (and who hasn’t joined yet)
Findings
• 4/9 participants mentioned they would like a list of who is present and who is
absent from the meeting “roll call”
• 2/9 participants failed to complete the task
• Most participants were able to find the attendee list by clicking on the number of
participants in the top left hand corner of the meeting screen (participants banner)
• Meeting is separated presenter vs attendee, no one appeared as attendee
Consider:
• A way to represent invitees who are not in the meeting in the participants banner
• Giving the “participants banner” a visual affordance
73
Know Who is Speaking in the Meeting
74
Know Who is Speaking in the Meeting
*n=8 (technical issue P7, & score not collected P2)
People in meetings love to talk over each other….I don’t
know how it [the app] would handle multiple people
talking at the same time.
- P8
75
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Participants Video Highlight
Unable to share usability
videos
Know Who is Speaking in the Meeting
Findings
• 4/8 participants could not determine who was speaking in the meeting
• 2 participants mentioned preferring a split pane option, seeing all meeting
attendees, over active speaker mode
• Delay as view switches between attendees bothered some users
• Primary differential to Skype’s competitors
Consider:
• Incorporating a version of the split pane for meetings as a visual cue
77
View Contact Card Details of One of the Attendees
78
View Contact Card Details of One of the Attendees
*n=9 (technical issue P7)
I expected to just click on his name to
see his contact info, but I had to do a
couple more clicks to find that. But it
wasn’t that big a deal.
- P5
79
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
View Contact Card Details of One of the Attendees
Findings
• 1/9 participants could not locate a meeting attendees contact card – too many
steps masking the information
• 2 participants expressed they expected to find the contact card by clicking on
another user’s image
• 1 participant did not understand what the contact card is
Consider:
• Explore ways to make the banner more discoverable
• Other issues may be the root cause of discoverability (previous task: meeting
attendees)
80
Meetings
Scenario 7b: One of the attendees will present and share their desktop. During the presentation, you may receive side IM
messages. Respond to Andre, but ignore others. Leave the meeting; Rejoin momentarily; then leave for good.
View the shared desktop to participate in the presentation
Have side IM conversations with a colleague.
Leave the meeting and rejoin the meeting (then leave)
81
View the Shared Desktop to Participate in the Presentation
82
View the Shared Desktop to Participate in the Presentation
*Desktop not shareable on Android App (task not possible to complete)
n=9 (technical issue P6)
Don’t see any alerts, which I
assume is going to happen...
- P5
83
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Shared Desktop Video Highlights
Unable to share usability
videos
View the Shared Desktop to Participate in the Presentation
Findings
• 9/9 participants could not view the desktop
• 5/8 users expressed interest in the shared desktop feature on an Android
• 3/8 users mentioned they expected to receive notifications about a shared
desktop
Consider:
• Addition of the shared desktop feature for the Android platform
• Include an affordance or notification when there is an incoming shared desktop
or another person is presenting
85
View the Shared Desktop to Participate in the Presentation
When I have to go into the office at 4 am and talk to the UK I
could just do it from home, that would be wonderful... When it's,
you know, it's not planned when when you have a workaholic boss like
mine sending you messages at midnight and he genuinely wants to
call across seas and hash it out right now. I put on a blouse but I am
sitting in my pjs, sitting at home in bed, I can do it all from my phone. I
am one of those people, I've got my smart tv, my smart tablet. I have
got everything you're supposed to have, but for some freakin
reason I use my phone. I watch my movies from my phone, I take
into the bath and I set it up on a little stand. I got a tablet, it doesn't
need a stand, it's got a larger screen, it's got better sound, but I don't
because it's ten feet away and I'm tired...I've got my phone with me. I
always have it with me. I love when it can do everything.
- P2
86
Have Side IM Conversations with a Colleague
87
...did not read as a text conversation...
- P8
Have Side IM Conversations with a Colleague
88 *n=9 (technical issue P6)
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Have Side IM Conversations with a Colleague
Findings
• No significant findings with actual task, but secondary issues were found
• 2 participants mentioned that the IM view “did not read as a text conversation”
and felt cluttered with interaction history
• 1 participant thought the shared meeting IM window was a side conversation
Consider:
• Simplifying IM window so it just displays IM conversations instead of an entire
history
89
Leave the Meeting and Rejoin the Meeting (then leave)
90
Here’s a rejoin button, how about that?
- P8
Leave the Meeting and Rejoin the Meeting (then leave)
91 *n=9 (technical issue P6)
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Leave the Meeting and Rejoin the Meeting (then leave)
Findings
• Participants experienced no difficulties leaving the meeting; hang up icon was
very straightforward
• Participants experienced no difficulties rejoining the meeting
• Rejoin button was obvious and easy to find
92
Manage Conversations
Scenario 8: During the last meeting, you remember you had ignored some IM messages. Find those missed messages and
respond to them. At the end of the task, return to the main screen view.
Find the missed conversations
Follow up on the missed communications
93
Find and Follow up on the Missed Communications
94
Bolding is a very common visual cue that
something new is there.
- P2
Find and Follow up on the Missed Communications
Would be nice if new messages came to
the top...
- P2
95 *n=9 (technical issue P6)
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Find and Follow up on the Missed Communications
Findings
• 4/9 referenced the bolded text as a common “visual cue” for an unread
message
• 8/9 participants succeeded in completing the task
• 1 participant found the entire history view in the IM conversation to be
“Disorienting”
• Chronological order?
• “all these things look the same except for the highlights…but I didn’t know if
that is new, if I missed it, or ignored it.” –P1
Consider:
• Simplify IM window
96
Status and Availability
Scenario 9: Change my availability status to Do Not Disturb. At the end of the task, return to the main screen view.
Change my availability status to: Do Not Disturb
97
Change my Availability Status to: Do Not Disturb
98
Change my Availability Status to: Do Not Disturb
Pretty obvious that clicking on the face
would bring up options about yourself.
- P1
99
Average
% Top-Two Box
Exceeds Criteria
% Change from SfB 2015
1
2
3
4
5
Satisfa
ction L
evel
Satisfaction Scores
Unable to share historical
ratings and top box scores
Change my Availability Status to: Do Not Disturb
Findings
• 10/10 participants succeeded in completing the task and rating it
• 7/10 participants immediately went to avatar image and changed their status
• 3/10 participants first checked contacts and calendar before touching the profile
picture
• Profile picture as personal settings seems to be the industry standard
Consider:
• Keep as is, profile picture as link to profile settings
Had a moment of hesitation... that
would be the least intuitive, but i
don’t disagree where it was, it must
be under me.
-P5
100
Conclusions
101
Recap
• Participants found IM conversations easy to navigate
• Participants found making and receiving calls to be easy
• Incoming IM notifications were missed by many users
• Ignoring an incoming call was a distraction for users
• Participant information was nestled within submenus
• Split pane view of users on a video call or in a meeting requested over active
speaker mode
102
Next Steps
• Implement improvements on high severity findings, most notably:
• Affordance to the “participant banner”
• Split pane screen of all users in a call/meeting, with the speaker or selected video taking over
the majority of the real estate
• Change the incoming IM/call notification
• Match the desktop sharing experience to iPhone and desktop
• Run another benchmark study after implementation of changes to monitor
satisfaction ratings increase
• Make process improvements
103
Process Improvement and Food for Thought
• Add more company contacts for a realistic work environment
• Add a task between leaving and joining the meeting
• Side IM with a colleague during the meeting was not a realistic experience with
realistic challenges as viewing shared desktop was not possible
• Even though meeting is separated as presenter vs attendee, no one ever
appeared as attendee
• Keeping profile settings under profile picture?
• History landscape
104
Appendix
105
Documents
Screener:
Data not available to share
Usability Test Kit:
Data not available to share
Data Collection:
Data not available to share
106