remote (mobile) usability testing lab
DESCRIPTION
Describes the system we developed for conducting remote usability tests. It also gives you insights on how we changed the system to perform mobile tests... .TRANSCRIPT
Remote (Mobile) Usability Testing Lab
Human.Interaction.Design.2010
Jan Moons
November 23rd, 2010
2 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
About
In this presentation we describe our experiences in developing and testing a system that makes it possible to
usability test (mobile) applications.
Disclaimer:Work in progress
Usability Testing?In a nutshell
4 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Usability Testing?
A technique to observe people, who are representatives of the target audience, using a product to discover errors and areas of improvement.
Typically measuring how well people respond in following areas:
Efficiency
Effectiveness
Satisfaction
5 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
5 Characteristics
Every usability test has the following characteristics:
1. Primary goal: Improving usability of a product
2. Participants represent real users
3. Participants perform real tasks
4. Observe and record (log) what participants do and say
5. Analyze the data, diagnose the real problem, and recommend changes to fix those problems
A Practical Guide to Usability Testing, J.S. Dumas and J.C.Redish
Usability Testing @ Bell LabsProblems and Vision
7 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Typical usability test @ Bell Labs
Participant
Observer
Logging software
Participants are inevitably invited to the
“lab”!
8 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Problems with State-Of-The-Art
1. Software installation needed on participant’s computer
Participants don’t appreciate installing software on their PC’s
Live recruitment is not possible
2. Software doesn’t run on all major operating systems
Target: MS Windows OS (XP/Vista)
Problematic: Mac OS, Linux, Solaris, etc…
3. Observing and recording over the Internet/Intranet hard or even not possible
Users cannot participate from their own “premises” with their own devices
Remote usability testing is an issue
4. Build own Remote Usability Testing system to solve these problems
9 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Vision
1. Design a system to remotely observe and record users’ actions, voice and facial expressions on an Internet/Intranet connected PC.
2. With the ability for the observer to analyze the data and diagnose the problems
3. Based on Web technologies that supports all major Operating Systems
4. But without having to install any special software on the participant’s or even the observer’s computer
10 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
APPAPP
Architecture
Java AppletFlash Applet
RTMP
Audio/Video
Streaming
Screencast
Streaming
RTMP A/V
JDBC
Observer’s browser
Tester’s browser
Remote Objects
11 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Observe / Replay
Participant’s webcam
Participant’s screen
Logging panelCurrent task
Mouse pointer
12 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Analyze
EffectivenessEfficiency
Data Analysis
Mobile Usability TestingPopular techniques
14 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Look-over-shoulder
http://www.wirelessrerc.org/for-consumers
Screen recording?
15 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Document Camera
http://www.easeofweb.com/mobile_usability_test.html
Fixed positionUser has to be seated
16 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Sled / Lamp
http://www.easeofweb.com/mobile_usability_testing.html
“Mobile” but very intrusive
17 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Pros and Cons
Document/Sled Camera:
Pros: Works with all mobile devices No need to install software on mobile Interaction with mobile is visible Ideal for lab testing
Cons: Very intrusive Experience is not natural No automatic key logging
18 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Screen-capturingJailbroken iPhone Rooted Android
19 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Pros and Cons
Screen-Capturing:
Pros: Less/non-intrusive More natural experience Testing in the field (e.g. Location-Based Services)
Cons: Software installation needed for recording
For each OS (different) implementation needed! Doesn’t show the users hands actually interacting with the mobile
Mobile Usability TestingNatural Experience
21 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Vision
1. Extend our custom-built usability testing lab for the computer to mobile devices
2. With the ability to observe and record:
1. The mobile screen / user’s actions
2. The user’s voice (environment noise)
3. The user’s face (optional)
3. Ultimate Goal: Making the experience as natural as possible for user testers
22 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Implementation
Prioritized:
1. Camera capturing mobile screen
All mobile devices
Ability to capture interactions (hands)
2. Screen-sharing + Audio/Video streaming
Ultimate goal of a natural experience; non-intrusive
1st focus Android mobile device
23 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Approach 1: Camera capturing mobile screen
DIY Document
CameraPC for
transmission of streams to server
Webcam Face
Capturing
Also ideal for testing Paper Prototypes!
24 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Architectural Changes
APPAPP
Camera 2
RTMP
Audio/Video
Streaming
Screen
Streaming
RTMP A/V
JDBC
Observer’s browser
Tester’s browser
Remote Objects
Camera 1
25 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Approach 2: A/V Screen-Capturing Screen capturingIssues!!!
26 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Observer
27 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Architectural Changes
APPAPP
Java AppFlash Applet
RTMP
Video
Streaming
Screencast
Streaming
RTMP A/V
JDBC
Observer’s browser
Remote Objects
Audio SIP
Tester’s
Mobile
ADB
Screen capture
Connected Computer
Also support for VNC
28 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
Future work…
1. Measuring satisfaction
Surveys
Emotion recognition (voice, facereading or combination)
2. Automatically generating (video) highlights of observations
3. Capturing the mobile’s sensors:
GPS-coordinates (for Location-Based Services)
Accelerometers, touch events (application / device interaction)
4. SIP/RTMP audio/video streaming
Last but not least, eat our own dog food and usability test the system ;-)
29 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights
reserved.
www.alcatel-lucent.comwww.alcatel-lucent.com