Alternative Approaches for Gathering End User Data

Anne de RidderSr. User Experience Architect

White Horse, Portland, OR

UW HCDE PresentationFebruary 26, 2010

Overview

Introduction

The Why and When of using alternate approaches

So what are these approaches anyway?– Case Study 1: Remote Unmoderated User Testing– Case Study 2: Low/High-Tech User Testing– Case Study 3a: Online Focus Groups– Case Study 3b: Survey + Moderated Interviews– Case Study 4: Online Card Sort

Questions/Discussion

Introduction MS in Technical Communication from UW HCDE in 1999 Professional experience in areas of user experience design,

usability, technical writing, and marketing communications Worked with big & small companies as both employee and

consultant – IBM, Sharp Microelectronics, JPL, Tektronix, Mountain Hardwear, Trane

Commercial Systems, several municipalities, environmental consulting firms, and many more!

Currently at White Horse– Digital marketing agency specializing in the convergence of emerging

and traditional media to create immersive Web experiences. • Web development, digital marketing, technical engineering, emerging media, audio/video

production

– Persona-led Design– Dedicated User Experience team

• If it is UX-related, we can do it!

The Why and When of using alternate approaches

Why the need for alternate approaches?– Additional tools in the HCD toolbox

When to use alternate approaches?– Defined by the project conditions:

• Project phase– What is the question you are trying to answer? And why?

• Project sponsor– Who are your “customers”?

– What is their objective?

• End user data requirements– Type of data; volume of data; quality of data

– Geographic distribution and language requirements; interface being evaluated; result objectives

• Timeframe

• Cost

So what are these approaches anyway?

Focus of this presentation is on Online tools

A lot of options out there that address many needs– Card sorts, navigation evaluation, surveys, user testing, and

more.

Variation among options within sub-categories– Considerable differences in quality and depth of data

collected

The option you select is based on what you’re looking for– Result objectives– End user data requirements

Case Study 1: Remote Unmoderated User Testing

Testing objective: – Validate that a redesigned corporate Web site addresses the task-based

and informational needs of the business’s 7 distinct user segments

– Collect data from a large number of geographically dispersed (US) group of users from each segment in a short turn around time with relatively “low” cost

• Minimum 75 completed tests (5 to 15 tests per segment); maximum 400 tests total

Recommendation: Remote Unmoderated User Testing– WebEffective

• Users invited via email invite and intercept on the Web site.• Task-based test using the client’s Web site

– Collection of both qualitative and quantitative data, though focus was on the qualitative.

– Collection of user behavioral data including:» Clicks, hovers, scrolling, text entries, form field selections

Case Study 1: Results 14 days of testing Tests Begun: 1420; Reportable: 224

– Disqualified: 585 (screener; filter); Dropped-Off: 611 (pre-background questions; 331 after background survey)

Biggest differences from traditional testing:

– It’s a lot of data!• 40 hours of data analysis; 70 slides of data summaries—and that was just

capturing the high level picture

– Need to “recreate” the user experience

– Need to recruit/include greater number of participants to off-set drop-offs, disqualifications, outliers, “no-navs”, “give-ups”

– (Not so) accurate measurement of confidence ratings

Case Study 1: Community Perspective

What does the overall research community say?– Independent, informal study shows statistically comparable

results from lab-based tests and remote unmoderated tests• Sauro, Tullis, Molich, Kirakowski, UPA 2009• http://www.measuringusability.com/unmoderated-testing.php

• Study also highlights the need to collect larger numbers of completed tests in order to have substantial sub-set of “qualifying” data

– “Beyond the Usability Lab: Conducting Large-scale Online User Experience Studies” by W. Albert, T. Tullis, and D. Tedesco, 2010

Case Study 2: “Low/High-Tech” Testing

User acceptance prototype testing “Live” tests via online interface

– Project A:• Low participant participation on day 1 of listening labs• Double-booked day 2 to supplement live sessions• Used GotoMeeting to allow users to interact with prototype plus

have ability to record sessions (voice; mouse movements)– Project B:

• 5 hour budget for user testing• Three distinct user segments• Used GotoMeeting to allow users to interact with prototype plus

have ability to record sessions (voice; mouse movements)• Problems with firewalls; did not allow users to fully engage with

prototype

Case Study 3: Online Focus Groups Web site redesign to incorporate more customer-centric approach

for finding financial products and services

Outcome of Task Analysis and Modeling Phase– Validate that interpretation and synthesis of past research

matches actual user behavior• Collect feedback on where experience and task flow

diagrams match/miss current user behavior and needs• 1,200 participants total (for statistical significance)

including both demographic and geographic segmentations

– Large-scale synchronous online focus groups• Allows quicker and cost-effective collection of significant

data across personas and geographic locations• FacilitatePro (www.facilitate.com)

Online Focus Group

Case Study 3b: Survey + Moderated Interviews

Design Concept Testing– Goal:

• Gain insight into user expectations about design elements, interaction flows, and outside influences

• Gain clear picture of the motivators behind target user’s assessments of design concepts

• Testing Outcome: Final concept direction selected based on user data; inform content strategy

• 600 users

– Method: iModerate iMpact• Collection of both qualitative and quantitative data

for each target user group• Survey to larger group (via any survey tool of choice)• Percentage of users intercepted and “interviewed”

by professional moderators

Case Study 4: Remote Card Sort

Web site redesign for a domain service provider Recommended a complete shift in information architecture and re-

categorization of offerings to better meet the needs of site users– Based on competitive benchmarking, buy flow analysis, customer

survey

Segmentation: business/personal use; web site maintenance experience

Remote open card sort (www.websort.net) recommended to determine:– Market segment variances in user groupings of particular items and

topic category naming

– Nomenclature participants used to describe the topic categories

Initial scope: 30 “recruited” participants – Limited budget for user research “How can we get user data in one

day?”

– Additional segmentation: 4 geographic markets

Outcome:– Customer felt confident in recommendations based on competitive

benchmarking report and internal stakeholder review

– Used card sort as an internal IA tool

Other Available Options

Remote User Testing Options WebEffective

– Including mobile

Usertesting.com– Data reported: Tasks; participant videos, summaries,

ability to “watch” keystrokes/mouse movements/clicks– Use their panel or your own customers; $39 per user

Loop11– Data reported: Task completion rate; Time per task; Most common success page;

Most common fail page; Most common first click; Most common navigation path; Detailed participant path analysis; Number of page views to complete tasks

– $350 up to 1,000 participants, unlimited tasks/questions– http://konigi.com/tools/submissions/loop11-online-unmoderated-user-testing

UserZoom– Including mobile– Data reported: Effectiveness ratios; Efficiency ratios; Click-stream paths; Click-

mapping; Users' suggestions & feedback; Satisfaction & perception indicators; Cluster analysis & dendograms

– www.userzoom.com

Online Focus Group Options

FacilitatePro– http://www.facilitate.com/solutions/focus-groups.html

Artafact – http://www.artafact.com/online-focus-groups.html#

Invoke – http://www.invoke.com/index/products_online_focusgroups

Itracks – http://www.itracks.com/Products/OnlineFocusGroup/tabid/73/Default.

aspx

Qual-vu– Online focus groups incorporating Video Diaries– http://www.qualvu.com/video_diary

Survey Tools

iModerate– http://www.imoderate.com/

ForeSee– Ability to tie satisfaction ratings to ROI based on proven

American Customer Satisfaction index– Initial survey during unique user session– Ability to track impact of satisfaction on future behaviors

(true conversion numbers based on cookie tracking)– http://foreseeresults.com/

IA validation tools

WebSort.net– Online Card Sort

UserZoom– Online Card Sort– www.userzoom.com

Optimal Workshop– Online Card Sort– Site Map testing – Task-based “click test” of mockup or screenshot– http://www.optimalworkshop.com/

Case Study 1: Test development and implementation

Test Design:– Test Kit development identical to traditional usability testing

• More consideration of user fatigue in completing “repetitive” follow-up questions.

– Additional UX time to:• Input tasks and questions into test system and define test logic, including

“minimum” participation requirements• Coordinate implementation of technical code on client site• QA and optimize test for online experience• 33 additional hours

Live Testing Period:– Monitor participation on a daily basis at a minimum

• Quotas; “auto-disqualified” participants; drop-off

Case Study 1: Participant Data Number of participants:

– Some navigations not captured– Some unofficial task “give-ups” due to lack of interest/time/other?– Inconsistent levels of participation

(need to scrub the reportable data even further)– Segment 1: 0– Segment 2: 2– Segment 3: 25 Better reflection of trends– Segment 4: 173 Easy to identify patterns/trends– Segment 5: 11 Starting to see trends/patterns– Segment 6: 10 – Segment 7: 3

– Jakob Nielson knows how many users are enough• http://www.useit.com/alertbox/20000319.html Qualitative Studies• http://www.useit.com/alertbox/quantitative_testing.html Quantitative

Studies