using automated testing tools to empower your user research

28
4149 Pennsylvania, Suite 202 Kansas City, MO 64111 Automated User Testing Presented by Chrys Sullivan Founder and CEO, Useagility

Upload: userzoom

Post on 15-Jan-2017

1.054 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Using Automated Testing Tools to Empower Your User Research

4149 Pennsylvania, Suite 202Kansas City, MO 64111

Automated User Testing

Presented by

Chrys SullivanFounder and CEO, Useagility

Page 2: Using Automated Testing Tools to Empower Your User Research

An educational program sponsored and supported by User Experience Professionals

Association.

Using Automated Testing Tools to Empower Your

User Research

Join now at www.uxpa.org for member-only pricing

UXPA Seattle Conference 2016May 31 – June 3uxpa2016.org

Page 3: Using Automated Testing Tools to Empower Your User Research

Rapidly Test Usability & Measure UXUserZoom is the All-in-One UX Research Platform for the Data-Driven

Enterprise

Page 4: Using Automated Testing Tools to Empower Your User Research

• Chat box is available if you have any questions

• Look out for a quick poll question

• There will be time for Q&A at the end

• We will be recording the webinar for future viewing

• All attendees will receive a copy of the slides/recording

• Continue the discussion using #UZWebinar

Quick Housekeeping

Page 5: Using Automated Testing Tools to Empower Your User Research

Let’s understand each other’s background, roles and experience levelsPolls

Page 6: Using Automated Testing Tools to Empower Your User Research

About Useagility

6

Twitter: @Useagility

[email protected]

www.Useagility.com

Page 7: Using Automated Testing Tools to Empower Your User Research

Today we’ll learn:

7

How unmoderated testing works

and how you can use it in your organization.

01

Guidelines for selecting the right testing methodology (moderated vs.

unmoderated) based on your project needs and goals.

02

Steps for setting up successful unmoderated

studies.

03

Advice for recruitment, screening and use of online

panels.

05

Tips for avoiding the most common pitfalls in

automated UX testing.

04

Using automated testing with agile design and

development sprints to accommodate tight timelines and satisfy usability needs.

06

Page 8: Using Automated Testing Tools to Empower Your User Research

Evolution of Usability Testing Technology

8

Lab-based testing - Moderated testing- In lab setting

Remote, facilitated testing- Moderated testing - Human facilitator and

participant interact via web-conference

Automated, un-facilitated testing- Unmoderated studies conducted online via web-based applications • No human facilitator• Tasks are pre-set by researcher,

delivered through testing interface

Page 9: Using Automated Testing Tools to Empower Your User Research

Benefits of Unmoderated

9

Easier recruitment and scheduling No worries about no-shows &

replacements

Automated incentive payments

Simultaneously test larger number of participants (Quantitative)

Less expensive per-study costs than moderated testing

Get results more quickly

Built-in metrics (time on task, heat maps, rating scale, SUS, etc)

Page 10: Using Automated Testing Tools to Empower Your User Research

Our Favorite Ways to Use Unmoderated User Testing

10

Help Teams “Fail Fast” Testing design wireframes and

prototypes (Axure, Invision) Quickly weed out approaches that

don’t work

Resolve internal disagreement Let users decide which design works

best

Support Expert Review / Heuristics Evaluations

Validate suspected usability issues

Lean UX Research with Agile Quickly test prototypes as part of agile

development cycles

Prioritize backlog Measure usability impact for dev prioritization

A/B testing

Click-through / conversion testing

Page 11: Using Automated Testing Tools to Empower Your User Research

Limitations & Pitfalls of Unmoderated

11

User panels and “professional testers”

Stakeholders can’t observe live during the study

Can’t revise tasks once the study has been launched

Can’t adjust approach as sessions progress to optimize depth of learning

Lacks ability to probe unexpected insights

No real-time support for users who get significantly off-track

IT’S NOT MODERATED

Page 12: Using Automated Testing Tools to Empower Your User Research

Selecting right methodology

12

Quantitative or Qualitative?

Moderated or Unmoderated?

Selecting the right methodology has the greatest impact on quality and effectiveness of the study.

Page 13: Using Automated Testing Tools to Empower Your User Research

Define Goals

13

Hard to get business stakeholders to define their goals. Ask them questions:

• What is the primary purpose of the study?

• What do you need to know coming out of this study?

• What will you do with the study findings?

Page 14: Using Automated Testing Tools to Empower Your User Research

Define additional details

14

After goals are defined, understand other key components:

• What is the scope of study? (Big, small, how focused)

• What are we testing with? (Prototype, wireframes, live site)

• Who are the target users? • What tasks and scenarios will help

achieve study goals?

Page 15: Using Automated Testing Tools to Empower Your User Research

Which methodology is best?

15

Will you need moderated follow-up and probing questions to be successful?

Are you at early concept and UX strategy phase?

Do you need to go deep on attitudinal findings, mental model and user preferences?

Is it the first time you have tested a design? First time testing with a new user type?

Will it take longer than 20 – 30 minutes to complete tasks?

If yes = Moderated

Page 16: Using Automated Testing Tools to Empower Your User Research

Which methodology is best?

16

Is the scope narrowly focused?

Can testing tasks be accomplished in 30 min or less?

Is testing of task success and usability a primary objective vs. diving deep on user attitudes and preferences?

Is there minimal reliance on attitudinal feedback that requires follow up probing?

Is this follow-on testing after moderated?

If Yes = Unmoderated

Page 17: Using Automated Testing Tools to Empower Your User Research

Creating successful test plans for unmoderated, automated usability testing has unique challenges.

Key Steps - Unmoderated Automated Usability Testing

PARTIPANTS TASKS FOLLOW-UP QUESTINS ANALYSIS

Get right participants Design study and tasks effectively

Use follow-ups appropriately

Analyze finings accurately

Page 18: Using Automated Testing Tools to Empower Your User Research

Participants

18

Many ways to recruit for unmoderated: 011. Email existing user base

2. Social media

3. Site intercept

4. Panel provide by UX testing vendor

*In all cases, screening for representative sample is very important*

Page 19: Using Automated Testing Tools to Empower Your User Research

PITFALL #1: Panels

19

PITFALL#1: “Professional Testers” and “Imposters”

SOLUTION: Over-screen

0101

1. Use 2x the screener questions as normal

2. As first task in study, ask a detailed question to verify user is who they say

3. Over-recruit so you can throw out participants you feel may not be valid

Page 20: Using Automated Testing Tools to Empower Your User Research

Writing Tasks

20

Crafting study plan & writing tasks 01

1. Remember, you won’t be there to help or clarify

2. Avoid leading verbiage that may cause bias in task success

Instead of: View your current grades in Algebra

Use: Imagine you want to know your current grade in Algebra

3. Ensure prototypes enable users to complete tasks

4. Anticipate how to proceed if they fail a task

5. Determine what metrics or data will be meaningful given goals and context

Page 21: Using Automated Testing Tools to Empower Your User Research

Use follow-up questions

21

011. Use follow-up questions to check for understanding

Example: If the task was to locate their weighted-average grade in Algebra class, ask what their “weighted-average” means.

2. Use follow-up questions to probe for deeper insights on why a user behaved as they did

Example: If the task required multiple steps in a process, ask “were there parts of that process you were unsure about or found confusing, tell me more about why you felt that way.”

Page 22: Using Automated Testing Tools to Empower Your User Research

PITFALL #2 – Tasks skipped, misunderstood, leading

22

PITFALL #2 – skipped, misunderstood and leading tasks

SOLUTION – Conduct pilot session

0102

1. Conduct one session as pilot

2. Watch results

3. Adjust tasks and flow, add follow up questions to understand “why” if necessary

Page 23: Using Automated Testing Tools to Empower Your User Research

Analysis

23

Analyzing unmoderated testing 01

1. Watch all sessions in full • (run at 1.5 speed)

2. Use integrated tools to tag video clips and make notes

Page 24: Using Automated Testing Tools to Empower Your User Research

Analysis - Getting Lazy!

24

PITFALL #3 – Relying too heavily on metrics

SOLUTION: Use data appropriately and in context

0103

1. Question self-reported task success rates

2. Balance stated “preference” for design with analysis of user behavior

3. Give thought to which metrics are applicable and relevant to your study based on context

• Time-on-task, heat maps, conversion, click-path

Page 25: Using Automated Testing Tools to Empower Your User Research

25

Wrapping It Up

Page 26: Using Automated Testing Tools to Empower Your User Research

Now You Know

26

What unmoderated testing is and how to use it

01

Importance of selecting right methodology: moderated or unmoderated

02

Key steps to setting up and running successful unmoderated testing

03

Common pitfalls in unmoderated testing and ways to avoid them

04

Page 27: Using Automated Testing Tools to Empower Your User Research

Thank You!

27

Thank you for joining us today! Useagility contact info:

www.Useagility.com

[email protected]

Twitter: @Useagility

Page 28: Using Automated Testing Tools to Empower Your User Research

Need more info?Get answers about UserZoom. We are here to help. Talk with one of our knowledgeable staff.

Contact us

UserZoom10 Almaden Boulevard, Suite 250San Jose, CA 95113

Toll free number: (866) 599 1550

www.linkedin.com/company/userzoom@UserZoomFollow us