using automated testing tools to empower your user research

Download Using Automated Testing Tools to Empower Your User Research

Post on 15-Jan-2017




0 download

Embed Size (px)


PowerPoint Presentation

Automated User Testing

Presented by Chrys SullivanFounder and CEO, Useagility

4149 Pennsylvania, Suite 202Kansas City, MO 64111

An educational program sponsored and supported by User Experience Professionals Association.Using Automated Testing Tools to Empower Your User ResearchJoin now at for member-only pricingUXPA Seattle Conference 2016May 31 June

Rapidly Test Usability & Measure UXUserZoom is the All-in-One UX Research Platform for the Data-Driven Enterprise

Chat box is available if you have any questionsLook out for a quick poll questionThere will be time for Q&A at the endWe will be recording the webinar for future viewingAll attendees will receive a copy of the slides/recordingContinue the discussion using #UZWebinarQuick Housekeeping

Lets understand each others background, roles and experience levelsPolls

About Useagility 6Twitter: @Useagility

Today well learn: 7How unmoderated testing works and how you can use it in your organization. 01Guidelines for selecting the right testing methodology (moderated vs. unmoderated) based on your project needs and goals. 02Steps for setting up successful unmoderated studies.03Advice for recruitment, screening and use of online panels. 05Tips for avoiding the most common pitfalls in automated UX testing.04Using automated testing with agile design and development sprints to accommodate tight timelines and satisfy usability needs.06


Evolution of Usability Testing Technology 8Lab-based testing Moderated testingIn lab setting

Remote, facilitated testingModerated testing Human facilitator and participant interact via web-conference

Automated, un-facilitated testing- Unmoderated studies conducted online via web-based applications No human facilitatorTasks are pre-set by researcher, delivered through testing interface

Benefits of Unmoderated

9Easier recruitment and scheduling No worries about no-shows & replacements

Automated incentive payments

Simultaneously test larger number of participants (Quantitative) Less expensive per-study costs than moderated testing

Get results more quickly

Built-in metrics (time on task, heat maps, rating scale, SUS, etc)

Our Favorite Ways to Use Unmoderated User Testing

10Help Teams Fail Fast Testing design wireframes and prototypes (Axure, Invision) Quickly weed out approaches that dont work Resolve internal disagreement Let users decide which design works bestSupport Expert Review / Heuristics Evaluations Validate suspected usability issues

Lean UX Research with Agile Quickly test prototypes as part of agile development cycles

Prioritize backlog Measure usability impact for dev prioritization

A/B testing

Click-through / conversion testing

Limitations & Pitfalls of Unmoderated

11User panels and professional testers

Stakeholders cant observe live during the study

Cant revise tasks once the study has been launched

Cant adjust approach as sessions progress to optimize depth of learning

Lacks ability to probe unexpected insights

No real-time support for users who get significantly off-track


Selecting right methodology 12Quantitative or Qualitative? Moderated or Unmoderated? Selecting the right methodology has the greatest impact on quality and effectiveness of the study.

Define Goals13Hard to get business stakeholders to define their goals. Ask them questions:

What is the primary purpose of the study?

What do you need to know coming out of this study?

What will you do with the study findings?

Define additional details14After goals are defined, understand other key components:

What is the scope of study? (Big, small, how focused) What are we testing with? (Prototype, wireframes, live site) Who are the target users? What tasks and scenarios will help achieve study goals?

Which methodology is best?

15Will you need moderated follow-up and probing questions to be successful?

Are you at early concept and UX strategy phase?

Do you need to go deep on attitudinal findings, mental model and user preferences?

Is it the first time you have tested a design? First time testing with a new user type?

Will it take longer than 20 30 minutes to complete tasks?

If yes = Moderated

Which methodology is best?

16Is the scope narrowly focused?

Can testing tasks be accomplished in 30 min or less?

Is testing of task success and usability a primary objective vs. diving deep on user attitudes and preferences?

Is there minimal reliance on attitudinal feedback that requires follow up probing?

Is this follow-on testing after moderated?

If Yes = Unmoderated

Creating successful test plans for unmoderated, automated usability testing has unique challenges. Key Steps - Unmoderated Automated Usability TestingPARTIPANTSTASKS FOLLOW-UP QUESTINSANALYSIS Get right participantsDesign study and tasks effectivelyUse follow-ups appropriatelyAnalyze finings accurately



Many ways to recruit for unmoderated: 01Email existing user base

Social media

Site intercept

Panel provide by UX testing vendor

*In all cases, screening for representative sample is very important*

PITFALL #1: Panels


PITFALL#1: Professional Testers and Imposters SOLUTION: Over-screen 0101Use 2x the screener questions as normal

As first task in study, ask a detailed question to verify user is who they say

Over-recruit so you can throw out participants you feel may not be valid


Writing Tasks


Crafting study plan & writing tasks 01Remember, you wont be there to help or clarifyAvoid leading verbiage that may cause bias in task successInstead of: View your current grades in AlgebraUse: Imagine you want to know your current grade in Algebra

3. Ensure prototypes enable users to complete tasks 4. Anticipate how to proceed if they fail a task 5. Determine what metrics or data will be meaningful given goals and context

Use follow-up questions


01Use follow-up questions to check for understanding

Example: If the task was to locate their weighted-average grade in Algebra class, ask what their weighted-average means.

Use follow-up questions to probe for deeper insights on why a user behaved as they did

Example: If the task required multiple steps in a process, ask were there parts of that process you were unsure about or found confusing, tell me more about why you felt that way.

PITFALL #2 Tasks skipped, misunderstood, leading


PITFALL #2 skipped, misunderstood and leading tasksSOLUTION Conduct pilot session 0102Conduct one session as pilot

Watch results

Adjust tasks and flow, add follow up questions to understand why if necessary



Analyzing unmoderated testing 01Watch all sessions in full (run at 1.5 speed)

Use integrated tools to tag video clips and make notes

Analysis - Getting Lazy!


PITFALL #3 Relying too heavily on metricsSOLUTION: Use data appropriately and in context 0103Question self-reported task success rates Balance stated preference for design with analysis of user behavior

Give thought to which metrics are applicable and relevant to your study based on contextTime-on-task, heat maps, conversion, click-path

25Wrapping It Up

Now You Know26What unmoderated testing is and how to use it 01Importance of selecting right methodology: moderated or unmoderated02Key steps to setting up and running successful unmoderated testing 03Common pitfalls in unmoderated testing and ways to avoid them 04

Thank You!27Thank you for joining us today!

Useagility contact

Twitter: @Useagility

Need more info?Get answers about UserZoom. We are here to help. Talk with one of our knowledgeable staff.Contact usUserZoom10 Almaden Boulevard, Suite 250San Jose, CA 95113

Toll free number: (866) 599 1550 us