Using Automated Testing Tools to Empower Your User Research
Post on 15-Jan-2017
Automated User Testing
Presented by Chrys SullivanFounder and CEO, Useagility
4149 Pennsylvania, Suite 202Kansas City, MO 64111
An educational program sponsored and supported by User Experience Professionals Association.Using Automated Testing Tools to Empower Your User ResearchJoin now at www.uxpa.org for member-only pricingUXPA Seattle Conference 2016May 31 June 3uxpa2016.org
Rapidly Test Usability & Measure UXUserZoom is the All-in-One UX Research Platform for the Data-Driven Enterprise
Chat box is available if you have any questionsLook out for a quick poll questionThere will be time for Q&A at the endWe will be recording the webinar for future viewingAll attendees will receive a copy of the slides/recordingContinue the discussion using #UZWebinarQuick Housekeeping
Lets understand each others background, roles and experience levelsPolls
About Useagility 6Twitter: @Useagility
Today well learn: 7How unmoderated testing works and how you can use it in your organization. 01Guidelines for selecting the right testing methodology (moderated vs. unmoderated) based on your project needs and goals. 02Steps for setting up successful unmoderated studies.03Advice for recruitment, screening and use of online panels. 05Tips for avoiding the most common pitfalls in automated UX testing.04Using automated testing with agile design and development sprints to accommodate tight timelines and satisfy usability needs.06
Evolution of Usability Testing Technology 8Lab-based testing Moderated testingIn lab setting
Remote, facilitated testingModerated testing Human facilitator and participant interact via web-conference
Automated, un-facilitated testing- Unmoderated studies conducted online via web-based applications No human facilitatorTasks are pre-set by researcher, delivered through testing interface
Benefits of Unmoderated
9Easier recruitment and scheduling No worries about no-shows & replacements
Automated incentive payments
Simultaneously test larger number of participants (Quantitative) Less expensive per-study costs than moderated testing
Get results more quickly
Built-in metrics (time on task, heat maps, rating scale, SUS, etc)
Our Favorite Ways to Use Unmoderated User Testing
10Help Teams Fail Fast Testing design wireframes and prototypes (Axure, Invision) Quickly weed out approaches that dont work Resolve internal disagreement Let users decide which design works bestSupport Expert Review / Heuristics Evaluations Validate suspected usability issues
Lean UX Research with Agile Quickly test prototypes as part of agile development cycles
Prioritize backlog Measure usability impact for dev prioritization
Click-through / conversion testing
Limitations & Pitfalls of Unmoderated
11User panels and professional testers
Stakeholders cant observe live during the study
Cant revise tasks once the study has been launched
Cant adjust approach as sessions progress to optimize depth of learning
Lacks ability to probe unexpected insights
No real-time support for users who get significantly off-track
ITS NOT MODERATED
Selecting right methodology 12Quantitative or Qualitative? Moderated or Unmoderated? Selecting the right methodology has the greatest impact on quality and effectiveness of the study.
Define Goals13Hard to get business stakeholders to define their goals. Ask them questions:
What is the primary purpose of the study?
What do you need to know coming out of this study?
What will you do with the study findings?
Define additional details14After goals are defined, understand other key components:
What is the scope of study? (Big, small, how focused) What are we testing with? (Prototype, wireframes, live site) Who are the target users? What tasks and scenarios will help achieve study goals?
Which methodology is best?
15Will you need moderated follow-up and probing questions to be successful?
Are you at early concept and UX strategy phase?
Do you need to go deep on attitudinal findings, mental model and user preferences?
Is it the first time you have tested a design? First time testing with a new user type?
Will it take longer than 20 30 minutes to complete tasks?
If yes = Moderated
Which methodology is best?
16Is the scope narrowly focused?
Can testing tasks be accomplished in 30 min or less?
Is testing of task success and usability a primary objective vs. diving deep on user attitudes and preferences?
Is there minimal reliance on attitudinal feedback that requires follow up probing?
Is this follow-on testing after moderated?
If Yes = Unmoderated
Creating successful test plans for unmoderated, automated usability testing has unique challenges. Key Steps - Unmoderated Automated Usability TestingPARTIPANTSTASKS FOLLOW-UP QUESTINSANALYSIS Get right participantsDesign study and tasks effectivelyUse follow-ups appropriatelyAnalyze finings accurately
Many ways to recruit for unmoderated: 01Email existing user base
Panel provide by UX testing vendor
*In all cases, screening for representative sample is very important*
PITFALL #1: Panels
PITFALL#1: Professional Testers and Imposters SOLUTION: Over-screen 0101Use 2x the screener questions as normal
As first task in study, ask a detailed question to verify user is who they say
Over-recruit so you can throw out participants you feel may not be valid
Crafting study plan & writing tasks 01Remember, you wont be there to help or clarifyAvoid leading verbiage that may cause bias in task successInstead of: View your current grades in AlgebraUse: Imagine you want to know your current grade in Algebra
3. Ensure prototypes enable users to complete tasks 4. Anticipate how to proceed if they fail a task 5. Determine what metrics or data will be meaningful given goals and context
Use follow-up questions
01Use follow-up questions to check for understanding
Example: If the task was to locate their weighted-average grade in Algebra class, ask what their weighted-average means.
Use follow-up questions to probe for deeper insights on why a user behaved as they did
Example: If the task required multiple steps in a process, ask were there parts of that process you were unsure about or found confusing, tell me more about why you felt that way.
PITFALL #2 Tasks skipped, misunderstood, leading
PITFALL #2 skipped, misunderstood and leading tasksSOLUTION Conduct pilot session 0102Conduct one session as pilot
Adjust tasks and flow, add follow up questions to understand why if necessary
Analyzing unmoderated testing 01Watch all sessions in full (run at 1.5 speed)
Use integrated tools to tag video clips and make notes
Analysis - Getting Lazy!
PITFALL #3 Relying too heavily on metricsSOLUTION: Use data appropriately and in context 0103Question self-reported task success rates Balance stated preference for design with analysis of user behavior
Give thought to which metrics are applicable and relevant to your study based on contextTime-on-task, heat maps, conversion, click-path
25Wrapping It Up
Now You Know26What unmoderated testing is and how to use it 01Importance of selecting right methodology: moderated or unmoderated02Key steps to setting up and running successful unmoderated testing 03Common pitfalls in unmoderated testing and ways to avoid them 04
Thank You!27Thank you for joining us today!
Useagility contact info:www.Useagility.com
Need more info?Get answers about UserZoom. We are here to help. Talk with one of our knowledgeable staff.Contact usUserZoom10 Almaden Boulevard, Suite 250San Jose, CA 95113
Toll free number: (866) 599 1550