how to effectively implement different online research techniques for rapid unmoderated feedback -...
TRANSCRIPT
How to effectively implement different online research techniques for rapid unmoderated feedback
Niyati Bedekar@nbedekar
Steve Fadden @sfadden
Presented at UXPA 2015, San Diego Slides: https://goo.gl/X8dolV
Agenda
Online techniques
Method toolkit
Common requests and solutions
Case studies and templates
Effective practices
Image source: http://pixabay.com/en/modesto-california-scenic-trail-205544/
Introductions
Steve Fadden Niyati Bedekar
University of Pune
Who We Are
Who are you?Years experience in
user research:
<11-22-55+
Image source: Karen Arnold (http://www.publicdomainpictures.net/view-image.php?image=45018)
Who are you?Total number of
employees:
1-20 21-100101-500500+
Image source: Karen Arnold (http://www.publicdomainpictures.net/view-image.php?image=45018)
Who are you?
Most recent research request?
Most common research request?
Jot down
Image source: http://pixabay.com/en/photos/note%20paper/
Online methods, especially asynchronous
Confession: Agility over comprehensiveness
Image source: https://www.flickr.com/photos/smithser/3735204251
Variety of Online Methods
Image source: https://flic.kr/p/jZUByi
Methods toolkit
What People Do
What People Say
Why & How to Fix
How Many & How Much
Behavioral
Attitudinal
Qualitative Quantitative
Rohrer, C. October 12, 2014. When to use which user experience research methods. Retrieved from http://www.nngroup.com/articles/which-ux-research-methods/
Toolkit is growing(Rohrer’s framework)
Image source: http://www.freestockphotos.biz/stockphoto/1772
Method (participant effort) Types of answers provided
Click Behavioral: Where to start or go next?
Preference Attitudinal: Compare between options
Recall Hybrid: What do you remember? What are your first impressions?
Sentiment Attitudinal: How does this make you feel?
Embedded questions Hybrid: What happens next, and why? How would you rate this?
Terminology/naming Attitudinal: What does something mean?
Commenting Hybrid: What comes to mind while reviewing a concept/flow? OR Open feedback
Go-to methods
Image source: http://www.geograph.org.uk/photo/1911269
Method (participant effort) Types of answers provided
Card sorting Hybrid: What items belong together and what should they be called?
Discussion groups / Focus groups
Attitudinal: What comes to mind while reviewing other feedback?
Unmoderated usability testing
Hybrid: What do you expect? What do you do? Why?
Additional methods to consider
Image source: http://www.geograph.org.uk/photo/1911269
Sample mockups used
“Finals week starts on June 1. Where would you first click to put a reminder on your calendar?”
Click methods (Behavior: Where do users click)
UsabilityTools
“Describe what you would expect to see after you clicked the area in the previous screen?”
Embedded question (Hybrid: What happens next)
Qualtrics
“Please click the variation you prefer. [after] Why did you choose it?”
Preference (Attitude: Which do you prefer)
Verify
“You will see a screen for 5 seconds. After reviewing the screen, you’ll be asked questions about it. [after] What do you remember?”
Recall (Hybrid: What do you remember)
Verify
“Review this screen and think about how it makes you feel.”
Sentiment (Attitude: How does this make you feel)
Verify
“Do you find this design to be attractive?”
Embedded question (Attitude: How do you rate this)
SurveyMonkey
“Label each marker with what you would call the icon.”
Terminology/naming (Attitude: What does this mean)
Verify
“This design shows what happens when you click the ‘+’ icon. Comment on areas you find confusing, problematic, helpful, usable.”
Commenting (Hybrid: What comes to mind)
Verify
Fast
Convenient
Focused
↑ People
↓ Resources
Benefits
Image source: http://commons.wikimedia.org/wiki/File:Double-alaskan-rainbow-airbrushed.jpg
Context
Environment
Participant profiles
Tasks
Responses
Challenges
Image source: http://commons.wikimedia.org/wiki/File:Angela-Whyte-Hurdle-Posed.jpg
Activity
Form groups of 3-5Review common requestsDiscuss how you typically researchConsider online solutionsDiscuss pros/cons
Discussion: Research requests
Image source: http://en.wikipedia.org/wiki/Fischer's_lovebird
Reference (for Activity)
Image source: http://www.geograph.org.uk/photo/1911269
Method (participant effort) Types of answers provided
Click Behavioral: Where to start or go next?
Preference Attitudinal: Compare between options
Recall Hybrid: What do you remember? What are your first impressions?
Sentiment Attitudinal: How does this make you feel?
Embedded questions Hybrid: What happens next, and why? How would you rate this?
Terminology/naming Attitudinal: What does something mean?
Commenting Hybrid: What comes to mind while reviewing a concept/flow? OR Open feedback
Group discussion: Share thoughts● Problem● Typical solution● Online research solution● Pros/cons
Discussion: Research requests
Image source: http://en.wikipedia.org/wiki/Fischer's_lovebird
Case Studies and Templates
Case Study 1: Evaluate new data export concept
Background
- New functionality for an existing product- Integrated with 3rd party software- To be implemented ASAP
Goals
- “Boil the ocean” to learn if concept was understood, desired, and usable
Methods
Embedded questionCritical incident
Embedded question Comprehension rating
CommentingOn each storyboard panel, after presenting full story
Embedded questionOpen feedback, questions, and expectations
“Consider the last time you had to export data. Describe why you needed to export data, and list the steps you remember from that process. (If you haven’t exported data before, or don’t remember the last time, just skip to the next question).”
Embedded question (Critical Incident)
“I’m pretty old school, so I export my credit card transaction data about every quarter. My credit card site has a button to export to CSV, so I just click that and it downloads to my computer.”
“We have our marketing, sales, and inventory data in different systems. I have to export data from each system in order to combine it into a spreadsheet for my stakeholders. The export process is easy. Combining the data is more involved.
“Consider the concept presented on the next 4 slides. After reading about the concept, you will be asked about what you found to be confusing, problematic, useful, and appealing about the concept.”
1.
New concept scenario
2.
3.
4.
100%
“How understandable is this concept?”
Embedded question (Comprehension)
Commenting (Identify strengths and weaknesses)
“You will now be shown each concept slide again. On each slide, indicate anything you found to be particularly confusing, problematic, useful, and appealing.”
1.
2.
3.
4.
100%“Doing this would require a lot of clicks, even for a small number of columns.”
“You should embed best practices for naming here. Otherwise, the result could be messy.”
“Will we be able to save the mappings? That could save time in the future.”
“Any final comments, questions, or feedback you’d like to share?”
Embedded question (Open feedback)
“It’s great that you don’t have to jump around different parts of the system to do this. Very valuable to be able to complete this from one place.”
“Seems very clear to me. I think anyone who has used [XYZ] would be able to understand it too.”
“Hi, I wanted to follow up to reiterate that this is a REALLY COOL idea and it fills a much needed requirement for our use of the product. Please consider me for future studies like this, because we need this functionality!”
Template 1: Exploring a new concept
NDA,Confidentiality, Demographics
Embedded Question: Critical incident to activate
[Present concept]Video, illustration, storyboard, description
Embedded Question: Comprehension rating, after presenting concept
Commenting: Concept slides (storyboards work well)
Embedded Question: Open feedback
Case Study 2: Identify problems and preferences for calendar range selection tools
Background
- Tool developed without support- Early stage prototype, only worked within company firewall- Team wanted feedback before further refinement
Goals
- Recruit internal participants only- Identify heuristic violations- Gauge preference compared to existing tools
Methods
ClickHow would you start task
Commenting(after using prototype) See screenshots of tool in different states
PreferenceCompare tool to existing tool
Embedded QuestionExplain preference and next steps
Template 2: Eliciting usability/heuristic feedback
NDA,Confidentiality, Demographics
Recall: What is remembered? [or]
Sentiment: How does this make you feel?
Click: How would you start this task?
Embedded Question: What would you expect to see after clicking?
Commenting: Open feedback, after engaging
Embedded Question: Usability rating
Case Study 3: Redesign chart type & update visual treatment
Background
- Existing component used frequently by customers and loved by many!- Not scalable- Prone to misinterpretation- Team wanted to test new designs
Goals
- Understand if users comprehend the new design- Gauge preference among 3 different approaches (including existing)- Mix of internal users and customers
Methods
Embedded questionUnderstandability of information
PreferenceAmong the various options
CommentingOpen feedback, expectations
Template 3: Redesigned visual treatment
NDA,Confidentiality
Embedded Question: to gather understanding of information on chart (randomize)
Preference: Which design do you prefer? (randomize)
Embedded Question: Why the selected design?
Commenting: Open feedback
Demographics
Case Study 4: Understand how people find content
Background
- Team assigned to build new system- Wanted to create a system where content was easy to locate
Goals
- Identify how users locate content- Discover differences based on content type- Understand pain points to see if they can be reduced or eliminated
Methods
Click(for each method) Where do you click first to locate this kind of content?
SentimentWhat feeling is associated?
CommentingOpen feedback, expectations
Embedded Question(after each method) What do you find most/least usable?
Template 4: Understanding behavior and expectations
NDA,Confidentiality, Demographics
Embedded Question: Critical incident to activate
Click: What do you do first?
Sentiment: How do you feel when you do this?
Commenting: What works well and not well?
Embedded Question: Open feedback
Effective Practices
Maintain your own panel; build a snowball
Image source: https://flic.kr/p/aZhJF
Image source: https://flic.kr/p/qSsTmF
Recruit from social media, communities, classifieds & search
Match screening to goals and use creative incentives
Image source: http://www.pexels.com/photo/numbers-money-calculating-calculation-3305/
Protect confidentiality and collect useful demographics
Image source: https://flic.kr/p/8xzAnc
Image source: http://pixabay.com/en/mark-marker-hand-leave-516279/
Order questions intentionally, limit required questions and total number
Image source: https://flic.kr/p/oWyYTz
Launch a pilot study, but remove data to save time later
Provide follow-up channels
Image source: http://commons.wikimedia.org/wiki/File:Old_British_telephones.jpg
Other hints and tips?
Image source: https://flic.kr/p/6yoj2L
Final Considerations & Surprises
People really do participate
Image source: https://flic.kr/p/g9agMi
Engagement and response quality are surprisingly high
Image source: https://flic.kr/p/hfWrxQ
Incentives don’t need to be high; no incentive could be the right price
Image source: http://commons.wikimedia.org/wiki/File:Money_Cash.jpg
Many participants (and researchers) want follow-up opportunities
Image source: https://flic.kr/p/7qcudQ
Triangulation is critical
Image source: https://flic.kr/p/2NJxPz
Thank you
Questions? Answers?
(Please leave your note with common research requests)
Additional Resources
Type of Test
Tools Click / Suc- cess
Prefer- ence
Recall Senti- ment
Ques- tion
Termin- ology/ Label
Com- menting
Card sorting
Discus- sion
Unmoder- ated usability + video on website
Metrics & Results
Verify ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Usabilla ✓ ✓ ✓ ✓
Loop11 ✓ ✓ ✓
UserTesting.com ✓ ✓
UserZoom ✓ ✓ ✓ ✓ ✓ ✓
Optimal Workshop
✓ ✓
Yahoo Groups, Facebook, LinkedIn
✓
Survey tools (Getfeedback, Qualtrics, SurveyMonkey)
✓ ✓ ✓ ✓
Examples of types of tests available (Incomplete list)
Chrisitan Rohrer’s NNG article about when appropriatenes of a method to help answer specific questions: http://www.nngroup.com/articles/which-ux-research-methods/
A review of usability and UX testing tools: http://www.smashingmagazine.com/2011/10/20/comprehensive-review-usability-user-experience-testing-tools/
How to select an unmoderated user testing tool to fit your needs: http://www.nngroup.com/articles/unmoderated-user-testing-tools/
List of tools for unmoderated testing:
1. http://remoteresear.ch/tools/2. http://www.infragistics.com/community/blogs/ux/archive/2012/11/07/6-tools-for-remote-
unmoderated-usability-testing.aspx
Kyle Soucy’s article in UX Matters (Unmoderated, Remote Usability Testing: Good or Evil?) http://www.uxmatters.com/mt/archives/2010/01/unmoderated-remote-usability-testing-good-or-evil.php
Additional Links
Icon Sources
http://www.flaticon.com/authors/freepik
http://www.flaticon.com/authors/icomoon
http://www.flaticon.com/authors/google
http://www.flaticon.com/authors/anton-saputro
http://creativecommons.org/licenses/by/3.0/
http://www.flaticon.com/authors/plainicon