ui/ux foundations - research
TRANSCRIPT
Our Goals
We want you to…
talk confidently to UX researchers
critically understand research presented to you
conduct basic UX research on your own
have a basis to continue learning about UX research
Activity (setup)Which of these problems do you feel strongly about?
Pittsburgh public transportation
Food delivery in Pittsburgh
Finding family-friendly activities in Pittsburgh
Pittsburgh public schools
Agenda09:00 - 09:20 Breakfast and Introductions
09:20 - 09:30 User-Centered Design
09:30 - 10:10 Surveys, Diary Studies, Interviews
10:10 - 10:55 Usability Studies, Field Studies
10:55 - 11:25 A/B Testing, Log Analysis
11:25 - 12:00 Adapting Your Methods
12:00 - 12:30 LUNCH
12:30 - 01:00 Interpreting Your Data
01:00 - 01:40 Special Topics: Dark UX Patterns
01:40 - 01:50 Case Studies
01:50 - 02:00 Group reflections & wrap-up Q&A
User-centered design (n.) - a framework of processes in which the needs, wants, and limitations of end users of a product are given extensive attention at each stage of the design process.
https://en.wikipedia.org/wiki/User-centered_design
UX Research & Design
Refine Build Learn
Data Collection, Validation, Evaluation
A/B Testing Contextual Inquiry Diary Studies Ethnography Field Studies
Focus Groups Hallway Testing Heuristic Evaluation Interviews Lab Testing
Log Analysis Remote Testing Think-Aloud Surveys User Observation
Need Finding
Refine Build Learn
Data Collection, Validation, Evaluation
A/B Testing Contextual Inquiry Diary Studies Ethnography Field Studies
Focus Groups Hallway Testing Heuristic Evaluation Interviews Lab Testing
Log Analysis Remote Testing Think-Aloud Surveys User Observation
Need Finding
Refine Build Learn
A/B Testing Contextual Inquiry Diary Studies Ethnography Field Studies
Focus Groups Hallway Testing Heuristic Evaluation Interviews Lab Testing
Log Analysis Remote Testing Think-Aloud Surveys User Observation
Data Collection, Validation, Evaluation
are good for learning:
overall impressions
who your users are (demographics)
outstanding opinions
who’s might participate in further research
Surveys
Common Survey Example
How likely is it that you’d recommend [brand] to a friend?
Not at all Likely Neutral Very
Likely
1 2 3 4 5 6 7 8 9 10
Net Promoter ScoreNPS: “research has shown that your NPS® acts as a leading indicator of
growth. If your organization’s NPS is higher than those of your competitors, you will likely outperform the market…”
https://www.netpromoter.com/know/
Surveys: Pros and ConsBenefits
Cheap (in $ and time)
Easy to recruit participants
May receive high response rate
Easy to analyze
Limitations
Limited in type & scope of data
Question interpretation issues
Response bias
Easy to misinterpret or *over*-interpret results
are good for….
day-to-day habits & patterns
when, how, and why they use your product
reflections on real problems encountered and how they were solved
Diary Studies
Diary Studies: Pros and ConsBenefits
A longitudinal scope of data
Get a look at the mundane, every-day interactions and behaviors
Limitations
Costly (in $ and time)
Difficult to recruit participants (& high attrition)
Relies on self-report
are good for….
user’s background
their use of technology
their goals and motivations
their pain points
what problems need to be addressed or solve
Interviews
Interviews: Pros and ConsBenefits
Cheap (in $)
Can target specific users or be opportunistic
Can engage with users personally
Can get the answer to lots of “why” questions
Limitations
Takes a moderately amount of time
Results indicate what people *say* they do (rather than actual behavior)
Interview TipsStart broad, then narrow-in. example: “Overall, how do you think Pittsburgh public transit compares to other cities?”
Ask clarifying questions, and use their words. example: “You said the bus system is hard to predict, could you explain that to me?”
It’s okay to play-dumb. (But be honest.) example: “I’ve never used public transit here. Can you tell me how you would find out the schedules and figure it out how to get downtown from here?”
More Interview TipsAvoid “Yes/No” questions.
Avoid asking about feelings. Ask about behaviors instead.
Don’t number your questions. Organize by topics you want to cover. Be prepared to skip around.
Always be prepared to go off-script.
Ask the question, then pause. Don’t rush to fill silence.
Activity - Part 1Partner with someone who is interested in a different topic than you. You will interview them on their chosen topic.
It’s your job to explore what the needs are and uncover the main issues, feelings, thoughts, and pain-points.
Round 1: (5 minutes) Develop your Script Goal: Individually, develop a rough script that you will use to interview your partner. Start broad to gather overall impressions, then narrow in on specific topic areas. Remember, you are trying to understand overall impression and the biggest pain-points.
Round 2: (20 minutes) Interview (10 minutes each) Goal: Take turns interviewing each other. Be sure to keep notes.
How do you know when you are done conducting interviews?
Saturation (n.) - when the same topics (or themes) keep emerging in your interviews, and conducting
more interviews results in no new themes.
Rule of thumb - 12 interviews for saturation
Need Finding
Refine Build Learn
Data Collection, Validation, Evaluation
A/B Testing Contextual Inquiry Diary Studies Ethnography Field Studies
Focus Groups Hallway Testing Heuristic Evaluation Interviews Lab Testing
Log Analysis Remote Testing Think-Aloud Surveys User Observation
Validation & Evaluation
Build Learn
A/B Testing Contextual Inquiry Diary Studies Ethnography Field Studies
Focus Groups Hallway Testing Heuristic Evaluation Interviews Lab Studies
Log Analysis Remote Testing Think-Aloud Surveys User Observation
Data Collection, Validation, Evaluation
Types of Usability TestingFormative Summative
A/B Testing ✔Field Studies ✔Hallway Testing ✔ ✔Heuristic Evaluation ✔ ✔Interviews ✔ ✔Log Analysis ✔Remote Testing ✔Surveys ✔ ✔Think-Aloud ✔ ✔Wizard of Oz ✔
Lab Usability StudiesFormative Summative
A/B Testing ✔Field Studies ✔Hallway Testing ✔ ✔Heuristic Evaluation ✔ ✔Hypothesis Testing ✔Interviews ✔ ✔Log Analysis ✔Remote Testing ✔Surveys ✔ ✔Think-Aloud ✔ ✔Wizard of Oz ✔
Lab Usability Testing
http://trydevkit.com/blog-post/a-beginner-s-guide-to-usability-testing/81da0af5-fb17-fd8e-016b-536948e32ced
http://trydevkit.com/blog-post/a-beginner-s-guide-to-usability-testing/81da0af5-fb17-fd8e-016b-536948e32ced
are good for….
learning how easy or difficult it is for users to learn and use your interface
if language and iconography are intuitive
how users encounter and recover from errors
Usability Studies
Lab Usability StudiesBenefits
Cheap (in $)
Observe user behavior as they encounter a design for the first time
See the consequences of design decisions first-hand
Limitations
Usually takes a moderate amount of time and set-up
Can sometimes feel staged, or unauthentic
Running a Usability StudyPlanning: create test plan, recruit participants
Pilot: practice with internal users, resolve any technical or logistical issues
Test session: run test plan, be present (formative) or simply observe (summative)
Debrief: short Q&A with participants, discuss observations with other study observers
Analysis
Contextual Usability StudiesFormative Summative
A/B Testing ✔Field Studies ✔Hallway Testing ✔ ✔Heuristic Evaluation ✔ ✔Hypothesis Testing ✔Interviews ✔ ✔Log Analysis ✔Remote Testing ✔Surveys ✔ ✔Think-Aloud ✔ ✔Wizard of Oz ✔
Field StudiesBenefits
Allows you to observe authentic, contextual, user behavior
Can observe the day-to-day experience users have with your product, across a longer period of time
Limitations
Significant cost ($)
Takes more time to run
Field Study Example[00:24:19.08] Boy says to the girl on his right: "you cheating”
[00:24:21.19] Girl to the left: "what? Its fun. ::mumble:: the simulation. Look.”
[00:24:25.21] The boy looks to the girl on his left, then back to the girl on the right, then down to his workbook in front of him. He puts his head on the table.
Contextual Usability StudiesFormative Summative
A/B Testing ✔Field Studies ✔Hallway Testing ✔ ✔Heuristic Evaluation ✔ ✔Hypothesis Testing ✔Interviews ✔ ✔Log Analysis ✔Remote Testing ✔Surveys ✔ ✔Think-Aloud ✔ ✔Wizard of Oz ✔
is good for….
asking “how much”, “how many”, “which one is better”
sampling from actual users
testing live apps/services
A/B Testing
A/B TestingBenefits
Low maintenance: release and wait for data
Can measure very specific questions
Live testing, measures actual vs. self-reported user behavior
Limitations
Missing context of why users take an action
May have no results, not guaranteed to be conclusive
Only measures certain user interactions
Running an A/B TestMake sure to:
test conditions simultaneously (fewer confounding factors)
be consistent, keep track of which users see which version
deploy tests cautiously; user research can help inform
Things to watch out for: don’t jump to conclusions, wait for statistical significance
waiting too long could cost you potential conversions
you might be interfering with user habit
Running an A/B TestMake sure to:
test conditions simultaneously (fewer confounding factors)
be consistent, keep track of which users see which version
deploy tests cautiously; user research can help inform
Things to watch out for: don’t jump to conclusions, wait for statistical significance
waiting too long could cost you potential conversions
you might be interfering with user habit
https://medium.com/@adlon/threats-of-a-b-tests-and-ux-research-adoption-time-and-incrementalism-991c0c3c61b6
is good for….
seeing page views, entry/exit, platforms, engagement
sampling from actual users
testing live apps/services
Log Analysis
Log AnalysisBenefits
Low maintenance: release and wait for data
Flexibility, can measure a wide range of data
Live testing, measures actual vs. self-reported user behavior
Limitations
Missing context of why users take an action
Often requires initial development overhead
Example: Google AnalyticsQuestion: how many mobile users does my app have?
http://www.smallbox.com/blog/quick-and-dirty-ux-3-things-google-analytics-can-tell-you-about-your-users
Example: Google AnalyticsQuestion: how many mobile users does my app have?
http://www.smallbox.com/blog/quick-and-dirty-ux-3-things-google-analytics-can-tell-you-about-your-users
Example: Google AnalyticsQuestion: what paths do users take on my site/app?
http://www.smallbox.com/blog/quick-and-dirty-ux-3-things-google-analytics-can-tell-you-about-your-users
Example: Google AnalyticsQuestion: how long are users spending on my site?
http://www.smallbox.com/blog/quick-and-dirty-ux-3-things-google-analytics-can-tell-you-about-your-users
Example: Google AnalyticsQuestion: how long are users spending on my site?
http://www.smallbox.com/blog/quick-and-dirty-ux-3-things-google-analytics-can-tell-you-about-your-users
Example: Google AnalyticsCustom logging: track any event you want (links, performance, etc.)
http://www.sitepoint.com/5-ways-use-google-analytics-ux-research/
Usability StudiesFormative Summative
A/B Testing ✔Field Studies ✔Hallway Testing ✔ ✔Heuristic Evaluation ✔ ✔Hypothesis Testing ✔Interviews ✔ ✔Log Analysis ✔Remote Testing ✔Surveys ✔ ✔Think-Aloud ✔ ✔Wizard of Oz ✔
How to Choose a Usability Study?
Triangulation (n.) - using two or more methods to discover and validate a finding.
Usability StudiesFormative Summative
A/B Testing ✔Field Studies ✔Hallway Testing ✔ ✔Heuristic Evaluation ✔ ✔Hypothesis Testing ✔Interviews ✔ ✔Log Analysis ✔Remote Testing ✔Surveys ✔ ✔Think-Aloud ✔ ✔Wizard of Oz ✔
Usability StudiesFormative Summative Quantitative Qualitative
A/B Testing ✔ ✔Field Studies ✔ ✔Hallway Testing ✔ ✔ ✔Heuristic Evaluation ✔ ✔ ✔Hypothesis Testing ✔ ✔Interviews ✔ ✔ ✔Log Analysis ✔ ✔Remote Testing ✔ ✔ ✔Surveys ✔ ✔ ✔Think-Aloud ✔ ✔ ✔Wizard of Oz ✔ ✔
what are you testing? what kind of results do you want?
Usability StudiesFormative Summative Quantitative Qualitative
A/B Testing ✔ ✔Field Studies ✔ ✔Hallway Testing ✔ ✔ ✔Heuristic Evaluation ✔ ✔ ✔Hypothesis Testing ✔ ✔Interviews ✔ ✔ ✔Log Analysis ✔ ✔Remote Testing ✔ ✔ ✔Surveys ✔ ✔ ✔Think-Aloud ✔ ✔ ✔Wizard of Oz ✔ ✔
what are you testing? what kind of results do you want?
Potential Pitfalls of Quantitative Research
Easy to make mistakes: phantom correlations
finding may not generalize (participant selection)
requires sound experimental design
But it’s a great supplement to qualitative research
Activity - Part 2Reflect on the one interview you conducted.
How has your knowledge grown? What do you still need to learn about?
What do you need to get there? (What do you need to do complete a full persona?)
How would you build a research plan for your topic?
Usability StudiesFormative Summative Quantitative Qualitative
A/B Testing ✔ ✔Field Studies ✔ ✔Hallway Testing ✔ ✔ ✔Heuristic Evaluation ✔ ✔ ✔Hypothesis Testing ✔ ✔Interviews ✔ ✔ ✔Log Analysis ✔ ✔Remote Testing ✔ ✔ ✔Surveys ✔ ✔ ✔Think-Aloud ✔ ✔ ✔Wizard of Oz ✔ ✔
what are you testing? what kind of results do you want?
Validation & EvaluationA/B Testing Contextual Inquiry Diary Studies Ethnography Field Studies
Focus Groups Hallway Testing Heuristic Evaluation Interviews Lab Studies
Log Analysis Remote Testing Think-Aloud Surveys User Observation
Data Collection, Validation, Evaluation
Learn
So Why Doesn’t My UI Work?
Seven Stages of Action
Mental Models
Psychological Biases
Dark UX Patterns
Seven Stages of Action
USER
SYSTEM
Establish goal
Form intention
Specify action sequence
Execute action
Perceive system
state
Interpret system
state
Evaluate system state
EXECUTION
EVALUATION
Gulf of Execution �How do I do it?�
Gulf of Evaluation �What does it mean?�
The GulfsGulf of Execution Does your app have good mappings? Can they easily figure out how to execute on their desired goal?
Gulf of Evaluation Does your app provide good feedback and visual cues? Can users easily interpret what the data the app is conveying to them?
confirmation bias (n) - the tendency to search for or interpret information in a way that confirms one’s preconceptions
possible pitfalls: you focus your questioning on behaviors that you expected to see, that confirm or validate your design
you discount negative comments about your design
diagnosis bias (n) - the tendency to label things based on initial impressions, and the difficulty or inability to change minds after the initial impression
possible pitfalls: discounting a participant’s responses based on their initial responses to selected questions
regression towards the mean (n) - when a non-random sample is selected, the average of that sample tends to regress towards the mean
possible pitfalls: thought your intervention was the reason for an improvement, but it was simply due to sampling
AnchoringFrame (n) - the way we present a decision may
highlight different attributes
A pound of meat that is 90% lean
or
A pound of meat that is 10% fat
FramingFrame (n) - the way we present a decision may
highlight different attributes
This treatment has a 90% chance of saving your life
or
This treatment has a 10% chance of failure, resulting in death
change blindness (n) - the tendency to overlook alterations, especially when they appear immediately
after a visual interruption
change blindness (n) - the tendency to overlook alterations, especially when they appear immediately
after a visual interruption
change blindness (n) - the tendency to overlook alterations, especially when they appear immediately
after a visual interruption
Case-Study: Anemia in Cambodia
Iron deficiency is a global problem
In the US: affects 3.5 million Americans each year
In Cambodia: affects 68% of children, 50% of adults
Image borrowed from: http://www.bustle.com/articles/84173-the-lucky-iron-fish-helps-fix-iron-deficiencies-just-by-boiling-it-with-food-and-it
UX Research & Design
Refine Build Learn
A/B Testing Contextual Inquiry Diary Studies Ethnography Field Studies
Focus Groups Hallway Testing Heuristic Evaluation Interviews Lab Studies
Log Analysis Remote Testing Think-Aloud Surveys User Observation
Data Collection, Validation, Evaluation
Feedback & QAQuestions? Comments?
Are there topics you wished we spent more time on?
How do you see some of these topics applying to your current work?
Thank you!
Meg Kurdziolek [email protected]
www.megkurdziolek.com
Karen Tang [email protected]
www.kptang.com
ResourcesSurveys:
http://uxmastery.com/better-user-research-through-surveys/
Interviews:
http://theuxreview.co.uk/user-interviews-the-beginners-guide/
http://www.nngroup.com/articles/interviewing-users/
https://whitneyhess.com/blog/2010/07/07/my-best-advice-for-conducting-user-interviews/
Usability Studies:
http://www.usability.gov/how-to-and-tools/methods/usability-testing.html