Research Methods to Measure Design Impact:
Techniques for rapid decision- making
Steve Fadden, Ph.D.Director, Analytics UX Research,
SalesforceLecturer, School of Information, UC
Berkeley
Workshop delivered at UX India 2016 - Impact by Design, October 22, 2016, Hyderabad, India
Agenda
1. Introductions
2. Research process
3. Methods
4. Activity
5. Debrief & Discuss
Introduction
About me (@sfadden)
About you
Focus:
▸ Design▸ Research▸ Engineering▸ Marketing▸ Management▸ Other?
Experience:
▸ >5 years▸ 2-5 years▸ <2 years
Image: https://pixabay.com/en/people-group-crowd-team-isolated-309099/
Process
Generative Formative Summative
Research process
Images: https://commons.wikimedia.org/wiki/File:Light_bulb_(yellow)_icon.svg ; https://commons.wikimedia.org/wiki/File:Korean_Traffic_sign_(Left_Turn_and_Right_Turn).svg ;https://pixabay.com/en/chart-line-line-chart-diagram-trend-148256/
Formative methods inform progress
▸ Developing ideas, stories, flows, designs
▸ Any level of fidelity
▸ Focused on improving
▸ Informs decision-making
▸ “Lightweight” techniques → faster decisions
Image: https://commons.wikimedia.org/wiki/File:Korean_Traffic_sign_(Left_Turn_and_Right_Turn).svg
Designing involves lots of decisions
Images: http://www.ichanical.com/wp-content/uploads/2015/08/agile-dev.png ; https://crowdsourcedtesting.com/resources/wp-content/uploads/2016/07/agile-methodolody_695x260.jpg ; http://www.softwaretestingandistqb.com/wp-content/uploads/2015/06/Waterfall-Development-Model.jpg
“Fail faster to succeed sooner.”
David Kelley, IDEO
Quote / Image: https://ssir.org/articles/entry/fail_faster_succeed_sooner ; https://churchill.imgix.net/images/profiles/Kelley_David.jpg?auto=format&h=512&w=512&q=30
Methods
Understanding problems
Background
Image: https://en.wikipedia.org/wiki/Lockheed_L-1649_Starliner
Ask Critical Incident questions
Gain insights:
▸ Evidence of problems▸ Opportunities to solve
Based on:
▸ Recent events▸ Specific details▸ Feelings and perceptions▸ Future behaviors and responses
Critical Incident process
1. Identify time since last experience
2. Gather details:a. Descriptionb. Actions takenc. Feelingsd. Outcomee. Future actions/responses desired
Reference: http://www.usabilitynet.org/tools/criticalincidents.htm
Example prompt
“Consider the last time you had to share something online. How long ago did this happen? What did you share? Describe the steps you took to share, and highlight any surprises or problems (if any) that happened.”
Example result
▸ Validates problem
▸ Identifies opportunities
▸ Clarifies expectations
▸ Details scenarios
▸ Builds empathy
“I needed to share a PDF with a friend, and we use Dropbox, but I hadn’t used it in a while. I logged in through my browser, dragged the PDF to Files, and then saw the PDF open. I expected Dropbox to start uploading. I hit back, created a folder in Dropbox, uploaded the PDF to it, clicked share to add my friend, and sent her the link.”
Investigating concepts
Background
Image / Reference: Rosson, M.B., & Carroll, J.M. (2002). Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Academic Press; Carroll, J.M. (1999). Five Reasons for Scenario-Based Design. In: Proceedings of the 32nd Hawaii International Conference on System Sciences.
Concept evaluation with commenting
Gain insights:
▸ Initial confusion▸ Acceptability of process
Based on:
▸ Description of goal and flow▸ Illustration of process: storyboard, steps,
video
Commenting process
1. Present overall scenarioa. Ensure understandingb. Capture open concerns/questions
2. Show steps of flow (or interaction), capturing feedback about:a. Concerns, confusionsb. Benefits, positivesc. Open questions
3. Capture final comments at end
Example prompt
“Consider the data export concept presented on the next 4 slides. As you read through the concept, comment on anything you find to be confusing, problematic, useful, or appealing about the concept.”
Example flow (comments gathered after each slide presented)
1.
2.
3.
4.
100%
Example feedback, slide 1
1.
“Makes sense so far. No comments yet.”
2.
Example feedback, slide 2
“Doing this would require a lot of clicks, even for a small number of columns.”
3.
Example feedback, slide 3
“You should embed best practices for naming here. Otherwise, the result could be messy.”
4.
100%
Example feedback, slide 4
“Will we be able to save the mappings? That could save time in the future.”
Example: Final comments
1.
2.
3.
4.
100%
“It’s great that you don’t have to jump around different parts of the system to do this. Very valuable to be able to complete this from one place.”
“Hi, I wanted to follow up to reiterate that this is a REALLY COOL idea and it fills a much needed requirement for our use of the product. Please consider me for future studies like this, because we need this functionality!”
Prioritizing needs
Kano model
Absent Present
Dissatisfaction
Satisfaction
Reference: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results
Kano model
Absent Present
Dissatisfaction
Satisfaction
Performance
(more is better)Basic
(required)
Exciting(truly unexpected)
Kano model (automobile)
Absent Present
Dissatisfaction
Satisfaction
Seat-warmersBluetooth
Intuitive GPS
Horsepower
Fuel economyAirbags
Adjustable seats
Kano question process
1. Assemble list of features
2. Pose realistic scenarios for each
3. For each question, assess:a. Satisfaction if availableb. Satisfaction if not availablec. Importance of feature
Scenario: “Imagine you are using a mobile app to find a restaurant.”
Question 1: “How would you feel if the app allowed you to filter results by distance?”
Question 2: “How would you feel if the app did not allow you to filter results by distance?”
Question 3: “How important is this function to you?”
Kano results
:( if present :) if present
:) if absent
:( if absent
Map answers to “product allowed you to...” questions
Map answers to “product did not allow you to...” questions
Kano results
:( if present :) if present
:) if absent
:( if absent
Kano results
:( if present :) if present
:) if absent
:( if absent
“We should explore and reconsider these...”
Kano results
:( if present :) if present
:) if absent
:( if absent
“Do we emphasize what people want (X-axis), or what they expect (Y-axis)?”
Activity (part 1)
“A leading transportation provider has announced a competition to design a product to improve passenger engagement with the community.
Work with your team to consider a concept.
Activity: Improving passenger engagement with community
Process
1. Discuss possible concepts
2. Plan questions to understand needs/problems
3. Conduct interview on non-team member
Scenario
A leading transportation provider has announced a competition to design a product to improve passenger engagement with the community. Work with your team to consider a concept and make decisions to improve it.
Identifying Expectations
Expectation testing
Gain insights:
▸ Mental model▸ Categories and labels▸ Areas of confusion
Based on:
▸ Nonworking interface▸ No text/labels▸ Basic task information
Reference: Thomas S. Tullis, 1998. A method for evaluating web page design concepts. CHI 98 conference summary on human factors in computing systems
“Greeking” technique
1. Identify important task
2. Write scenario
3. Write “first step” question
4. Gather feedback
5. Ask for group/category names
Example
Scenario: “You are exploring colleges, and need to make sure the courses look interesting and financial support is available.”
Instruction: “Indicate where you would click to access a list of courses?”
Try it!
Where would you click to
1. ...access a list of courses?
2. ...determine if financial support is available?
3. ...contact the department in charge of Athletics?
What would you call each group?
Sample results
Where would you click to
1. ...access a list of courses?
2. ...determine if financial support is available?
3. ...contact the department in charge of Athletics?
What would you call each group?
12
3
“Academics” “Administrative”
“Connections”
Uncovering impressions
First impressions matter
Impressions are formed by 50ms
▸ Primarily based on visual appeal▸ Do not change with additional viewing time▸ Are not influenced by actual usability issues
5-second test reveals usability impression
1. Show interface for 5 seconds2. Gather feedback on appeal and usability
Reference: Phillips, C. & Chaparro, B.S. (2009). Visual Appeal vs. Usability: Which One Influences User Perceptions of a Website More? http://usabilitynews.org/visual-appeal-vs-usability-which-one-influences-user-perceptions-of-a-website-more/
Initial prompt
“You will be shown an interface for 5 seconds. After viewing the interface, indicate your response to the following questions.”
Prompt after viewing
Mark how you feel about the interface you just saw. The interface is:
Very VeryAttractive - - - - - - - - - Unattractive
Very VeryEasy - - - - - - - - - Hard
Very VeryEfficient - - - - - - - - - Inefficient
Responses found to correlate with System Usability Scale
Strongly Strongly Agree Disagree
1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.3. I thought the system was easy to use.4. I think that I would need the support of a
technical person to be able to use this system.5. I found the various functions in this system were
well integrated.6. I thought there was too much inconsistency in this
system.7. I would imagine that most people would learn to use
this system very quickly.8. I found the system very cumbersome to use.9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.
Assessing sentiment
Use semantic differential scales to quickly assess sentiment
Gain insights:
▸ How people feel about your concept, interface, experience
▸ Alignment with intended goals
Based on:
▸ Exposure to concept, interface, or task▸ Marking experience on 5-point scale
Reference: Source: Johnson, F. 2012. Using semantic differentials for an evaluative view of the search engine as an interactive system, EuroHCIR2012.
Semantic differential process
1. Determine goal: a. Granularity: Overall concept, interface,
task performance
b. Criteria: utility, efficiency, satisfaction
2. Administer as survey or interview prompt
3. Analyze based on participant characteristics, interface differences, tasks
Sample prompt and responses
“Mark how you feel about this experience.”
Powerful - - - - - SimplisticAttractive - - - - - UnattractiveValuable - - - - - Not valuableRelevant - - - - - IrrelevantSatisfying - - - - - FrustratingFast - - - - - SlowPredictable - - - - - UnpredictableIntuitive - - - - - RigidEasy - - - - - Difficult
Example result from interface A
“I selected these values because I really like how the interface looks, but I’m not sure what to do with it. What would I ever use this for?”
Utility
Efficiency
Control
Affect
Appearance
Example result from interface B
“Even though the interface looks really plain and boring, I like that I have complete control when doing this task, and it’s a lot faster than how I need to accomplish this today.”
Utility
Efficiency
Control
Affect
Appearance
Combined results (A & B)
Utility
Efficiency
Control
Affect
Appearance
“Utility and Appearance are both critical for this product -- we need to design for both!”
Evaluating usability
Usability evaluation
Goals
▸ Explore concepts▸ Assess effectiveness▸ Compare competitors
Measures
▸ Learnability▸ Efficiency▸ Effectiveness (Errors)▸ Satisfaction
Process
1. Plan (goals, participant criteria)
2. Prepare (recruit, test materials)
3. Conduct (active observers)
4. Analyze (issues, insights)
5. Report (fixes, future opportunities)
Usability evaluation
Rapid Iterative Testing & Evaluation (RITE) - identify and fix problems
Discount usability - identify and prioritize problems
Traditional testing - benchmark, summarize performance
Aim to:
● Test representative users● Follow script with actual tasks● Use high-fidelity interface
Key elements:
● Observation by key decision makers (PM, design, development)
● Agree on critical problems and make changes to interface after each session
Reference: Medlock, M.C., Wixon, D., Terrano, M., Romero, R., & Fulton, B. (2002). Using the RITE method to improve products; a definition and a case study, Presented at the Usability Professionals Association.
Example result
● RITE with 4 participants over 2 days
● Key changes: Terminology, embedded instructions, icons
● Stakeholder gratitude and desire for more
“I haven’t selected the columns yet! Why can’t I go back?”
“It says ‘save’ but I’m just modifying the settings. Unless I can re-use this later?”
Activity (part 2)
“A leading transportation provider has announced a competition to design a product to improve passenger engagement with the community.
Work with your team to refine concept.
Activity: Improving passenger engagement with community
Process
1. Sketch a flow or interface
2. Plan methods to:a. Refine conceptb. Refine interface
3. Try one!
▸ Problem: Critical incident questions
▸ Concept: Commenting▸ Expectation: Greeking▸ Impression: 5s test▸ Sentiment: Semantic
differential▸ Prioritization: Kano
questions▸ Usability: RITE
Debrief
Report-out
1. Your concept2. Research plan3. Refinements you made4. Lesson(s) learned
Tips & Tricks
Recruit participants for the research purpose
Image: https://pixabay.com/en/think-about-salaried-worker-1184858/
Order research activities intentionally and don’t overdo it
Image: https://pixabay.com/en/photos/aim/
Pilot with each other
Image: https://pixabay.com/en/men-team-women-blue-craftsmen-158690/
Triangulate, triangulate, triangulate
Image: https://commons.wikimedia.org/wiki/File:3_triangles.svg
Others?
Questions?
Image: https://pixabay.com/en/hand-fingers-silhouette-human-294018/
Thank you!
@sfadden on Twitter
slideshare.net/stevefadden1