ux india 2016 workshop: research methods to measure design impact - techniques for rapid decision...

Research Methods to Measure Design Impact:

Techniques for rapid decision- making

Steve Fadden, Ph.D.Director, Analytics UX Research,

SalesforceLecturer, School of Information, UC

Berkeley

Workshop delivered at UX India 2016 - Impact by Design, October 22, 2016, Hyderabad, India

Agenda

1. Introductions

2. Research process

3. Methods

4. Activity

5. Debrief & Discuss

Introduction

About me (@sfadden)

About you

Focus:

▸ Design▸ Research▸ Engineering▸ Marketing▸ Management▸ Other?

Experience:

▸ >5 years▸ 2-5 years▸ <2 years

Image: https://pixabay.com/en/people-group-crowd-team-isolated-309099/

Process

Generative Formative Summative

Research process

Images: https://commons.wikimedia.org/wiki/File:Light_bulb_(yellow)_icon.svg ; https://commons.wikimedia.org/wiki/File:Korean_Traffic_sign_(Left_Turn_and_Right_Turn).svg ;https://pixabay.com/en/chart-line-line-chart-diagram-trend-148256/

Formative methods inform progress

▸ Developing ideas, stories, flows, designs

▸ Any level of fidelity

▸ Focused on improving

▸ Informs decision-making

▸ “Lightweight” techniques → faster decisions

Image: https://commons.wikimedia.org/wiki/File:Korean_Traffic_sign_(Left_Turn_and_Right_Turn).svg

Designing involves lots of decisions

Images: http://www.ichanical.com/wp-content/uploads/2015/08/agile-dev.png ; https://crowdsourcedtesting.com/resources/wp-content/uploads/2016/07/agile-methodolody_695x260.jpg ; http://www.softwaretestingandistqb.com/wp-content/uploads/2015/06/Waterfall-Development-Model.jpg

“Fail faster to succeed sooner.”

David Kelley, IDEO

Quote / Image: https://ssir.org/articles/entry/fail_faster_succeed_sooner ; https://churchill.imgix.net/images/profiles/Kelley_David.jpg?auto=format&h=512&w=512&q=30

Methods

Understanding problems

Background

Image: https://en.wikipedia.org/wiki/Lockheed_L-1649_Starliner

Ask Critical Incident questions

Gain insights:

▸ Evidence of problems▸ Opportunities to solve

Based on:

▸ Recent events▸ Specific details▸ Feelings and perceptions▸ Future behaviors and responses

Critical Incident process

1. Identify time since last experience

2. Gather details:a. Descriptionb. Actions takenc. Feelingsd. Outcomee. Future actions/responses desired

Reference: http://www.usabilitynet.org/tools/criticalincidents.htm

Example prompt

“Consider the last time you had to share something online. How long ago did this happen? What did you share? Describe the steps you took to share, and highlight any surprises or problems (if any) that happened.”

Example result

▸ Validates problem

▸ Identifies opportunities

▸ Clarifies expectations

▸ Details scenarios

▸ Builds empathy

“I needed to share a PDF with a friend, and we use Dropbox, but I hadn’t used it in a while. I logged in through my browser, dragged the PDF to Files, and then saw the PDF open. I expected Dropbox to start uploading. I hit back, created a folder in Dropbox, uploaded the PDF to it, clicked share to add my friend, and sent her the link.”

Investigating concepts

Background

Image / Reference: Rosson, M.B., & Carroll, J.M. (2002). Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Academic Press; Carroll, J.M. (1999). Five Reasons for Scenario-Based Design. In: Proceedings of the 32nd Hawaii International Conference on System Sciences.

Concept evaluation with commenting

Gain insights:

▸ Initial confusion▸ Acceptability of process

Based on:

▸ Description of goal and flow▸ Illustration of process: storyboard, steps,

video

Commenting process

1. Present overall scenarioa. Ensure understandingb. Capture open concerns/questions

2. Show steps of flow (or interaction), capturing feedback about:a. Concerns, confusionsb. Benefits, positivesc. Open questions

3. Capture final comments at end

Example prompt

“Consider the data export concept presented on the next 4 slides. As you read through the concept, comment on anything you find to be confusing, problematic, useful, or appealing about the concept.”

Example flow (comments gathered after each slide presented)

1.

2.

3.

4.

100%

Example feedback, slide 1

1.

“Makes sense so far. No comments yet.”

2.

Example feedback, slide 2

“Doing this would require a lot of clicks, even for a small number of columns.”

3.

Example feedback, slide 3

“You should embed best practices for naming here. Otherwise, the result could be messy.”

4.

100%

Example feedback, slide 4

“Will we be able to save the mappings? That could save time in the future.”

Example: Final comments

1.

2.

3.

4.

100%

“It’s great that you don’t have to jump around different parts of the system to do this. Very valuable to be able to complete this from one place.”

“Hi, I wanted to follow up to reiterate that this is a REALLY COOL idea and it fills a much needed requirement for our use of the product. Please consider me for future studies like this, because we need this functionality!”

Prioritizing needs

Kano model

Absent Present

Dissatisfaction

Satisfaction

Reference: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results

Kano model

Absent Present

Dissatisfaction

Satisfaction

Performance

(more is better)Basic

(required)

Exciting(truly unexpected)

Kano model (automobile)

Absent Present

Dissatisfaction

Satisfaction

Seat-warmersBluetooth

Intuitive GPS

Horsepower

Fuel economyAirbags

Adjustable seats

Kano question process

1. Assemble list of features

2. Pose realistic scenarios for each

3. For each question, assess:a. Satisfaction if availableb. Satisfaction if not availablec. Importance of feature

Scenario: “Imagine you are using a mobile app to find a restaurant.”

Question 1: “How would you feel if the app allowed you to filter results by distance?”

Question 2: “How would you feel if the app did not allow you to filter results by distance?”

Question 3: “How important is this function to you?”

Kano results

:( if present :) if present

:) if absent

:( if absent

Map answers to “product allowed you to...” questions

Map answers to “product did not allow you to...” questions

Kano results

:( if present :) if present

:) if absent

:( if absent

Kano results

:( if present :) if present

:) if absent

:( if absent

“We should explore and reconsider these...”

Kano results

:( if present :) if present

:) if absent

:( if absent

“Do we emphasize what people want (X-axis), or what they expect (Y-axis)?”

Activity (part 1)

“A leading transportation provider has announced a competition to design a product to improve passenger engagement with the community.

Work with your team to consider a concept.

Activity: Improving passenger engagement with community

Process

1. Discuss possible concepts

2. Plan questions to understand needs/problems

3. Conduct interview on non-team member

Scenario

A leading transportation provider has announced a competition to design a product to improve passenger engagement with the community. Work with your team to consider a concept and make decisions to improve it.

Identifying Expectations

Expectation testing

Gain insights:

▸ Mental model▸ Categories and labels▸ Areas of confusion

Based on:

▸ Nonworking interface▸ No text/labels▸ Basic task information

Reference: Thomas S. Tullis, 1998. A method for evaluating web page design concepts. CHI 98 conference summary on human factors in computing systems

“Greeking” technique

1. Identify important task

2. Write scenario

3. Write “first step” question

4. Gather feedback

5. Ask for group/category names

Example

Scenario: “You are exploring colleges, and need to make sure the courses look interesting and financial support is available.”

Instruction: “Indicate where you would click to access a list of courses?”

Try it!

Where would you click to

1. ...access a list of courses?

2. ...determine if financial support is available?

3. ...contact the department in charge of Athletics?

What would you call each group?

Sample results

Where would you click to

1. ...access a list of courses?

2. ...determine if financial support is available?

3. ...contact the department in charge of Athletics?

What would you call each group?

12

3

“Academics” “Administrative”

“Connections”

Uncovering impressions

First impressions matter

Impressions are formed by 50ms

▸ Primarily based on visual appeal▸ Do not change with additional viewing time▸ Are not influenced by actual usability issues

5-second test reveals usability impression

1. Show interface for 5 seconds2. Gather feedback on appeal and usability

Reference: Phillips, C. & Chaparro, B.S. (2009). Visual Appeal vs. Usability: Which One Influences User Perceptions of a Website More? http://usabilitynews.org/visual-appeal-vs-usability-which-one-influences-user-perceptions-of-a-website-more/

Initial prompt

“You will be shown an interface for 5 seconds. After viewing the interface, indicate your response to the following questions.”

Prompt after viewing

Mark how you feel about the interface you just saw. The interface is:

Very VeryAttractive - - - - - - - - - Unattractive

Very VeryEasy - - - - - - - - - Hard

Very VeryEfficient - - - - - - - - - Inefficient

Responses found to correlate with System Usability Scale

Strongly Strongly Agree Disagree

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.3. I thought the system was easy to use.4. I think that I would need the support of a

technical person to be able to use this system.5. I found the various functions in this system were

well integrated.6. I thought there was too much inconsistency in this

system.7. I would imagine that most people would learn to use

this system very quickly.8. I found the system very cumbersome to use.9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

Assessing sentiment

Use semantic differential scales to quickly assess sentiment

Gain insights:

▸ How people feel about your concept, interface, experience

▸ Alignment with intended goals

Based on:

▸ Exposure to concept, interface, or task▸ Marking experience on 5-point scale

Reference: Source: Johnson, F. 2012. Using semantic differentials for an evaluative view of the search engine as an interactive system, EuroHCIR2012.

Semantic differential process

1. Determine goal: a. Granularity: Overall concept, interface,

task performance

b. Criteria: utility, efficiency, satisfaction

2. Administer as survey or interview prompt

3. Analyze based on participant characteristics, interface differences, tasks

Sample prompt and responses

“Mark how you feel about this experience.”

Powerful - - - - - SimplisticAttractive - - - - - UnattractiveValuable - - - - - Not valuableRelevant - - - - - IrrelevantSatisfying - - - - - FrustratingFast - - - - - SlowPredictable - - - - - UnpredictableIntuitive - - - - - RigidEasy - - - - - Difficult

Example result from interface A

“I selected these values because I really like how the interface looks, but I’m not sure what to do with it. What would I ever use this for?”

Utility

Efficiency

Control

Affect

Appearance

Example result from interface B

“Even though the interface looks really plain and boring, I like that I have complete control when doing this task, and it’s a lot faster than how I need to accomplish this today.”

Utility

Efficiency

Control

Affect

Appearance

Combined results (A & B)

Utility

Efficiency

Control

Affect

Appearance

“Utility and Appearance are both critical for this product -- we need to design for both!”

Evaluating usability

Usability evaluation

Goals

▸ Explore concepts▸ Assess effectiveness▸ Compare competitors

Measures

▸ Learnability▸ Efficiency▸ Effectiveness (Errors)▸ Satisfaction

Process

1. Plan (goals, participant criteria)

2. Prepare (recruit, test materials)

3. Conduct (active observers)

4. Analyze (issues, insights)

5. Report (fixes, future opportunities)

Usability evaluation

Rapid Iterative Testing & Evaluation (RITE) - identify and fix problems

Discount usability - identify and prioritize problems

Traditional testing - benchmark, summarize performance

Aim to:

● Test representative users● Follow script with actual tasks● Use high-fidelity interface

Key elements:

● Observation by key decision makers (PM, design, development)

● Agree on critical problems and make changes to interface after each session

Reference: Medlock, M.C., Wixon, D., Terrano, M., Romero, R., & Fulton, B. (2002). Using the RITE method to improve products; a definition and a case study, Presented at the Usability Professionals Association.

Example result

● RITE with 4 participants over 2 days

● Key changes: Terminology, embedded instructions, icons

● Stakeholder gratitude and desire for more

“I haven’t selected the columns yet! Why can’t I go back?”

“It says ‘save’ but I’m just modifying the settings. Unless I can re-use this later?”

Activity (part 2)

“A leading transportation provider has announced a competition to design a product to improve passenger engagement with the community.

Work with your team to refine concept.

Activity: Improving passenger engagement with community

Process

1. Sketch a flow or interface

2. Plan methods to:a. Refine conceptb. Refine interface

3. Try one!

▸ Problem: Critical incident questions

▸ Concept: Commenting▸ Expectation: Greeking▸ Impression: 5s test▸ Sentiment: Semantic

differential▸ Prioritization: Kano

questions▸ Usability: RITE

Debrief

Report-out

1. Your concept2. Research plan3. Refinements you made4. Lesson(s) learned

Tips & Tricks

Recruit participants for the research purpose

Image: https://pixabay.com/en/think-about-salaried-worker-1184858/

Order research activities intentionally and don’t overdo it

Image: https://pixabay.com/en/photos/aim/

Pilot with each other

Image: https://pixabay.com/en/men-team-women-blue-craftsmen-158690/

Triangulate, triangulate, triangulate

Image: https://commons.wikimedia.org/wiki/File:3_triangles.svg

Others?

Questions?

Image: https://pixabay.com/en/hand-fingers-silhouette-human-294018/

Thank you!

@sfadden on Twitter

slideshare.net/stevefadden1