c-rite: how to run impactful iterative studies in a fast paced environment v2
DESCRIPTION
In this talk, I will describe C-RITE - a user research technique I developed as the head of UX research for Android. My research team at Android not only works on the Android operating system but also Google services (apps, the Play store, Play vertical), Google Now/Voice, Hardware (phones, tablets), Android Wear, Automotive, TV and many other cool projects. Android is a fast-paced environment that ships products on cycles ranging from few weeks to a few months. This method worked well for us because it helped the team to iterate quickly and make big qualitative user experience jumps with confidence.TRANSCRIPT
![Page 1: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/1.jpg)
C(ollab) RITE How to run impactful iterative studies in a fast paced environment
Jhilmil JainAndroid
#crite
@jhilmiljain
#androidux
Wednesday, July 30, 14
![Page 2: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/2.jpg)
C-RITE• Iterative UX research method suitable for agile
environments
• From research to new design solution in 1 day
• 7 research sessions in 2 days
• Collaborative design workshop each day
• Prototype changes between day 1 and 2
• Can be repeated weekly
• Stakeholders collaborate in all UX phases
#crite #androiduxWednesday, July 30, 14
![Page 3: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/3.jpg)
ThePlayers
The Players and Their Roles
Wednesday, July 30, 14
![Page 4: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/4.jpg)
Research AnalysisReporting
&Prioritization
Design PrototypingPlanning
& Recruiting
Traditional Roles
Reporting &
Prioritization
Planning & Recruiting Research Analysis Design Prototyping
Reporting &
Prioritization
Planning & Recruiting
Designer Designer
Researcher
PM PM
EngineerWednesday, July 30, 14
![Page 5: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/5.jpg)
Collaboration in C-RITE
Research Analysis Design PrototypingReporting
&Prioritization
Planning & Recruiting
Designer
PM
Engineer
Researcher
Wednesday, July 30, 14
![Page 6: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/6.jpg)
Time Line - Traditional Usability
Research AnalysisReporting
&Prioritization
Design PrototypingPlanning
& Recruiting
1- 2 weeks 1- 2 weeks 1/2 - 1week 1/2 -1week
4 - 8 weeks
#crite #androidux
1/2 - 1 week1/2 -1 week
Wednesday, July 30, 14
![Page 7: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/7.jpg)
Time Line - RITE
Research AnalysisReporting
&Prioritization
Design PrototypingPlanning
& Recruiting
1 - 2 weeks 1 - 2 weeks
2 - 4 weeks
#crite #androiduxWednesday, July 30, 14
![Page 8: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/8.jpg)
Time Line - C-RITE
Research AnalysisReporting
&Prioritization
Design PrototypingPlanning
& Recruiting
1-2 weeks 2.5 days
1.5 - 2.5 weeks
#crite #androiduxWednesday, July 30, 14
![Page 9: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/9.jpg)
Time Line - Cyclic C-RITE
Research AnalysisReporting
&Prioritization
Design PrototypingPlanning
& Recruiting
1 week 2.5 days
1.5 weeks
#crite #androiduxWednesday, July 30, 14
![Page 10: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/10.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
Example: TV Voice Search
Wednesday, July 30, 14
![Page 11: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/11.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
1st TV Voice Search Design
Wednesday, July 30, 14
![Page 12: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/12.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Research Timing
10:00 am 11:00 am 1:00 pm 2:00 pm10:00 am 11:00 am 1:00 pm
Day 1 Day 2
#crite #androidux
Wednesday, July 30, 14
![Page 13: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/13.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Collaborative Observation
10:00 am 11:00 am 1:00 pm 2:00 pm10:00 am 11:00 am 1:00 pm
Day 1 Day 2
#crite #androidux
Wednesday, July 30, 14
![Page 14: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/14.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
Observation Board Example
Wednesday, July 30, 14
![Page 15: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/15.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
User Goals Example
Observation ExampleGoal example
P1P1 P2P2Bob Lin Ali Nur Bob Lin Ali Nur
User knows what to say 2 1 2 2
The system understood user’s intent 5 3 1 1
The system doesn’t slow down user w unnecessary confirmations 4 4 2 2
User can easily tell where speech will work 2 2 2 2
User comfortable using their voice 4 4 4 4
Rating on a scale from 1- 5 1 = poor 5 = good
Wednesday, July 30, 14
![Page 16: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/16.jpg)
Mocks Example
Wednesday, July 30, 14
![Page 17: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/17.jpg)
Mocks Example
Wednesday, July 30, 14
![Page 18: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/18.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
Collaborative Observation
#crite #androidux
• Re-call to count observations, analysis step reduced
• Disagreements about observations are resolved immediately
• Research protocol can be adjusted based on stakeholder feedback
• Researcher can explain what observations mean
• Small sample size discussion is obsolete
Wednesday, July 30, 14
![Page 19: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/19.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Analysis
10:00 am 11:00 am 1:00 pm 2:00 pm10:00 am 11:00 am 1:00 pm
Day 1 Day 2
2:00 pm 3:00 pm
#crite #androidux
Wednesday, July 30, 14
![Page 20: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/20.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
From Observation to Insight
Observation ExampleGoal example
Wednesday, July 30, 14
![Page 21: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/21.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
Prioritizing Insights
Goal example
Insights
Wednesday, July 30, 14
![Page 22: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/22.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Design Workshop
3:00 pm
Day 1
10:00 am 11:00 am 1:00 pm 2:00 pm 10:00 am 11:00 am 1:00 pm 2:00 pm
Day 2
6:30 pm5:30 pm
#crite #androidux
Wednesday, July 30, 14
![Page 23: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/23.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
Design Workshop
Wednesday, July 30, 14
![Page 24: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/24.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
Design Workshop Actions
Observation ExampleGoal example
Wednesday, July 30, 14
![Page 25: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/25.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
Action Item Owners & Due Dates
Wednesday, July 30, 14
![Page 26: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/26.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
Before & After
Wednesday, July 30, 14
![Page 27: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/27.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence
Wednesday, July 30, 14
![Page 28: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/28.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence• Designers demonstrate problem solving skills
Wednesday, July 30, 14
![Page 29: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/29.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence• Designers demonstrate problem solving skills• Engineers solve technical constraints
Wednesday, July 30, 14
![Page 30: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/30.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence• Designers demonstrate problem solving skills• Engineers solve technical constraints• Researchers guide insight interpretation
Wednesday, July 30, 14
![Page 31: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/31.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence• Designers demonstrate problem solving skills• Engineers solve technical constraints• Researchers guide insight interpretation• PM gives business and scheduling input
Wednesday, July 30, 14
![Page 32: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/32.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
C-RITE Design Workshop
#crite #androidux
• Turn panic into confidence• Designers demonstrate problem solving skills• Engineers solve technical constraints• Researchers guide insight interpretation• PM gives business and scheduling input• Everyone participates in creative process
Wednesday, July 30, 14
![Page 33: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/33.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Reporting
Day 1
Send action items to team
Day 2
Send action items to team Short report
Day 3
#crite #androidux
Wednesday, July 30, 14
![Page 34: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/34.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Prototyping
3:00 pm10:00 11:00 1:00 2:00
Day 2
6:30 pm
Day 1
10:00 11:00 1:00 2:00 5:30 pm
#crite #androidux
Wednesday, July 30, 14
![Page 35: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/35.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Cycles
Week1 Week 2 Week 3
#crite #androidux
Wednesday, July 30, 14
![Page 36: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/36.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Recruiting
• Create a panel of users that meet general recruitment criteria
• Panel members commit to availability at short notice on research days
• Each week, schedule panel members• Recruit for specific criteria when needed
#crite #androidux
Wednesday, July 30, 14
![Page 37: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/37.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Preparations
• Set expectations with all stakeholders beforehand about time commitment, collaboration and workshop format
• Decision makers need to participate in C-RITE studies and workshops
#crite #androidux
Wednesday, July 30, 14
![Page 38: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/38.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
C-RITE Summary
#crite #androidux
• Iterative UX research method suitable for agile environments
• From research to new design solution in 1 day
• 7 research sessions in 2 days
• Collaborative design workshop each day
• Prototype changes between day 1 and 2
• Can be applied weekly
• Stakeholders collaborate in all UX phases
Wednesday, July 30, 14
![Page 39: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/39.jpg)
Thank you
#crite #androiduxWednesday, July 30, 14
![Page 40: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/40.jpg)
Appendix
Wednesday, July 30, 14
![Page 41: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/41.jpg)
Agile Methods: RITE
• Changes are made as soon as a problem is identified, sometimes after 1 session
• 1-2 sessions a day. Breaks between sessions long enough to make changes
• 1-2 weeks of sessions
• Requires stakeholder attendance and participation
Medlock, Wixon, Fulton, Terrano and Romero, UPA, 2002
#crite #androiduxWednesday, July 30, 14
![Page 42: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/42.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
Agile Methods: KRUG + RITE
• 4 participants over 2 days, at least 1 hour between sessions to make changes
• Researcher sends out observations and recommendations 30 min after last session
• Team debrief of observations and action items on day 2, 1 hour after last session
• No collaborative design workshop
McGinn, Chen, Journal of Usability Studies, 2013
Wednesday, July 30, 14
![Page 43: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/43.jpg)
Research Analysis Reporting &Prioritization Design Prototyping
Planning & Recruiting
Agile Methods: PULSE
• 3 participants, 1 day every week
• Analysis and write-up of results by researcher within 2 days
• No collaborative analysis and design workshop
Konno, Agile 2012
Wednesday, July 30, 14
![Page 44: C-RITE: How to run impactful iterative studies in a fast paced environment V2](https://reader034.vdocuments.us/reader034/viewer/2022042814/555daaa5d8b42a63328b4589/html5/thumbnails/44.jpg)
Research Analysis Reporting &Prioritization Design Implementation
Planning & Recruiting
Variations We’ve Tried
• C-RITE Remote• If time zones allow, stakeholders participate through ‘Hangouts’
• Sessions are recorded and watched with time delay
• Next day debriefs and design workshops for incompatible time zones
• C-RITE Diary Studies• In-lab diary kick-offs and debriefs are run in C-RITE format
• Researcher analyzes and summarizes diary data for debrief and design workshop
Wednesday, July 30, 14