research operations at scale (christian rohrer at designops summit 2017)
TRANSCRIPT
R E S E A R C H O P E R AT I O N S AT S C A L E : E N A B L I N G O T H E R S W H I L E M A I N TA I N I N G E X C E L L E N C E
C H R I S T I A N P. R O H R E R , P H D
@christianrohrer and [email protected]
https://www.linkedin.com/in/crohrer/
S O …
W H AT C O M E S T O M I N D ?
K N O W N A S A C R E D I T C A R D
C O M PA N Y
B U T I T ’ S A C T U A L LY A B A N K …
G O A L : A T E C H C O M PA N Y
( T H AT J U S T H A P P E N S
T O D O B A N K I N G )
Desirability
ViabilityFeasibility
T O P S T R AT E G Y:
H U M A N - C E N T E R E D D E S I G N
Desirability
ViabilityFeasibility
(AND This means RESEARCH!)
Desirability
ViabilityFeasibility
B U T W H AT A B O U T… E F F I C I E N C Y R I S K R E G U L AT I O N S
D E S I G N R E S E A R C H AT A B A N K I S M I R E D B Y ” G O O ”
Research GooRisk: Strategy Leaks
Risk: Privacy Laws
Risk: Compliance
Reg: File 1099s
Reg: Consumer ProtectionReg: Complaints
Inefficiency: Recruiting & Scheduling
Inefficiency: Technical/Tool
Complexity
AT C A P I TA L O N E , T H E R E A R E D E D I C AT E D R O L E S F O R R E S E A R C H O P E R AT I O N S A N D C U R R I C U L A
Research Operations
• Research Operations Leads: Research request intake, coordinate studies, manage recruitment vendors, scheduling participants, communicating research schedules
• Technicians: manage lab and field equipment, videography
Research Curricula
• Instructors: Develop and teach courses on research methods & procedures to non-research employees
• Specialists: Provide remote research tool licenses, access and training (e.g., survey tools, online studies, diary apps, etc.)
Y O U C A N R U N A S T U D Y L I K E A N
E V E N T
T H I S R E S U LT S I N 3 M A J O R B E N E F I T S
R I S K & R E G U L AT I O N A R E H A N D L E D I N A D VA N C E O R B Y
T R A I N E D P E R S O N N E L
O T H E R “ G O O ” L I K E R E C R U I T I N G , S C H E D U L I N G A N D T E C H N I C A L S E T U P I S
L A R G E LY R E M O V E D
E X A M P L E : “ W I N E A N D D I N E ”
Should you do this?
CONTEXT
R E S E A R C H O P E R AT I O N S M U S T A D D R E S S C H A L L E N G E S F R O M T H R E E K E Y S O U R C E S
Recent Trends in Research
The Context of your business or industry
Universal Challenges inherent in research
What
about
you? IT/INTERNAL
B2BB2C
Major distinctions: • Customers’ needs • Users’ choices • Customer/User Access
Or maybe you’re a startup.
Who are your users today?
What about tomorrow’s users?
Are you getting direct access to your users?
Diffusion of innovation
© 2015 Christian Rohrer
WHAT PEOPLE SAY
WHAT PEOPLE DO
WHY & HOW TO FIX
HOW MANY & HOW MUCH
QUALITATIVE (DIRECT) QUANTITATIVE (INDIRECT)
ATTITUDINAL
BEHAVIORAL
You really should be doing both qualitative and quantitative research
R E S E A R C H O P E R AT I O N S M U S T A D D R E S S T H E K E Y S O U R C E S O F P O T E N T I A L C H A L L E N G E S
Recent Trends in Research
The Context of your business or industry
Universal Challenges inherent in research
Trends in Research
AGILE IS HERE (DEAL WITH IT)• Why? Agile is a must
to attract engineers
• So UCD must fit Agile
• By default, everything goes into a sprint
• Everyone’s on the same team (pod)
• Code early, no docsSource: http://scrumology.com/an-excerpt-from-improving-agility-agile-certifications/
Source:
T O P R E S E A R C H C H A L L E N G E S W I T H T R A D I T I O N A L A G I L E
• No UX roles defined
• Little up front research
• Hard to fit research into a sprint
• Need to feed development
• Dev centricity
R E C O N C I L I N G P R O C E S S E S
Source: Peter
Source: IDEO
Source: Michael James & Luke Walter
Developer & Product Owner Centric
Designer & User CentricNote: all of these are “human-centered” (just for very different humans)
W O R K I N G W I T H A G I L E – 2 M A I N A P P R O A C H E S
• Approach 1: Use agile methodology, but in a separate design phase
• Approach 2: Use an equally attractive, but more user-centered compatible processes (e.g., Lean)
• IMO: Doing strict agile, in which UX is simply added to Dev Pods does NOT work for Human-Centered Design processes. You cannot design or research anything significant in the same sprint as development.
24
E X A M P L E F R O M I N T E L S E C U R I T Y ’ S T R U E K E Y D E S I G N P R O C E S S ( A P P LY I N G A G I L E T O E A R L I E R D E S I G N P H A S E S )
2 weeks CXD sprint2-3 days before sprint starts
What is relevant here:
- Uses the same agile tools and nomenclature
- Design happens BEFORE Development
- Validate/Optimize designs/flows before code (on every sprint)
- Research is scheduled in the ”design sprints” before design executes
Dev
L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N
BUILD
MEASURELEARN
Products (MVP)
UsageUptake
L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N
BUILD
MEASURELEARN
Products (MVP)
UsageUptake
Improvements
Viable/Desirable
Usability
Interest
Prototypes
Ideas & Concepts
M O D I F I E D L E A N A P P R O A C H : B U I L D - S T U D Y- L E A R N
BUILD
MEASURELEARN
Products (MVP)
UsageUptake
Improvements
Viable/Desirable
Usability
Interest
Prototypes
Ideas & Concepts
STUDY
Source: It’s our Research by Tomer Sharon
Today, researchers can’t be the only ones driving research, or it would be a huge…
Another Trend: Research Participation
D I Y R E S E A R C H
• Over the past 5-6 years, everyone wants to do research:
• All design functions
• Product managers
• Developers
• Even executives!
• Removing barriers to research now essential.
• Just enough training and support are needed
D I Y R E S E A R C H T I P S
• Get the right vendors for you
• Reach the right user tyoe
• Remove goo
• Get tools that work
• Provide training and support
• Incorporate into YOUR process
R E S E A R C H O P E R AT I O N S M U S T A D D R E S S T H E K E Y S O U R C E S O F P O T E N T I A L C H A L L E N G E S
Recent Trends in Research
The Context of your business or industry
Universal Challenges inherent in research
UniversalChallenges
The 5 Purposes of Research1. Understand people and problems
2. Inspire creativity and innovation
3. Validate concepts and solutions
4. Improve and inform designs
5. Assess and measure experience
News flash!These require different skill
sets, not always found
in one person.
T H E TA L E N T P O O L : T H E R E A R E M A N Y S K I L L S A N D TA L E N T S
• It’s hard to find people that can
“do it all”
• Not everybody wants to
• You should allow for BOTH
specialists and hybrid players
on your team
• This enlarges the talent pool
T H E TA L E N T P O O L : S M A L L , G R O W S S L O W LY, V I C I O U S LY F O U G H T O V E R
“We will always be outnumbered.”
-Whitney Quissenberry, panel member at UXPA2016, “UX Past, Present & Future”
• Universities that produce Engineers: 1000s
• Universities that produce UX Professionals: about 75
Conclusion: Don’t waste your design and research talent. Make them as efficient and strategic as you can. Source: indeed.com
W H AT K I N D O F R E S E A R C H W O R K S B E S T F O R D E S I G N E R S ? W H Y ?
W H AT K I N D O F R E S E A R C H A N D I N S I G H T S W O R K B E S T F O R E X E C U T I V E S A N D P R O D U C T P R O F E S S I O N A L S ? W H Y
M A N Y T H I N K S I M P L I S T I C A L LY:
“ B I G N U M B E R S
G O O D . S M A L L N U M B E R S
B A D . ”
“Sorry, but I’ll never make a decision based on research
with just 5 people.” –Product Management
OK, LET’S JUST ACKNOWLEDGE THAT METRICS MATTER
39
• “You can’t manage what you can’t measure”
• “I need a dashboard to control my business”
• “How does our NPS compare?”
• “Invest in data scientists and big data”
• “I want to be more scientific”
• “Build, measure, learn”
• “The design needs to be validated”
Conversion rates, downloads, NPS, customer reviews, A/B testing performance, uptime, abandon rate, analytics data, path/click flows
NOT measuring the experience. They measure the product performance.
A User-Centered Model of User Experience
Look & FeelThe visual and industrial design is clear,
professional and appropriate SoundClear and relevant language, content and information architecture.
The price of entry. In today’s world of technology, data, and design, there is no excuse for
something to be hard.
Ease
Emotional Outcome
It doesn’t matter how easy it is if it doesn’t address the real human need in the right way.
Effectiveness
Sources:
• Rohrer’s Simple Model of UX (2006-2016)
• Forrester: Three Es of CX (2015)40
The PURE Method Pragmatic Usability Rating by Experts
Source: Link goes here
42
PURE: A way to measure “friction”
PURE provides a cognitive load “score” based on : 1. A well-defined user type2. A small set of “fundamental tasks”3. The “performance” of those tasks
(e.g., the Happy Path) for that user type, based on heuristics & design principles
Like in golf, smaller numbers are better, and green is good.
In PURE, we judge the experience based on a ”rubric,” as in skating & gymnastics
• A panel of raters each silently rates an experienceflow they are all witnessing at the same time
• A 1-3 rubric defines howmuch “friction” there is, basedon known UX heuristics
• PURE rates every step of the task (like a “move”)
• Rather thanaveraging scores,the panel decides on the score of each step
Photo credit: Ezra Shaw/Getty Images
A DIFFERENT KIND OF METRIC – ONE FROM A PANEL OF “EXPERTS”CONSIDER AN ANALOGY FROM THE REALM OF MOVIES… THE TOMATOMETER
PURE is kind of like this: An score based on expert analysis
ACTUAL PURE SCORECARD OVER TIME (VERSION OVER VERSION)
45
ACTUAL COMPETITIVE PURE SCORECARD
46
E X E C S W A N T N U M B E R S … S O G I V E T H E M N U M B E R S ! H E R E ’ S W H AT H A P P E N S W I T H P U R E
• People are intrigued. Then, either:
o Owners and Designers of the product want to fix ease of use problems
o Owners and Designers of the product push back, refuting PURE
• Either outcome is good! We are finally focusing on improving an important aspect of user experience: Ease of Use.
• It’s better to have numbers from actual user studies, but if you can’t do it easily, this is acceptable. It will cause positive change in your organization.
K N O W L E D G E M A N A G E M E N T
I S H A R D
R E Q U I R E M E N T S O F A G O O D K N O W L E D G E M A N A G E M E N T S Y S T E M
• Easy to find what we have and know
• Easy to contribute new info and docs
• Must handle more than text docs (shared cognitive artifacts like journey maps)
• Tags and metadata
• Include social and behavioral components (who contributed, who knows more, who used what, who likes what)
• A place where knowledge can grow and be used, not just be stored and accessed
A J O U R N E Y M A P O F K N O W L E D G E
Idea
Question
Research
Finding
Finding
Counter Finding
Counter Finding
Better Idea Another
TestConclusion
Fail
Hypothesis
Pivot
Finding
THEIdea
Winner!
Usability Study
Find & Fix
Find & Fix
Find & Fix
Find & Fix
Find & Fix
Find & Fix
Build &Release
Market data
Success!
T H E P R O B L E M S PA C E I S L A R G E
• Remove risk, deal with regulation, and address other industry-specific challenges
• Teach/empower non-researchers
• Find a way too fit into agile
• Support qualitative research
• Provide quantitative data
• Keep track of and better use your insights across your design process
T H E P U R P O S E O F C A P I TA L O N E R E S E A R C H
We believe opportunity lies in the understanding of the complexity of people.
DEEP EXPERTISE INTERPRETATON EMPOWERMENTPARTNERSHIP
Ultimately, we help bring insights into design.
@christianrohrer and [email protected]
https://www.linkedin.com/in/crohrer/
For more on PURE, contact me or use #puremethod
Thank you