how to build a winning experimentation program & team | optimizely anz webinar 8
TRANSCRIPT
![Page 1: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/1.jpg)
Building a Winning Experimentation
Team December 2017
Managing Director, Australia & New Zealand
/danross9
Dan Ross
![Page 2: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/2.jpg)
By the end of today’s session, you will have learned:
• Why experimentation is central to competing and innovating
• Areas to assess when building your experimentation capability
• How organisational culture helps scale an experimentation program
![Page 3: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/3.jpg)
The world’s leading companies utiliseexperimentation to build a culture that fosters
innovation and agility
![Page 4: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/4.jpg)
![Page 5: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/5.jpg)
Optimization Is A Journey
![Page 6: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/6.jpg)
2015 2017
![Page 7: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/7.jpg)
![Page 8: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/8.jpg)
“If you want to gain a competitive advantage, master the science of
conducting online tests. The returns you reap – in cost savings, new
revenue, and improved user experience – can be huge.”
“…experiment with everything” approach has
surprisingly large payoffs.”
“…rigorous online experiments should be standard
operating procedure.”
![Page 9: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/9.jpg)
![Page 10: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/10.jpg)
“Our success
at Amazon is a function of
how many experiments we do
per year, per month, per week, per day…”
J E F F B E Z O S
![Page 11: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/11.jpg)
“Every decision is an experiment.”
M E . R I G H T N O W
![Page 12: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/12.jpg)
Soon after Ron Johnson left Apple to become the CEO of J.C. Penney, in
2011, his team implemented a bold plan that eliminated coupons and
clearance racks, filled stores with branded boutiques, and used technology to
eliminate cashiers, cash registers, and checkout counters. Yet just 17
months after Johnson joined Penney, sales had plunged, losses had
soared, and Johnson had lost his job. The retailer then did an about-face.
Had J.C. Penney done thorough experiments on its CEO’s
proposed changes, the company might have discovered
that customers would probably reject them.Source: Harvard Business Review,
“The Discipline of Business Experimentation” December 2014
EVERY CHANGE IS AN EXPERIMENT...CONTROLLED OR NOT
Stefan Thomke ,
Harvard Business School
![Page 13: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/13.jpg)
Fortune 1000
2012 2013 2014 2015 2016 2017
0
100
200
300
400
500
600
700
50
150
+56%
![Page 14: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/14.jpg)
100
200
300
400
500
600
700
Fortune 1000 Culture of Experimentation Index
50
150
2012 2013 2014 2015 2016 2017
0
+56%
+756
%
![Page 15: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/15.jpg)
1,000,000+E X P E R I M E N T S
![Page 16: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/16.jpg)
![Page 17: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/17.jpg)
CULTURE OF EXPERIMENTATION
![Page 18: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/18.jpg)
0
10
100
1000
VE
LO
CIT
Y
MATURITY
10000
![Page 19: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/19.jpg)
VE
LO
CIT
Y
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000
![Page 20: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/20.jpg)
VE
LO
CIT
Y
EXPERIMENTATION
PROGRAM
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000
![Page 21: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/21.jpg)
VE
LO
CIT
Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION
MATURITY
0
10
100
1000
10000
![Page 22: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/22.jpg)
The key to experimentation is to have both the right tools (software) in combination with the right
people and processes
![Page 23: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/23.jpg)
Experimentation Maturity Model
![Page 24: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/24.jpg)
So how do you bring this concept to your organisation?
The first step is to understand what we see as the common attributes within these market leading
organisations.
These attributes when combined together represent a “Culture of Experimentation”
![Page 25: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/25.jpg)
Experimentation is deeply
embedded into company
culture
Dedicated, customised technology
conducts experiments and surfaces
insights
Attributes of Mature “Culture of Experimentation”
Organisations
Enfranchised teams work on
independent optimisation
roadmaps
Central groups manage data
collection and analysis,
empowers individuals with
resources and process
Strategy and metrics for digital
performance directly linked to
company success and widely
circulated
Collective expertise creates virtuous
cycle of learning and best practices
![Page 26: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/26.jpg)
These Culture of Experimentation Organisations would be considered “best-in-class” globally.
So if that is what the end state of building an experimentation program looks like, where do most
organisations stand today?
Most organisations would be considered Experimentation Hero Organisations
![Page 27: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/27.jpg)
Goals and processes poorly
formalised, lack executive
sponsorship
Experimentation are focused
marketing and sales channels
only versus every business
department
Attributes of Early Stage “Experimentation Hero”
Organisations
Consumer intentions and digital
behavior poorly understood
Experimentation treated as series of
tactics to ‘improve conversion rate’ Skill sets and tools divided across
functional silos or not available at all
Lone individuals or small teams
working through design, production,
and analysis despite limited experience
![Page 28: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/28.jpg)
The vast majority of organisations fall somewhere between Mature “Cultures of Experimentation”
and Early Stage “Experimentation Hero” Organisations
…but most exemplify more criteria from the 2nd set of attributes
![Page 29: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/29.jpg)
Mature Cultures of
Experimentation
Early Stage Experimentation Organisations
Embedded into company
culture
Enfranchised teams
Empowered individuals
Dedicated technology
conducts experiments
Metrics linked to company
success
Collective expertise
Lack executive sponsorship
Customer digital behaviors
poorly understood
Experimentation seen only as
Conversion Rate optimisation
Focused solely on sales and
marketing funnels
Tools and skilled people in
silos
Limited expertise available
![Page 30: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/30.jpg)
This is the called the Alignment Gap
The gap between these two sets of attributes represents the key opportunity to create and sustain
a competitive advantage in experimentation
Mature Cultures of
Experimentation
Early Stage Experimentation Organisations
![Page 31: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/31.jpg)
This gap has numerous root causes for why an
organisation may be on one side or the other including
business model, organisational structure, technology
investments, and available trained staff.
Yet, there are measures that any organisation can take
to efficiently leapfrog these hurdles to the front of their
competitive group.
![Page 32: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/32.jpg)
So how do you bridge The Alignment Gap?
![Page 33: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/33.jpg)
To bridge the Alignment Gap your organisation must operationalise
your experimentation program
![Page 34: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/34.jpg)
How
Operationalising
Experimentation
Identify Team and Support Their Growth
Embrace Experimentation Methodology
Understand Operational Metrics
Understand Success Factors
Getting Started Today
![Page 35: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/35.jpg)
Identify Your Team
![Page 36: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/36.jpg)
CORE EXPERIMENTATION TEAMSKILLSETS & TEAM ROLE
Executive Sponsor Project Manager Technical Lead Developer Content
![Page 37: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/37.jpg)
Invest in your most valuable assets:
Identify a Team and Support Professional Growth
Certify Measure Socialise
![Page 38: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/38.jpg)
You can also jumpstart your experimentation program or enhance
your existing team’s capabilities by utilising a best-in-class partner
![Page 39: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/39.jpg)
Once you have a team in place we move to the next key concept
![Page 40: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/40.jpg)
The practice of experimentation follows a defined set of repeatable actions called the
Experimentation Methodology
![Page 41: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/41.jpg)
Ideation
Generate ideas for experiments
Experimentation Methodology
![Page 42: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/42.jpg)
Planning
Design
individual
experiments
and
campaigns
Experimentation Methodology
![Page 43: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/43.jpg)
Developmen
t
Build your experiment and launch it
Experimentation Methodology
![Page 44: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/44.jpg)
Analysis Evaluate the results
of experiment
Experimentation Methodology
![Page 45: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/45.jpg)
Experimentation Methodology
Analysis
Developmen
t
Planning
Ideation
![Page 46: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/46.jpg)
The Experimentation Methodology is universal framework for any sized organisation, but you have
to map it for your business.
To learn more about mapping the Experimentation Methodology please visit the Optimizely Knowledge Base for Optimisation Methodology: Running your
Optimisation Program
![Page 47: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/47.jpg)
Experimentation Maturity Model
![Page 48: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/48.jpg)
Most team’s don’t effectively measure their of use of the Experimentation
Methodology.
This is a missed opportunity.
Structured measurement gives you leading indicators of performance called
Operational Metrics.
![Page 49: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/49.jpg)
These Operational Metrics are the Success Factors for having a world-class
experimentation program
![Page 50: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/50.jpg)
Experimentation Success
Factors
VelocityAgility
Efficiency
Quality
Throughpu
t
Operational Metrics
Impact Rate
Drag
Iteration=
=
=
=
![Page 51: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/51.jpg)
Exp. Methodology Planning Development Ideation Analysis
Operational
MetricsThroughput Drag Impact Rate Iteration
Why it’s
Important
Large percentage of
experiments don’t win.
Improving the quantity of
experiments will ultimately
generate more success.
Knowing how long it takes to
both build an experiment and
how long it takes for each
stage of the experimentation
methodology is important for
measuring overall program
improvement.
To give your team the best
chance of creating a
winning experiment you
need to gather insightful
data to inform hypothesis
generation.
The process of aggregating,
interpreting, and sharing the
experiment results within the
organisation to inform the
next experiment.
Description
(What You Measure)
1.) Overall number of
experiments per property
2.) Number of experiments
per month and week for
each property.
1.) Number of hours spent
redeveloping experiments due
to QA or troubleshooting
issues compared to
‘productive’ time.
2.) Length in hours and days
of the entire experiment
production cycle.
1.) Percentage of
experiments generating
meaningful result (with a
statistically significant
winner or loser).
1.) Percentage of
experiments put into
production and iterated upon.
2.) Number of days from
current experiment to iterated
upon experiment.
Success Factors Velocity Efficiency Quality Agility
Description Quantity of experiments
being ran.
Rate at which tests move
through both Development
stage and the overall
Experimentation Methodology.
Average likelihood that a
given experiment will
produce business impact
(either positive or negative)
The quantity and speed the
experimentation program acts
on experiment results and
iterates upon them.
![Page 52: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/52.jpg)
It is critical to create a habit of optimizing your experimentation program toward these Operational Metrics to ensure you are
continually measuring and improving on your Success Factors.
![Page 53: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/53.jpg)
Now that we know the Experimentation Methodology, Operational Metrics, and Success Factors how do we get started
today?
![Page 54: How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6490057f8b9a82568b4ad1/html5/thumbnails/54.jpg)
The most successful experimentation programs result in business
transformation. Get on this journey today!
Executives
Prioritise and evangelise for experimentation within your business
Identify your team and invest in their success by giving them the time
availability, resources, and goals for the overall experimentation program that
align with business objectives
Executives and Practitioners
Begin mapping the Experimentation Methodology to your organisation
Practitioners
Learn the best practices recommended by Optimizely for starting your
experimentation program
Getting Started Today