business model innovation by experimentation
DESCRIPTION
How to maximize learning and minimize risk All new products start as a series of unvalidated assumptions. The most critical assumptions are usually implicit and relate to the purpose of the product and the value it is intended to deliver. The more key assumptions involved, the greater the risk. It is enough to have 7 key assumptions about which you are 90% certain for the combined odds of success to be below 50%. Contrary to popular belief, when we know very little about a situation, it only takes a small amount of new data to realise significant insights. Unfortunately, people often underestimate the value of information and misunderstand risk. As Product Owners we are often afraid to test our assumptions. We routinely pile on additional risk without a second thought. Risk management is the bread and butter of the finance and insurance industries. Isn't it time we evolved? In this fast paced and practical session we will explore answers to the following questions: - What is risk and how do we quantify and manage it? - How do we assess the value of information? - How can experimentation reduce risk and where does it fit in the product development cycle? - What makes a good experiment? - How to run experiments in a cost effective manner? - What are good metrics?TRANSCRIPT
![Page 1: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/1.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Business Model Innovation by ExperimentationYoav AviramEnergized Work@yobo
![Page 2: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/2.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
IT Projects60% don’t meet schedule, budget or quality goals - IBM
50% failed to achieve what they set out to achieve - KPMG
17% go so badly that they can threaten the very existence of the
company - Calleam Consulting
![Page 3: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/3.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Innovative IT Projects (startups) 95% of technology startups fail - Allmand Law
93% of Angel investments never achieve expected ROI - University of
Washington
80% of VC investments never achieve expected ROI - National VC
Association
![Page 4: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/4.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
WHY?
![Page 5: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/5.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Software, the Tip of the Iceberg
Too late
Poor quality
Missing functionality
![Page 6: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/6.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Agile
An iterative and incremental approach to development, where
requirements and solutions evolve through collaboration between self-
organizing, cross-functional teams
- Wikipedia
![Page 7: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/7.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Who is the customer?
How do we acquire new users?
How do we convert users to paying customers?
How should the product look and behave?
How is our product used?
What problems does the product attempt to solve?
What features provide the greatest value?
...the Product Iceberg
![Page 8: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/8.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Lean Startup
A combination of business-hypothesis-driven experimentation, iterative
product releases, and "validated learning"
- The Lean Startup, Eric Ries
![Page 9: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/9.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
...the Business Model IcebergWho are our customers?
Who is the competition?
How do we reach our customers?
What are the revenue streams?
What is the investment needed?
What is the operational cost?
What is our unique selling point?
What team members to recruit?
What are our business goals?
![Page 10: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/10.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Business Model
A plan for the successful operation of a business, identifying sources of
revenue, the intended customer base, products, and details of
financing
- Oxford Dictionary
![Page 11: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/11.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Business Model Innovation
- Business Model Generation, Alexander Osterwalder
![Page 12: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/12.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Product-Market Fit
There is a poorly met need a reachable market faces
There is a solution people are happy to pay for
There is a way to package and deliver the solution in a cost effective
manner
![Page 13: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/13.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Think About Risk
![Page 14: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/14.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Uncertainty
The lack of complete certainty, that is, the existence of more than one
possibility. The "true" outcome/state/result/value is not known.
- How to Measure Anything, Douglas Hubbard
![Page 15: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/15.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Quantifying Uncertainty
A set of probabilities assigned to a set of possibilities
“There is a 60% chance this market will double in five years”
![Page 16: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/16.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Risk
A state of uncertainty where some of the possibilities involve a loss,
catastrophe, or other undesirable outcome.
- How to Measure Anything, Douglas Hubbard
![Page 17: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/17.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Quantifying Risk
A set of possibilities each with quantified probabilities and quantified
losses
“There is a 40% chance the proposed oil well will be dry, with a loss of
$12 million in exploratory drilling costs”
![Page 18: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/18.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
The Problem with RiskTechnology is used to solve complex problems
Our mind is wired to oversimplify complexity (we are not very good at
understanding probabilities)
We are solution oriented
We are optimistic (overconfident) by nature
People (grossly) underestimate risk & uncertainty
![Page 19: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/19.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Compound Risk
Combination of two or more related risks
Compound probabilities are (very) counter intuitive
![Page 20: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/20.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Monty Hall
A contestant on a game show is given the choice of three doors.
Behind one door is a new car, behind the others, goats.
After a contestant picks a door, the host, who knows what's behind all
the doors, opens an unchosen door, which reveals a goat.
He then asks the contestant, "Do you want to switch doors?"
![Page 21: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/21.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Should the contestant switch doors?
![Page 22: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/22.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Yes!The odds of winning are 2 out of 3 if you
switch
![Page 23: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/23.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Compound Probabilities
This answer is so counter intuitive, most people get it wrong
It’s why they made a game show off of it: the trick works every time
The lesson:
We need to give serious consideration to risk
We must not rely on our intuition when considering risk & uncertainty
![Page 24: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/24.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Risk Management 101Avoid by eliminating the situation or activity that presents it
Transfer through insurance or through other types of contracts
Reduce by hedging your bets or reducing uncertainty
Retain: because some risks are worth assuming
![Page 25: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/25.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Risk Reduction
![Page 26: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/26.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
The Investor Mindset
Spread your bets on a portfolio of investments instead of one
Decide on an investment strategy
Split the investment into small incremental bets: abort bad products
quickly and ramp up investment in the good ones
![Page 27: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/27.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Uncertainty Reduction
![Page 28: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/28.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Value of InformationInformation reduces uncertainty
Reduced uncertainty improves decisions
Improved decisions have observable consequences with measurable
value
- Information Theory (1948)
![Page 29: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/29.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Sources of InformationExisting within the company
Other’s research
Experiments
Live product
![Page 30: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/30.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
A scientific procedure undertaken to make a discovery, test
a hypothesis, or demonstrate a known fact- Wikipedia
Experiment
![Page 31: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/31.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Experimental Discovery
![Page 32: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/32.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Prioritising Assumptions Uncertainty around making a decision is high (little existing information)
The value of the opportunity, or the cost of a mistake, is high
Experimentation is least expensive in terms of cost and time
![Page 33: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/33.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Designing Good Experiments
Attempt to falsify a hypothesis
Are objective, measurable and repeatable
Are controlled (variables tested in isolation)
Are cost effective
Influence a decision
Experiments aren't free
It's an investment decision
Information may be cheaper elsewhere
![Page 34: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/34.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Measurement
A measurement is an observation that results in information (reduction
of uncertainty) about a quantity
![Page 35: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/35.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Qualitative Measurements
The soft stuff
Subjective, messy and hard to quantify
Customer surveys, focus groups, interviews
But … provide insights about Perceived Value
![Page 36: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/36.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Good Qualitative QuestionsAvoid leading phrases such as "do you agree that..."
Make customers part with money or sign up, rather than asking them whether they would
Use follow up questions to get to the ‘why?’
![Page 37: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/37.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Quantitative Measurements
Start by decide on the right Metric (what to measure)
Then set lines in the sand: what success and failure look like
Always validate results with qualitative data
![Page 38: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/38.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Good MetricInfluences a decision
Comparative
Rates and Ratios are easier to compare and act upon
Measure what’s important to your customers
![Page 39: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/39.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Putting it all TogetherThe key to the success of IT Projects is better risk management
Information reduces uncertainty & risk
Experiments are the most effective way to obtain new information
Experiments have the biggest impact in a high value, high uncertainty
and little data situations
Hedge your bets by making small incremental investments
Challenge the distinction between business people and techies
![Page 40: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/40.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Reading List
The Lean Startup by Eric Ries
Business Model Generation by Alexander Osterwalder
How To Measure Anything by Douglas W. Hubbard
Lean Analytics by Alistair Croll and Benjamin Yoskovitz
![Page 41: Business Model Innovation by Experimentation](https://reader033.vdocuments.us/reader033/viewer/2022052900/555b1adad8b42aa9738b4d2b/html5/thumbnails/41.jpg)
© 2013 ENERGIZED WORK LIMITED - ALL RIGHTS RESERVED - WWW.ENERGIZEDWORK.COM
Questions?
Yoav AviramEnergized Work
[email protected]@yobo