science and sensibility: thoughts on experimentation and growth
TRANSCRIPT
![Page 1: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/1.jpg)
Hilary Roberts | Senior Product Manager @Skyscanner | @hilcsr
sciencesensibility&
![Page 2: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/2.jpg)
13 years ago
@hilcsr
![Page 3: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/3.jpg)
today
@hilcsr
![Page 4: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/4.jpg)
0"
10"
20"
30"
40"
50"
60"
monthly unique visitors (millions)
@hilcsr
![Page 5: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/5.jpg)
learn buildminimise total time
measure
lean startup
@hilcsr
![Page 6: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/6.jpg)
startup enterprise
L B
M
L B
M
@hilcsr
![Page 7: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/7.jpg)
startup at scale
L B
M
L B
M
L B
M
L B
M
L B
M
@hilcsr
![Page 8: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/8.jpg)
0
75
150
225
300
Jan-15
Mar-15
May-15
Jul-15
Sept-15
Nov-15
Jan-16
Mar-16
May-16
Jul-16
Sep-16
experiment iterations
@hilcsr
![Page 9: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/9.jpg)
0
75
150
225
300
Jan-15
Mar-15
May-15
Jul-15
Sept-15
Nov-15
Jan-16
Mar-16
May-16
Jul-16
Sep-16
experiment iterations hypotheses
@hilcsr
![Page 10: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/10.jpg)
hypothesis
Based on insight, we predict that product change
will cause impact.
@hilcsr
![Page 11: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/11.jpg)
hypothesis
Based on the insight that travellers disproportionately choose certain holiday
destinations, we predict that ordering destinations by popularity instead of price
will cause more people to convert.
@hilcsr
![Page 12: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/12.jpg)
two possible experiment outcomes
successful failedthe hypothesis was correct the hypothesis was incorrect
@hilcsr
![Page 13: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/13.jpg)
0
75
150
225
300
Jan-15
Mar-15
May-15
Jul-15
Sept-15
Nov-15
Jan-16
Mar-16
May-16
Jul-16
Sep-16
experiment iterations hypotheses
@hilcsr
![Page 14: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/14.jpg)
case study No.1
Based on the insight that active users of our flights product are also likely to like our hotels
product, we predict that landing returning travellers on our hotels homepage instead of
our flights homepage will cause more travellers to use both products.
@hilcsr
![Page 15: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/15.jpg)
case study No.120% increase in travellers booking both flight and hotel
Negligible impact on overall flights metrics
@hilcsr
![Page 16: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/16.jpg)
Skyscanner you are really annoying me with your hotels and car hire options! You are not called HotelScanner or CarHireScanner. You are Sky as in aeroplanes!
If you want to diversify into other services then you should have thought of that before calling your website SKYscanner. Imagine Compare The Market had limited themselves by calling it Compare The Car Insurance Market!
They can now ease into the Travel Insurance market with no issue because they didnt specify which market they are comparing. You cant! and please stop defaulting me to Hotels I'm here for flights!!
@hilcsr
![Page 17: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/17.jpg)
case study No. 120% increase in travellers booking both flight and hotel
Negligible impact on overall flights metrics…but some complaints and added friction for >95% of travellers
bottom line Achieved the desired impact, but qualitatively was not the experience we wanted to provide.
@hilcsr
![Page 18: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/18.jpg)
We could not measure the downside in the experiment.It never could have failed.
invalidnot possible to fail
successful failedthe hypothesis was correct the hypothesis was incorrect
possible experiment outcomes
@hilcsr
![Page 19: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/19.jpg)
invalid tests
Observed outcome: surprisingly one-sided results.
Characterised by inability to measure the upside/downside.
Typically occur when you’re in love with your idea.
Can be avoided by taking hard decisions up-front.
Trigger: “If we can’t agree, why don’t we just test it?”@hilcsr
![Page 20: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/20.jpg)
case study No. 2
Based on the insight that our “Aha!” moment is when travellers conduct their
first search on Skyscanner, we predict that adding search controls to our travel articles will cause more readers to become active
users of our product.@hilcsr
![Page 21: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/21.jpg)
@hilcsr
![Page 22: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/22.jpg)
case study No. 2Negligible increase in percentage of travellers who did a search.
Negligible difference between the variants.
bottom line Our solution didn’t match the context.
Why did we run this test?
@hilcsr
![Page 23: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/23.jpg)
The method we chose was not capable of having the impact we wanted.It never could have succeeded.
invalidnot possible to fail
successful failedthe hypothesis was correct the hypothesis was incorrect
flailednot possible to succeed
possible experiment outcomes
@hilcsr
![Page 24: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/24.jpg)
flailed tests
Observed outcome: Zilch.
Characterised by mismatch between desired impact and proposed solution.
Typically occur when over-focusing on ‘MVP’.
Can be avoided through broader brainstorming.
Trigger: “It’s quick and easy. Let’s just try it.”@hilcsr
![Page 25: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/25.jpg)
invalidnot possible to fail
successful failedthe hypothesis was correct the hypothesis was incorrect
flailednot possible to succeed
two four possible experiment outcomes
@hilcsr
![Page 26: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/26.jpg)
“Any time a team attempts to justify its failures by resorting to learning as an excuse, it is engaged in pseudoscience….
We cannot afford to breed a new pseudoscience around pivots, MVPs, and the like.”
The Lean Startup (page 279)@hilcsr
![Page 27: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/27.jpg)
We cannot afford to breed a new pseudoscience around being data driven.
@hilcsr
![Page 28: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/28.jpg)
Invalid or flailed experiments can look and feel data driven, but they’re pure waste.
@hilcsr
![Page 29: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/29.jpg)
We’re improving our experiment design through experiment and hypothesis templates,
thought experiments and a culture of peer-review.
@hilcsr
![Page 30: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/30.jpg)
To avoid waste, we must balance both science and sensibility.
@hilcsr
![Page 31: Science and Sensibility: Thoughts on Experimentation and Growth](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6e0a527f8b9a505f8b4fb7/html5/thumbnails/31.jpg)
More about what we’re learning at Skyscannerhttp://codevoyagers.com
UX comic library from@steve_cable
Hilary Roberts | @hilcsr
Thank you