optimization works, but nobody said it was easy - antti sivula, frosmo at frosmox16
TRANSCRIPT
An Inconvenient TruthAbout Optimization
FrosmoX16
Antti Sivula 25.10.2016
Optimization is easy! Or is it?
2
• AB testing is simple
• You have an idea for modifying your site so you go ahead
and create an AB test using a testing tool
• After a couple of days you already have a positive result
• This is easy, right?
Optimization is complicated
3
• Getting valid results requires correct testing
• Sample sizes and biases
• Seasonality and business cycles
• Invalid data
• False positives and false negatives
• A cornucopia of alternatives (each with their own pros and cons)
• AB testing vs. MVT
• Multi-armed bandits
• Machine learning
Getting real results is difficult
4
• Are you testing and measuring the right things?
• You have to understand and influence the entirety of your business
• Optimization can quickly become an overwhelming task
• Not enough authority
• Not enough data
• Not enough time
There is no silver bullet
Optimization requires time and effort
6
• Expectations regarding speed, simplicity and immediate impact of
optimization are often unrealistic
• The systems that we want to optimize are usually complex
• Understanding, selecting and implementing appropriate methods is difficult
• Data collection and analysis is paramount
Optimization works
What now?
8
• You should do optimization
• If you’re not optimizing, you are paying an opportunity cost
• Manage expectations and cultivate cooperation
• Lay the foundation for optimization
• Data collection and analysis
• Understand your customers
• Develop your organization
• Start with a single achievable step and work your way up
