growth experiments: lessons from a failed attempt
TRANSCRIPT
growthGrowth Experiments: Lessons from a failed attempt
about meVassilena Valchanova
• 10 years of marketing experience
• Check out valchanova.me
• Digital Strategist @ Enhancv
growthAbout Enhancv
• B2C SaaS startup
• Smart resume building platform
• 3 years old
• 300,000 users
our storyOur Growth Team Story
WHY?
Speed up and learn more
HOW?
Company-wide growth
experiment process
our storyThe Death of Experiments
• 10 experiment meetings
• 6 experiments carried out
• Participation from
different team members
our storyTime to Reset
RESET
The Growth ProcessPROCESS
GROWTH
Analyze
1.
Set
Objective
2.
Ideate
3.
Prioritize 4.
Test
5.
growthBuild YOUR TEAM
setupBuild Your TeamGROWTH LEAD
• Sets the process
• Leads the meetings
• Clears obstacles
MARKETER
• Acquisition
• Copy
• Driving traffic
PRODUCT/DEV
• Technical
implementation
• Automation
DESIGNER
• Mockups
• Wireframes
• Marketing visuals
DATA ANALYST
• Initial data
• Experiment setup
• Analysis
CUSTOMER SUCCESS
• Customer feedback
• Critical questions
• Early results
our storyAlways Use The Same Team
• Ownership
• Helps them prepare
• Teams require time to
form
growthEstablish CLEAR RULES
setupClear Rules
• Is the experiment clear and
specific?
• Is the experiment related to the
objective?
• Can it be executed and analyzed
in 2 weeks?
setupDictatorship is good in meetings
growthStart with a SPECIFIC PROBLEM
analyzeSpecific Problem
• Work with data
• Growth lead and management
decide
• Concrete objective
analyzeWhat’s Concrete?
Our current blog new visitors is 4,000 per month.
We need to increase it by 15% without losing the
quality of the audience.
Let’s focus on acquiring new users
growthRespect everyone’s TIME
ideateICE scores are cool1-10 average of three scores:
IMPACT
If successful, how big is the difference?
CONFIDENCE
How sure are we it will work?
EASE
How easy is it to implement?
(time/people/money)
ideateRespect everyone’s time
• Team members write down
ideas before the meeting.
• Each scores their idea.
• Only doubtful scores are
discussed.
• Scores are not the last word.
growthEnsure there is TEST CADENCE
testTest Cadence
• Two week experiment
cycles
• 5 experiments per cycle
• 30% success rate
• 32 successful experiments
per year
growthSetup experiments THE RIGHT WAY
testExperiment TypesA/B Tests Testing 2 versions simultaneously, only 1 difference per version.
Multivariate Tests Testing many versions at the same time, lots of differen-ces per version.
Sequential Tests Testing many versions one after the other.
Smoke Tests Landing page collecting leads to gauge interest.
Trigger Tests Measuring click rate of ads to see interest in a value proposition.
Wizard of Oz Deliver value by hand, if a feature requires lots of dev.
High Hurdle People pay with time (surveys, long forms) to see how engaged they are with a proposition.
Design your Own There are other options…
testGet to Know Statistics
• VWO’s Significance Calculator
• Optimizely’s Sample Size Calculator
• Google Optimize
growthSettle on a working TOOLSET
testTools to Try
growthThank you! Any Questions? The slides: valchanova.me/slides