growth experiments: lessons from a failed attempt

growthGrowth Experiments: Lessons from a failed attempt

about meVassilena Valchanova

• 10 years of marketing experience

• Check out valchanova.me

• Digital Strategist @ Enhancv

growthAbout Enhancv

• B2C SaaS startup

• Smart resume building platform

• 3 years old

• 300,000 users

our storyOur Growth Team Story

WHY?

Speed up and learn more

HOW?

Company-wide growth

experiment process

our storyThe Death of Experiments

• 10 experiment meetings

• 6 experiments carried out

• Participation from

different team members

our storyTime to Reset

RESET

The Growth ProcessPROCESS

GROWTH

Analyze

1.

Set

Objective

2.

Ideate

3.

Prioritize 4.

Test

5.

growthBuild YOUR TEAM

setupBuild Your TeamGROWTH LEAD

• Sets the process

• Leads the meetings

• Clears obstacles

MARKETER

• Acquisition

• Copy

• Driving traffic

PRODUCT/DEV

• Technical

implementation

• Automation

DESIGNER

• Mockups

• Wireframes

• Marketing visuals

DATA ANALYST

• Initial data

• Experiment setup

• Analysis

CUSTOMER SUCCESS

• Customer feedback

• Critical questions

• Early results

our storyAlways Use The Same Team

• Ownership

• Helps them prepare

• Teams require time to

form

growthEstablish CLEAR RULES

setupClear Rules

• Is the experiment clear and

specific?

• Is the experiment related to the

objective?

• Can it be executed and analyzed

in 2 weeks?

setupDictatorship is good in meetings

growthStart with a SPECIFIC PROBLEM

analyzeSpecific Problem

• Work with data

• Growth lead and management

decide

• Concrete objective

analyzeWhat’s Concrete?

Our current blog new visitors is 4,000 per month.

We need to increase it by 15% without losing the

quality of the audience.

Let’s focus on acquiring new users

growthRespect everyone’s TIME

ideateICE scores are cool1-10 average of three scores:

IMPACT

If successful, how big is the difference?

CONFIDENCE

How sure are we it will work?

EASE

How easy is it to implement?

(time/people/money)

ideateRespect everyone’s time

• Team members write down

ideas before the meeting.

• Each scores their idea.

• Only doubtful scores are

discussed.

• Scores are not the last word.

growthEnsure there is TEST CADENCE

testTest Cadence

• Two week experiment

cycles

• 5 experiments per cycle

• 30% success rate

• 32 successful experiments

per year

growthSetup experiments THE RIGHT WAY

testExperiment TypesA/B Tests Testing 2 versions simultaneously, only 1 difference per version.

Multivariate Tests Testing many versions at the same time, lots of differen-ces per version.

Sequential Tests Testing many versions one after the other.

Smoke Tests Landing page collecting leads to gauge interest.

Trigger Tests Measuring click rate of ads to see interest in a value proposition.

Wizard of Oz Deliver value by hand, if a feature requires lots of dev.

High Hurdle People pay with time (surveys, long forms) to see how engaged they are with a proposition.

Design your Own There are other options…

testGet to Know Statistics

• VWO’s Significance Calculator

• Optimizely’s Sample Size Calculator

• Google Optimize

growthSettle on a working TOOLSET

testTools to Try

growthThank you! Any Questions? The slides: valchanova.me/slides