![Page 1: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/1.jpg)
What we learned from running 200 tests a year?
Martijn Scheijbeler / Director of Marketing / [email protected] / @MartijnSch
![Page 2: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/2.jpg)
Emerce Conversion - @MartijnSch
Title
![Page 3: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/3.jpg)
Martijn Scheijbeler
Marketing Director at The Next WebFormer Lead SEO & Analytics
[email protected] @MartijnSch
![Page 4: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/4.jpg)
ProcessPROCESS
LearningsLEARNINGS
![Page 5: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/5.jpg)
How it allStarted
@MartijnSch
THE START
![Page 6: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/6.jpg)
Why we feelThe need for testing?
@MartijnSch
![Page 7: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/7.jpg)
“So what would we have
to do to grow x10?”
Boris (CEO)
@MartijnSch
“I have no idea what I’m doing”
![Page 8: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/8.jpg)
INDEX
20,000 140,000 6,000visitors startups listed square meters
7,500,000unique visitors
INDEX.COConference
![Page 9: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/9.jpg)
Why am I showing you this slide?
@MartijnSch
![Page 10: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/10.jpg)
MEDIA News
E-commerce
EVENTS Events
Conferences
WORK SPACES TQ
INTELLIGENCE Index
![Page 11: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/11.jpg)
But how did we get to 200 A/B Tests a Year?
@MartijnSch
![Page 12: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/12.jpg)
Testing Figures Data from 2015
2,5 CRO Team in 2015
38.000.000 Sessions with Tests
0 CRO Team in 2014
5-7 Tests per week
Priceless Amounts of fun
35% Traffic v.s. Tests
@MartijnSch
![Page 13: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/13.jpg)
Quantity v.s. Quality?
@MartijnSch
![Page 14: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/14.jpg)
“If you double the number of experiments you’re going to double your inventiveness”
- Jeff Bezos
@MartijnSch
![Page 15: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/15.jpg)
What makes it different: Publishers v.s. E-commerce?
@MartijnSch
![Page 16: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/16.jpg)
The opportunity is 350-400 tests a
year. As ROI per test is low we
need to run a lot to have a positive
return.
Efficiency <> Costs
@MartijnSch
![Page 17: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/17.jpg)
How is our (marketing) team Structured?
@MartijnSch
![Page 18: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/18.jpg)
SEOCRO/ANALYTICSTNW DEALS
INDEX.COCONTENT
GROWTH
@MartijnSch
Front-end Developers
![Page 19: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/19.jpg)
“Better ask for forgiveness than permission”
- Boris Veldhuijzen van Zanten
@MartijnSch
![Page 20: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/20.jpg)
F*CK it, we’ll do it liveIt’s not rocket science
“Innovate through in-house solutions”
@MartijnSch
![Page 21: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/21.jpg)
We failed, not once but also not twice…
@MartijnSch
![Page 22: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/22.jpg)
SO WE STARTED TESTING WITH GOOGLE ANALYTICS CONTENT EXPERIMENTS
GOOGLE TAG MANAGER IS MAKING OUR LIVES EASIER + SCALABLE
![Page 23: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/23.jpg)
WE DIDN’T COMMUNICATE ENOUGH, WE SHOULD HAVE!
MOVING MOST OF OUR COMMUNICATION TO SLACK
![Page 24: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/24.jpg)
What’s ourProcess
@MartijnSch
PROCESS
![Page 25: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/25.jpg)
We needed to learn, Knowledge building?
@MartijnSch
![Page 26: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/26.jpg)
Thanks for all your support!
We learn from you!
![Page 27: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/27.jpg)
FIND THE PERFECT PROCESS TO FIX EVERYTHING > ALWAYS
WHY? MORE EFFICIENCY: THE MORE TESTS, THE HIGHER THE INVENTIVENESS
![Page 28: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/28.jpg)
What are (y)our
Problems?@MartijnSch
![Page 29: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/29.jpg)
“We can’t create any content”“No management approval”
“Whatever b*llshit reason”
@MartijnSch
![Page 30: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/30.jpg)
“Our legal department is your Editorial team”
- Google Employee
@MartijnSch
![Page 31: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/31.jpg)
How did we overcome our fails?
@MartijnSch
![Page 32: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/32.jpg)
We didn’t move to any ‘real’
CRO tool but decided to
iterate on our own: GTM <> JS.
GTM
@MartijnSch
Download:
Github.com/MartijnSch/CRO
![Page 33: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/33.jpg)
Building our own solutions for
saving testing documentation
+ analysis of tests.
Innovate
@MartijnSch
![Page 34: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/34.jpg)
We moved most of our
communication towards
Slack. Making the teams more
involved in the whole process.
Always communicate
@MartijnSch
![Page 35: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/35.jpg)
We hired two front-end
developers, to make sure in
the long-run we can test
bigger things.
Front-end Dev
@MartijnSch
![Page 36: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/36.jpg)
Improvements
5-7 Tests per week
2-4 Tests per week
30-40 Tests per year
20% More efficient
@MartijnSch
![Page 37: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/37.jpg)
What’s in our
Future?@MartijnSch
![Page 38: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/38.jpg)
CONTINUOUS IMPROVEMENT
WHO’S ABLE TO DEPLOY WINNING VARIANTS WITHIN HOURS?
![Page 39: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/39.jpg)
MOVING TESTING TO OUR BACK-END
COLLECTING DATA + SEGMENTATION MORE EASILY + LESS FLICKERING
![Page 40: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/40.jpg)
GO BIG, BIGGER, BIGGEST!
TESTING SMALL DOESN’T WORK (THAT WELL), GO BIGGER!
![Page 41: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/41.jpg)
TLDR: Process
1. Have the right company vision 2. Have the right team structure 3. Have access to top level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation
@MartijnSch
![Page 42: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/42.jpg)
What testsdid we learn from
@MartijnSch
LEARNINGS
![Page 43: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/43.jpg)
MEDIA News
eCommerce
EVENTS Events
Conferences
WORK SPACES TQ
INTELLIGENCE Index
![Page 44: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/44.jpg)
Sharing
@MartijnSch
![Page 45: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/45.jpg)
Sharing brings in more loyalty,
more engagement and
hopefully more traffic.
Share buttons
![Page 46: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/46.jpg)
• Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users
Winner: Variant
Original:
Variant
![Page 47: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/47.jpg)
Engagement
@MartijnSch
![Page 48: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/48.jpg)
Engagement is important,
focus on getting more clicks
on the related stories in the
sidebar.
Rankings!
![Page 49: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/49.jpg)
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users
Winner: Variant
Original: Variant:
![Page 50: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/50.jpg)
Engagement is important,
focus on getting more clicks
on the related stories below
the article.
Related Stories
![Page 51: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/51.jpg)
• Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users
Winner: Variant
Original:Variant
![Page 52: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/52.jpg)
So what if we remove the
whole sidebar, wouldn’t people
be more focused on the stuff
that is out there?
Big, bigger
![Page 53: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/53.jpg)
• Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users
Winner: Original
![Page 54: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/54.jpg)
TLDR: Test learnings
1. Know what you’re testing 2. Focus on your user segments 3. Iterate on what you’re already testing 4. Focus on high significance 5. Test big, bigger, biggest!
@MartijnSch
![Page 55: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/55.jpg)
Now you know how to run200 A/B Tests a Year?
@MartijnSch
![Page 56: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/56.jpg)
But did we get to200 tests?
@MartijnSch
![Page 57: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/57.jpg)
“Why didn’t we run 200 tests?”
“What went wrong?”
“Do I need to start crying?”
Boris (CEO)
![Page 58: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/58.jpg)
“Well, if this guy wouldn’t have stuck
his fingers into a lawn mower, YES!”
Missing out on ±20 tests in 2015.
The Lawn Mower
![Page 59: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/59.jpg)
Take Aways:
1. Focus on running X tests a year 2. Improve the process
3. Have the right (free) tools
4. Have the right people
5. Document/Communicate everything 6. Know what the future looks like 7. Have fun!
@MartijnSch
![Page 60: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/60.jpg)
Emerce Conversion - @MartijnSch
Title
![Page 61: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/61.jpg)
See you atConference?
![Page 62: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/62.jpg)
Thanks! Have fun!
@MartijnSch
![Page 63: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/63.jpg)
LET’S HAVE BEER AT THE ‘BORREL’ OR COFFEE IN THE NEAR FUTURE!
![Page 64: Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?](https://reader031.vdocuments.us/reader031/viewer/2022030312/58ed5de71a28abec378b4605/html5/thumbnails/64.jpg)
Do you have any
Questions?@MartijnSch