digital data tips tuesday

Our testing culture & our testing programMartijn Scheybeler / Leads Growth, SEO & Analytics / [email protected] / @MartijnSch

• How we created a culture of testing?• How were / are we able to scale so fast?• How it influences our (company / teams) culture?• Summary: How can YOU scale your testing program?

What do I like to talk about…

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Why we do what we do?• Team structure & dynamics• Creating experiment ideas + planning• Creating & coding experiments• Analysing experiments• What’s next?

… and it goes something like this


Why we do what we do?


It’s all about:


VELOCITY!


or in other words:


FAIL OFTEN, FAIL FAST


Team Structure & Dynamics


• Design, Dev, QA, Project Management, Legal, CEO?YOU’RE DOING IT WRONG!

• Can you design, code & analyse? You don’t need a team.• Cross-functional or not?

Team structure


Team structure


• 1 Web Analyst: Designing, Coding & Analysis• 2 Interns: Coding, Analysis• Me: Strategy, Coding, Analysis• Support: 1 designer + 1 front-end developer.

Not cross functional (at the moment)!

Team structure


What does our culture looks like?

• Better ask for forgiveness, than permission.• F*CK IT, WE’LL DO IT LIVE!• It’s not rocket science!• We’ll do a much as possible inhouse + open.

TNW ‘Core Values’


• “This is how you access our live servers, need GTM as well?”• “Legal, who?”• “We still have 5 design reviews laying around, DAMMIT!?”• “10 new tests this week! YOLO!”

Team dynamics


“You don’t have decades of experience I see. This is not how it works in the real world.”


- Random Enterprise Employee

Take away: make sure you have buy in!

Boris (CEO) “I had a very good idea this weekend! Can we make experiments more aggressive.”

Experiment Ideas + Planning


“ We don’t want you to run this test! ”


• Ok, we do have a planning• “It doesn’t need to be rocket science”• We don’t have a huge backlog

Ideas + Planning, Planwhat?


• Designer: “I created a new design, we should test it!”• Developer: “I looked at your A/B testing script: refactored it”• Analytics: “We have a backlog but we could use new ideas”• Me: “Can we do 20 tests a week?”

Creating experiments


Deploying in 3..2..1! Flexibility v.s. Scale v.s. €0 Costs

• Designed by our CRO team• Working with our design team to test very big things• Responsible: no way to hide behind bad code• Responsible: communicate everything to the company

Creating experiments


</Coding Experiments>


• Google Tag Manager• A/B Testing & Multivariate Testing: http://tnw.to/testingJS • Custom JavaScript / jQuery for changes• Supported by the dataLayer for targeting / segmenting• We could do better: back-end testing!

Coding experiments


http://bit.ly/testingJS

Analysing Experiments


Automated reporting via TNW Analytics: -20%

Ideas, Backlog, Documentation, Analysis


• Send variant through Custom Dimensions• Track everything with event tracking• ‘Tracking plans’ so we also know what we track

Google Analytics


• Google Analytics Reporting API• Custom Dimensions / Event Tracking / Goals• Automated reporting• Ultimate pleasure and fun + no annoyed team members

Analysing experiments


TNW Analytics: Adding experiments + ideasAutomated reporting via TNW Analytics: -20%

• Grow our team: 20 people?• Cross functional, get developers on board, a mistake?• Automate & Innovate even more• Drink more …?

What’s next?


Create / Embrace a culture of testing: don’t create silos

Make the whole team + organisation come up with ideas

Learn to code so you can create experiments yourself / faster

Want to scale? Make sure you have the resources to automate

Take aways


Martijn ScheybelerLead Growth, SEO & Analytics - TNW

Email: [email protected]: @MartijnSch

Thank you!Slides: http://slideshare.net/MartijnScheijbeler

Code: http://github.com/martijnsch/cro


mailto:[email protected]

http://slideshare.net/MartijnScheijbeler

http://github.com/martijnsch/cro

digital data tips tuesday

Marketing