digital data tips tuesday

Our testing culture & our testing programMartijn Scheybeler / Leads Growth, SEO & Analytics / martijn@thenextweb.com / @MartijnSch

• How we created a culture of testing?• How were / are we able to scale so fast?• How it influences our (company / teams) culture?• Summary: How can YOU scale your testing program?

What do I like to talk about…

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Why we do what we do?• Team structure & dynamics• Creating experiment ideas + planning• Creating & coding experiments• Analysing experiments• What’s next?

… and it goes something like this

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Why we do what we do?

Digital Data Tips Tuesday - #ddtt - @MartijnSch

It’s all about:

Digital Data Tips Tuesday - #ddtt - @MartijnSch

VELOCITY!

Digital Data Tips Tuesday - #ddtt - @MartijnSch

or in other words:

Digital Data Tips Tuesday - #ddtt - @MartijnSch

FAIL OFTEN, FAIL FAST

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Team Structure & Dynamics

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Design, Dev, QA, Project Management, Legal, CEO?YOU’RE DOING IT WRONG!

• Can you design, code & analyse? You don’t need a team.• Cross-functional or not?

Team structure

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Team structure

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• 1 Web Analyst: Designing, Coding & Analysis• 2 Interns: Coding, Analysis• Me: Strategy, Coding, Analysis• Support: 1 designer + 1 front-end developer.

Not cross functional (at the moment)!

Team structure

Digital Data Tips Tuesday - #ddtt - @MartijnSch

What does our culture looks like?

• Better ask for forgiveness, than permission.• F*CK IT, WE’LL DO IT LIVE!• It’s not rocket science!• We’ll do a much as possible inhouse + open.

TNW ‘Core Values’

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• “This is how you access our live servers, need GTM as well?”• “Legal, who?”• “We still have 5 design reviews laying around, DAMMIT!?”• “10 new tests this week! YOLO!”

Team dynamics

Digital Data Tips Tuesday - #ddtt - @MartijnSch

“You don’t have decades of experience I see. This is not how it works in the real world.”

Digital Data Tips Tuesday - #ddtt - @MartijnSch

- Random Enterprise Employee

Take away: make sure you have buy in!

Boris (CEO) “I had a very good idea this weekend! Can we make experiments more aggressive.”

Experiment Ideas + Planning

Digital Data Tips Tuesday - #ddtt - @MartijnSch

“ We don’t want you to run this test! ”

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Ok, we do have a planning• “It doesn’t need to be rocket science”• We don’t have a huge backlog

Ideas + Planning, Planwhat?

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Designer: “I created a new design, we should test it!”• Developer: “I looked at your A/B testing script: refactored it”• Analytics: “We have a backlog but we could use new ideas”• Me: “Can we do 20 tests a week?”

Creating experiments

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Deploying in 3..2..1! Flexibility v.s. Scale v.s. €0 Costs

• Designed by our CRO team• Working with our design team to test very big things• Responsible: no way to hide behind bad code• Responsible: communicate everything to the company

Creating experiments

Digital Data Tips Tuesday - #ddtt - @MartijnSch

</Coding Experiments>

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Google Tag Manager• A/B Testing & Multivariate Testing: http://tnw.to/testingJS • Custom JavaScript / jQuery for changes• Supported by the dataLayer for targeting / segmenting• We could do better: back-end testing!

Coding experiments

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Analysing Experiments

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Automated reporting via TNW Analytics: -20%

Ideas, Backlog, Documentation, Analysis

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Send variant through Custom Dimensions• Track everything with event tracking• ‘Tracking plans’ so we also know what we track

Google Analytics

Digital Data Tips Tuesday - #ddtt - @MartijnSch

• Google Analytics Reporting API• Custom Dimensions / Event Tracking / Goals• Automated reporting• Ultimate pleasure and fun + no annoyed team members

Analysing experiments

Digital Data Tips Tuesday - #ddtt - @MartijnSch

TNW Analytics: Adding experiments + ideasAutomated reporting via TNW Analytics: -20%

• Grow our team: 20 people?• Cross functional, get developers on board, a mistake?• Automate & Innovate even more• Drink more …?

What’s next?

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Create / Embrace a culture of testing: don’t create silos

Make the whole team + organisation come up with ideas

Learn to code so you can create experiments yourself / faster

Want to scale? Make sure you have the resources to automate

Take aways

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Martijn ScheybelerLead Growth, SEO & Analytics - TNW

Email: martijn@thenextweb.comTwitter: @MartijnSch

Thank you!Slides: http://slideshare.net/MartijnScheijbeler

Code: http://github.com/martijnsch/cro

Digital Data Tips Tuesday - #ddtt - @MartijnSch

Digital Data Tips Tuesday - #ddtt - @MartijnSch