is this normal? common challenges testing programs face
TRANSCRIPT
Is This Normal? Debunking myths & getting real about the challenges optimization programs face.
Ryan Garner for Clearhead Jessica Vasbinder for Warner Music Group
Hi. I’m Ryan Garner. !This is Jessica Vasbinder. !!This is the part where we tell you about ourselves. !!!!
(@ryantgarner)
(@entweetment)
We ran our first set of tests. Made buttons bigger, changed copy, tested different colors. And zip, zero… !
Is this normal? !
!
!
!
• It’s okay to crawl before you walk, but set expectations. – You may (probably will) NOT materially improve your bottom
line. !
• Risk (Disruption) & Reward (Lift) are correlated. – Small tests often provide small returns…especially for smaller
businesses!
We don’t know where to start testing or which ideas are good and which are bad or how to keep momentum. !
Is this normal? !
!
!
!
• “Always start at the bottom of the funnel.”
• “Test the whole funnel simultaneously.”
• “NO, Find the bottlenecks IN the funnel.”
• “Navigation is always a good place to test.”
• “Forms, baby, forms.”
Look Inward Look Outward
Your product roadmap User tests!
Your customers’ feedback Competitors
Ideas from around the company Vendors trying to sell you stuff you
Your previous test results Companies you admire
OMFG! We have 1,000+ testing ideas. Lots and lots of them seem good. Now where do we start? !
Is this normal? !
!
!
!
• Create a simple scoring system and get the cream to the top. – It should need no real explanation. – No new acronyms! –Why? No hurt feelings & better ideas!! !
• Then use your own judgment. !• Repeat regularly.
We launched a test that broke our site! I thought building tests was easy. !
Is this normal? !
!
!
!
• Let’s be real here. When you launch a test, you are pushing new code onto your production website. – Should just anyone be doing this? !• Do you really want a high velocity, high returning optimization program? – Of course you do! Otherwise it’s just your normal dev process. !
• So mitigate the risks by.. – Using your best developers and designers. Not your marketers –Monitor, Monitor, Monitor.
Leadership wants a growth estimate from all our winning tests, but when we add it up, it seems too good to be true. !
Is this normal? !
!
!
!
!
• There is no perfect way to do this. !• Stress the difference between statistical significance and
observed lift. !• Trust that there is improvement and move on.
– The only accurate way to confirm lift is to re-test. – Do you really want to do that? !• And if you must show something…
• It’s not as easy as you though it was going to be.
• Your initial efforts don’t double your business.
• You struggle to consistently find great testing ideas.
• You break stuff.
• You are not sure how to accurately measure your return.
Thank you.Ryan Garner [email protected] www.clearhead.me
Jessica Vasbinder [email protected]