testing for success

Download Testing for Success

Post on 19-May-2015

176 views

Category:

Technology

0 download

Embed Size (px)

DESCRIPTION

The presentation discusses the significance of testing and how to execute a successful testing program.

TRANSCRIPT

  • 1. > Tes&ng for Success < Elements of a Successful Tes0ng Program

2. > Agenda Why Test? Problem Diagnosis Deciding what to Test Test Execu0on and Measurement Test Repor0ng December 2011 Datalicious Pty Ltd 2 3. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010 > Why Test? December 2011 Datalicious Pty Ltd 3 4. 1. Why does your EVERYONES business/organisa0on exist? GOT AN 2. How can your business/OPINION organisa0on improve? December 2011 Datalicious Pty Ltd 4 5. > Why Test? 1. Systema0c Innova0on 2. Avoid costly mistakes 3. Know why things go right, know why things go wrong 4. BeRer employee engagement Requires planning and governance! December 2011 Datalicious Pty Ltd 5 6. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010 > Problem Diagnosis December 2011 Datalicious Pty Ltd 6 7. > What is the business problem? Acquisi0on Up-Sell Reten0on Advocacy Analy&cs and metrics frameworks December 2011 Datalicious Pty Ltd 7 8. > Case Study December 2011 Datalicious Pty Ltd 8 9. > Further Diagnosis PROBLEM: Sales through online Not enough site trac High home page bounce rate Low conversion on product page Checkout fallout December 2011 Datalicious Pty Ltd 9 10. > Further Diagnosis II Source: www.feng-gui.com December 2011 Datalicious Pty Ltd 10 11. > Some&mes the small things count December 2011 Datalicious Pty Ltd 11 12. > Further diagnosis III Wrong message? Wrong channel? Wrong person? Wrong 0me? December 2011 Datalicious Pty Ltd 12 13. > Tes&ng as risk mi&ga&on Roll-out Channel Press TV Radio Outdoor Oer, Crea&ve, Call-to- Oer, Call- Oer, Call- eDM/DM Call-to- Ac&on to-Ac&on to-Ac&on Ac&on Test Paid Channel Search Oer Oer Oer Oer Crea&ve, Display Oer, Call-Oer, Call- - Crea&ve Media to Ac&on to Ac&on December 2011 Datalicious Pty Ltd 13 14. > Tes&ng as standard prac&ce Test Market Control Market (no ATL) % Uplib in Sales TimeDecember 2011 Datalicious Pty Ltd 14 15. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010 > Deciding what to Test December 2011 Datalicious Pty Ltd 15 16. Dont reinvent the wheel December 2011 Datalicious Pty Ltd 16 17. > What are the solu&on(s)? December 2011 Datalicious Pty Ltd 17 18. > Consumer Empathy What are your visitors trying to achieve by visi2ng your site? December 2011 Datalicious Pty Ltd 18 19. > Consumer Empathy 1. Make it visible People cant convert if they cant nd your Buy Now buRon 2. Make it relevant Need to resolve consumer reserva0ons/ques0ons 3. Make it easy Easy naviga0on, easy form comple0on, easy to read, quick page load December 2011 Datalicious Pty Ltd 19 20. > Start with the basics 1. The headline Have a headline! Headline should be concrete Headline should be rst thing visitors look at 2. Call to ac&on Dont have too many calls to ac0on Have an ac0onable call to ac0on Have a big, prominent, visible call to ac0on 3. Social proof Logos, number of users, tes0monials, case studies, media coverage, etc December 2011 Datalicious Pty Ltd 20 21. > Start with the basics December 2011 Datalicious Pty Ltd 21 22. > Case Study December 2011 Datalicious Pty Ltd 22 23. > Further Examples TEST A EXISTING December 2011 Datalicious Pty Ltd 23 24. > Further Examples EXISTING TEST December 2011 Datalicious Pty Ltd 24 25. > Deciding What to Test Test Selec0on Checklist Is the measurement infrastructure in place already? [ ] Can I readily execute the solu0on? [ ] Do I have enough sample to draw valid conclusions? [ ] Will this prove the value of tes0ng in the business? [ ] December 2011 Datalicious Pty Ltd 25 26. > Do you have the repor&ng? For each of Segment X, Y and Z... Test Channel ATL DM eDM Online Online Mailroom Response Call Centre Channel Bricks & Mortar Channels in Aggregate December 2011 Datalicious Pty Ltd 26 27. > Oine conversions from online Tying oine conversions back to online campaign and research behavior using standard cookie technology by triggering virtual online order conrma0on pages for oine sales using email receipts. Website.com Phone Virtual Order Research Orders @ Conrma&on Online Ad Website.com Retail Virtual Order Campaign Research Orders @ Conrma&on Website.com Online Online Order Virtual Order Research Orders Conrma&on @ Conrma&on Cookie Cookie Cookie December 2011 Datalicious Pty Ltd 27 28. > Search call to ac&on for oine December 2011 Datalicious Pty Ltd 28 29. > OTP Response Dierent numbers for dierent media channels Dierent numbers for dierent product categories Dierent numbers for dierent conversion steps Call origin becoming useful to shape call script Feasible to pause numbers to improve integrity December 2011 Datalicious Pty Ltd 29 30. > Whose help do you need? Technology/IT UX AgencyAnalytics!Your boss, Your boss boss Creative Agency Customer Contact ManagementDecember 2011 Datalicious Pty Ltd 30 31. > Proving the Value GO BIG December 2011 Datalicious Pty Ltd 31 32. > How much sample do I need? BAU/Baseline Conversion Rate # on Segments, # of Treatments n Expected in Conversion Time in Market [Digital Only] December 2011 Datalicious Pty Ltd 32 33. > Sta&s&cal Signicance Q. How much am I willing to accept that the dierence in the results between my test group and control group may have been due to chance? A. Not much. I want to be condent that if I repeated the test 100 &mes, then I would observe this dierence 95 &mes. This is 95% condence December 2011 Datalicious Pty Ltd 33 34. > Type I and Type II Error Type I: Accept result to be true when its actually false (false posi&ves) Type II: Accept result to be false when its actually true (false nega&ves) December 2011 Datalicious Pty Ltd 34 35. > Es&ma&ng Sample Size (%s) 2 # p1 (1 p1 ) + p2 (1 p2 ) &n = (1.645+1.282) * % 2($ Where: n = es0mated sample size for each group p1 = expected conversion rate for your test treatment p2 = expected conversion rate for your control treatment = expected minimum percentage point dierence between test and control results The value of 1.645 reects that we accept Type I error probability of .05 The value of 1.282 reects that we accept Type II error probability of .10 December 2011 Datalicious Pty Ltd 35 36. > Es&ma&ng Sample Size (%s) Typical Champion (control) vs. Challenger (test) A|B test, typical champion response rate of 2.5%. Only going to replace Champion with Challenger if Challenger response rate is 3.0% (0.5% is a meaningful dierence) 2 ! 0.025* 0.975 + 0.030 * 0.970 $n = (1.645+1.282) * #2&"0.005%Sample size = 18,326 For each of the Champion and Challenger groups If 1.0% our meaningful dierence then sample size is only 5,378 December 2011 Datalicious Pty Ltd 36 37. > Es&ma&ng Sample Size ($s) (1.645 +1.282)2 * (s12 + s2 ) 2n=2 Where: n = number of observa0ons for each group s1 = expected standard devia0on of value for your test treatment s2 = expected standard devia0on of value for your control treatment = expected minimum dierence in value between test and control results The value of 1.645 reects an accepted Type I error probability of .05 The value of 1.282 reects an accepted Type II error probability of .10 December 2011 Datalicious Pty Ltd 37 38. > Standard Devia&on Standard devia0on is measure of the variability of your results, whether some your results are quite dierent to your mean (average) result or whether they are quite similar. n (x x ) i i=1s= n 1 Where: n = number of observa0ons xi = the result for the ith observa0on x = mean (average) for your data December 2011 Datalicious Pty Ltd 38 39. > Es&ma&ng Sample Size ($s) Typical Champion (control) vs. Challenger (test) A|B test, typical champion mean response value of $20, typical response rate of 5% Only going to replace Champion with Challenger if Challenger mean response value is is $30 ($10 is a meaningful dierence) Standard devia0on of Champion results is $5 (based on past results). Well assume the same for the Challenger. 2 22 (1.645 +1.282) * (5 + 5 ) n= 2 10Number of observa0ons = 4.3 (~5) for each of the Champion and Challenger groups. Then divide through with the expected response rate to get minimum sample size of 86 for each of Challenger and Control groups (4.3/0.05) December 2011 Datalicious Pty Ltd 39 40. > Further Complexity I If we wanted to test the performance of Challenger vs. Champion for dierent segments of consumers: Response Rate Champion Challenger A % % Segment B % % C % % Using same assump0ons as in earlier example need 18,326 per cell, 18,326*6=109,956 in total . December 2011 Datalicious Pty Ltd 40 41. > Further Complexity II If we wanted to test the performance of Challenger vs. Champion for dierence segments of consumers AND had 3 dierent types of Champion crea0ve: Response Rate Challenger Challenger Challenger Champion #1 #2 #3 A % % % % Segment B % % % % C % % % % Using same assump0ons as in earlier exam