What, why and how to A/B test with AI

Download What, why and how to A/B test with AI

Post on 08-Jan-2017

175 views

TRANSCRIPT

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Patrik Matell CONVERSION SPECIALIST CONVERSIONISTA

    What, Why and How to Test with AI

    WEBINAR

    Jonathan Epstein CHIEF MARKETING OFFICER SENTIENT TECHNOLOGIES

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    What is artificial intelligence (AI)? Can it make better decisions than me?

    Do we need another type of testing? Why?

    How can AI be used for A/B and multivariate testing? Can I trust it?

    When should I use AI Testing or not? What does it require?

    Is it possible to get faster results when AI testing? What kind of results?

    Agenda

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    The CRO Wheel - How its done today The reality AI will start at

  • Experimentation is the phase where the optimization wheel hits the reality.

    Where the rubber hits the road

  • Two things will happen - both important

    Movement forward

    Gravel spray

  • The A/B tests two results

    Movement forward

    Gravel spray

    The A/B test will show a positive result, and when you permanently implement the new version you get instant effect on your business.

    Data from the A/B test give you new, and often unexpected, user behaviour insights. You can use those insights to create new experiments, and let the wheel roll forward.

  • The Optimization Wheel

    Data Hypothesis

    Experiments

    Plan & Prioritize

    Research & Analysis

    Behaviour Analysis

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    What is Artificial Intelligence (AI)? Can it Really Make Better

    Decisions than Me?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    A field of computer science focused on building intelligent machines

    An ideal intelligent machine Perceives its environment

    Takes actions that maximize its success against some goal

    Learns from the results of prior actions and adapts

    AI is all around you already! Siri, Waze, Airline pricing

    Is it smarter than me? Depends on what you mean by smart. But it is a lot faster.

    What is Artificial Intelligence?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    AI comes in many shapes and forms

    each of which excels at certain problems.

    DEEP LEARNING EVOLUTIONARY ALGORITHMS

    NEURO-EVOLUTION

    BAYESIAN

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Do we need another type of testing?

    Why?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Agility and experimentation make up the fundamental oxygen for corporate success going forward. If youre not trying 100 ideas or even 1,000 ideas, youll get stuck.

    Peter Diamandis FOUNDER OF THE X PRIZE

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    But

    Only 1 in 7 A/B tests is a success. Visual Website Optimizer

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    And

    Traffic required to get from 5.0% to 5.25% using A/B tests:

    136,000 per sample - Optimizely calculator

    Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    How can AI be used for A/B and multivariate testing?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Evolution: A Proven Efficient Design System

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    FITNESS What makes one candidate (website, trading strategy, animal) more viable than another?

    COMBINATION By combining successful candidates, you can achieve greater fitness, and also prune out bad attributes.

    MUTATION An element of randomness helps discover new successful combinations and ensures you find the best solutionnot just a good one.

    Evolutionary Algorithms: AI Based on Natural Selection

    Sentient Aware uses visual conversation to detect in-the-moment shopper intent and deliver personalized recommendations!

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    After the operator inputs the various test hypotheses, Ascend tests a first generation each individual change is measured against the goal

    Each box represents a design Each letter represents a hypothesis

    Sentient Ascend Applies AI to Testing

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    How Sentient Ascend Works

    + 0 - + - + 0 + +

    and determines which ones have good genetic material.

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    How Sentient Ascend Works

    The system then combines the successful candidates into a new generation to test

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    How Sentient Ascend Works

    and again selects the winners in that population.

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    How Sentient Ascend Works

    The breeding continues, with mutations to ensure we explore all corners.

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    How Sentient Ascend Works

    Within 4 generations you see massive impact; by 8 to 10 generations Ascend converges on the winning design

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    What is Massively Multivariate Testing? Test dozens of site

    improvements at once

    Determine the best combination of those changes to achieve your growth goals

    Evolutionary algorithms can determine the best of 1 million designs while only testing 500-600 of them

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Easy Multipage Funnel Optimization Optimize your website funnelswhether they are a single page, or

    multiple pages

    Test global site elements, too

    Home PDP Cart

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Always-On Continuous Optimization Enables true always-on

    site optimization

    Build on top of your existing tests and add new ideas into the mix

    Whether you add new ideas or not, Ascend always tests in the background and adapts your site presentation to provide the best results if your audience changes for any reason

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    When should I use AI Testing or not? What does it require?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    1. Do I have enough traffic?

    2. Can I get the ROI I need? Base-level Ascend subscription =

    USD 3,000 per month

    0 to 300,000 unique users/month to your properties; no limit on testing

    Consider cost/design tested, cost/hypothesis tested, or cost/increase in KPIs

    Is my site large enough to use this type of testing?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    How long is a test cycle?

    What sort of feedback does the product provide?

    What sorts of goals can I optimize for?

    Will I understand what hypotheses worked, as well as what combinations worked?

    How would AI testing change my workflow?

    How long are AI tests? How are results shown?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Does Ascends AI testing replace A/B testing or would I use both?

    Will this replace my job or make it better? Should I be concerned?

    What does this mean to me & my A/B tests?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    What sort of information/analytics should I be gathering to set up my hypotheses for AI testing?

    What sort of ideas can I test?

    What sort of ideas should I test?

    Can I come up with enough ideas to use AI testing?

    Should I think about hypotheses differently?

  • Copyright 2016 Sentient Technologies LLC. All rights reserved.

    The CRO Wheel - How is it affected? AI optimizations effect on your workflow

  • The Optimization Wheel

    Data Hypothesis

    Experiments

    Plan & Prioritize

    Research & Analysis

    Behaviour Analysis

  • The Prioritization Loop

    Feasibility Analysis

    P.I.E Prioritization

    I.A.R Formula Experiment Roadmap

  • Design

    Develop Deploy & Monitor

    Follow-up

    Experiment Execution

  • Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved. Copyright 2016 Sentient Technologies LLC. All rights reserved.

    Thank you!

    LEARN MORE

    conversionista.se PATRIK@CONVERSIONISTA.SE

    sentient.ai/ascend JONATHAN.EPSTEIN@SENTIENT.AI