Download - What, why and how to A/B test with AI

Copyright © 2016 Sentient Technologies LLC. All rights reserved. Copyright © 2016 Sentient Technologies LLC. All rights reserved. Copyright © 2016 Sentient Technologies LLC. All rights reserved.

Patrik Matell CONVERSION SPECIALIST CONVERSIONISTA

What, Why and How to Test with AI

WEBINAR

Jonathan Epstein CHIEF MARKETING OFFICER SENTIENT TECHNOLOGIES


•  What is artificial intelligence (AI)? Can it make better decisions than me?

•  Do we need another type of testing? Why?

•  How can AI be used for A/B and multivariate testing? Can I trust it?

•  When should I use AI Testing or not? What does it require?

•  Is it possible to get faster results when “AI testing”? What kind of results?

Agenda

Copyright © 2016 Sentient Technologies LLC. All rights reserved.

The CRO Wheel - How it’s done today The reality AI will start at

Experimentation is the phase where the optimization wheel hits the reality.

“Where the rubber hits the road”

Two things will happen - both important

Movement forward

Gravel spray

The A/B tests two results

Movement forward

Gravel spray

The A/B test will show a positive result, and when you permanently implement the new version you get instant effect on your business.

Data from the A/B test give you new, and often unexpected, user behaviour insights. You can use those insights to create new experiments, and let the wheel roll forward.

The Optimization Wheel

Data Hypothesis

Experiments

Plan & Prioritize

Research & Analysis

Behaviour Analysis


What is Artificial Intelligence (AI)? Can it Really Make Better

Decisions than Me?


•  A field of computer science focused on building “intelligent” machines

•  An ideal “intelligent” machine

•  Perceives its environment

•  Takes actions that maximize its success against some goal

•  Learns from the results of prior actions and adapts

•  AI is all around you already!

•  Siri, Waze, Airline pricing

•  Is it smarter than me?

•  Depends on what you mean by smart. But it is a lot faster.

What is Artificial Intelligence?


AI comes in many shapes and forms…

…each of which excels at certain problems.

DEEP LEARNING EVOLUTIONARY ALGORITHMS

NEURO-EVOLUTION

BAYESIAN


Do we need another type of testing?

Why?

Copyright © 2016 Sentient Technologies LLC. All rights reserved. Copyright © 2016 Sentient Technologies LLC. All rights reserved.

“ Copyright © 2016 Sentient Technologies LLC. All rights reserved.

“ Agility and experimentation make up the fundamental oxygen for corporate success going forward. If you’re not trying 100 ideas or even 1,000 ideas, you’ll get stuck.

Peter Diamandis FOUNDER OF THE X PRIZE


But…

Only 1 in 7 A/B tests is a success. – Visual Website Optimizer


And…

Traffic required to get from 5.0% to 5.25% using A/B tests:

136,000 per sample - Optimizely calculator

Copyright © 2016 Sentient Technologies LLC. All rights reserved. Copyright © 2016 Sentient Technologies LLC. All rights reserved.


How can AI be used for A/B and multivariate testing?


Evolution: A Proven Efficient Design System


FITNESS What makes one candidate (website, trading strategy, animal) more viable than another?

COMBINATION By combining successful candidates, you can achieve greater fitness, and also prune out bad attributes.

MUTATION An element of randomness helps discover new successful combinations and ensures you find the best solution—not just a good one.

Evolutionary Algorithms: AI Based on Natural Selection

Sentient Aware uses visual conversation to detect in-the-moment shopper intent and deliver personalized recommendations!


After the operator inputs the various test hypotheses, Ascend tests a first generation – each individual change is measured against the goal…

Each box represents a design Each letter represents a hypothesis

Sentient Ascend Applies AI to Testing


How Sentient Ascend Works

+ 0 - + - + 0 + +

…and determines which ones have good “genetic material.”



The system then combines the successful candidates into a new generation to test…



…and again selects the winners in that population.



The breeding continues, with mutations to ensure we explore all corners.



Within 4 generations you see massive impact; by 8 to 10 generations Ascend converges on the winning design


What is Massively Multivariate Testing? •  Test dozens of site

improvements at once

•  Determine the best combination of those changes to achieve your growth goals

•  Evolutionary algorithms can determine the best of 1 million designs while only testing 500-600 of them


Easy Multipage Funnel Optimization •  Optimize your website funnels—whether they are a single page, or

multiple pages

•  Test global site elements, too

Home PDP Cart


Always-On Continuous Optimization •  Enables true always-on

site optimization

•  Build on top of your existing tests and add new ideas into the mix

•  Whether you add new ideas or not, Ascend always tests in the background and adapts your site presentation to provide the best results if your audience changes for any reason


When should I use AI Testing or not? What does it require?


1.  Do I have enough traffic?

2.  Can I get the ROI I need? •  Base-level Ascend subscription =

USD 3,000 per month

•  0 to 300,000 unique users/month to your properties; no limit on testing

•  Consider cost/design tested, cost/hypothesis tested, or cost/increase in KPIs

Is my site large enough to use this type of testing?


•  How long is a test cycle?

•  What sort of feedback does the product provide?

•  What sorts of goals can I optimize for?

•  Will I understand what hypotheses worked, as well as what combinations worked?

•  How would AI testing change my workflow?

How long are AI tests? How are results shown?


•  Does Ascend’s AI testing replace A/B testing or would I use both?

•  Will this replace my job or make it better? Should I be concerned?

What does this mean to me & my A/B tests?


• What sort of information/analytics should I be gathering to set up my hypotheses for AI testing?

• What sort of ideas can I test?

• What sort of ideas should I test?

• Can I come up with enough ideas to use AI testing?

Should I think about hypotheses differently?


The CRO Wheel - How is it affected? AI optimization’s effect on your workflow

The Optimization Wheel

Data Hypothesis

Experiments

Plan & Prioritize

Research & Analysis

Behaviour Analysis

The Prioritization Loop

Feasibility Analysis

P.I.E Prioritization

I.A.R Formula Experiment Roadmap

Design

Develop Deploy & Monitor

Follow-up

Experiment Execution


Thank you!

LEARN MORE

conversionista.se [email protected]

sentient.ai/ascend [email protected]

Download - What, why and how to A/B test with AI

Top Related