qa: how i learned to stop breaking and start loving tests

OPTIMIZATION QA: How to Avoid Launching Tests that Break Your SiteWhen you break shit you pollute data, lose money and eventually get fired. Here’s how to avoid that pain and look like a hero in the process.

@joeljharvey #convcon

Joel Harvey, COO Conversion Sciences


Goal


What’s QA?


Let’s say that …


30% of your visitors

get a broken experience on

your site


YOUR

SITE IS

BROKEN ON ONE OF

THESE

SCREENS


No One On Your Team Knows


20% RPVIE BUG


INTERNETEXPLORER


You’re about to hire this guy as your CPA


QA MATH


Internet Explorer Traffic: Feb 2013

Internet Explorer Traffic: Feb 2014


120% Increase Internet Explorer (IE) Revenue

20% Increase in All Revenue

20% Increase in IE Average Order Value

167% Increase in IE Conversion Rate

$1,500,000 Per Year


Who Are You?


Who Am I?- COO & Managing Partner,

Conversion Sciences- Conversion Scientist- Data Geek- Entrepreneurial MBA


1. RE-DEFINE QA

2. QA STRATEGY

3. QA TOOLS

4. QA PROCESS

RE-DEFINING QA

#1


STOP STEALING FROM YOURSELF


30% of your visitors

get a broken experience on

your site

QA DRIVES REVENUE


OLD BORING DEFINITION


QA is just a Cost Center

We can’t support everything!

QA prevents us from being agile

People on really old devices probably aren’t buyers

QA is an IT / Development Task


QA IS A MARKETING FUNCTION


QA STRATEGY

#2


INFUSE QA INTO YOUR OPTIMIZATION DNA


1. CULTURE

3. TECH PARADIGMS

2. REAL USER USE CASES

QA IS OPTIMIZATION


1. Understand your visitors


2. Make sure control is not broken


3. Generate great hypotheses & test them



4. Don’t pollute your test data

5. Do what the data tells you to do


BUILDING A QA CULTURE



#QAHAMMER


Build Destroy

HAMMER UP


BUY IN



20% RPVIE BUG

REAL USE CASES


90%


VISITORS DON’T USE

YOUR SITE THE WAY YOU

THINK THEY DO@joeljharvey #convcon


SCREEN RECORDING


USER TESTING

DOCUMENT AND QA ALL USE CASES


YOUR JOB IS TO BREAK YOUR

SITE


WHAT SHOULD YOU

SUPPORT?@joeljharvey #convcon

5%



Old New

BROWSER SUPPORT

Internet Explorer 8 Chrome 7 Firefox 4 Safari 4.1 iOS 5.1 *

Internet Explorer 11 Chrome 33 Firefox 27 Safari 7.0 iOS 7.1 *


OS SUPPORT

Old New

Windows XP OS X 10.4 (Tiger) Android 2.3 iOS 5.1 Windows Phone 7

Windows 8 OS X 10.9 (Maverick) Android 4.4 iOS 7.1 Windows Phone 8.1

5% MINIMUM


QA TOOLS

#4


BUILD YOUR QA’TION STATION


SEE YOUR SITES LIKE

YOUR VISITORS

iOS 7.1?



SOFTWAREvs.

HARDWARE


iPad, iOS 5

iPad, iOS 7.1

Windows Vista

Android 4.3

iPhones: iOS4, 5, 7 Android:

2.3.6, 4.0.4,

Windows 8, no

updates

OS X 10.6 Windows 8, regular updates

+ VM’s

OS X 10.8

OS X10.9

OS X 10.7 + VM’s

Windows XP

Windows 7


?


1. Browser

2. Operating System

3. “Other”

- Visitor Internet Speed

- Browser Plugins

- Flash

- Resolution

QA’TION STATION CHECKLIST

QA PROCESS

#4


BE THE HAMMER

QA EXPERIENCES,

NOT PAGES


TEST QA PROCESS


1.Build Your Test


2. Use Cookie, Query

String and/or Location Targeting so Only You

See Experience


3. Pause / Disable All

But One Experience


4. Review Live

Experience In All Supported

Browser/OS Combos (like REAL USERS)


5. Clear Cookies, Pause

Good-To-Go Experience and

Unpause the Next One


6. Repeat


A FEW TIPS

RANDOMIZE YOUR

APPROACH

CAUTION WITH CLASSES

WATCH SCRIPT STACKS

FEAR THE WYSIWYG

OWNERSHIP


Need to Borrow a QA Hammer?

Call 1-888-961-6604 and mention this session for a chance to win a Free QA Audit ($15,000 Value)

Conversion Conference San Francisco, 2014

Joel Harvey, Conversion Sciences

[email protected]

qa: how i learned to stop breaking and start loving tests

Business