responsive design tested: what a recent experiment reveals about the potential roi of mobile design

Responsive Design Tested What a recent experiment reveals about the potential ROI of mobile design

We’re sharing on Twitter!#WebClinic

Today’s speakers

Austin McCraw Senior Editorial Analyst

MECLABS

Jon Powell Senior Manager, Research

and Strategy MECLABS

What is responsive design?

ResponsiveUnresponsive

What is responsive design?

C = 4m + 3v + 2(i - f) - 2a©

The MECLABS Conversion Index

Friction: Psychological resistance to a given element in the conversion process, typically in the form of:

• Unnecessary length

• Unnecessary difficulty

What impact does responsive design have on friction experienced by the customer?

?

Background: A large news media organization trying to determine whether it should invest in responsive mobile design.

Goal: To significantly increase the number of free trial sign-ups.

Research Question: Which design will generate the highest rate of free trial sign-ups: responsive or unresponsive?

Test Design: A/B multifactor split test

Experiment ID: TP1933Record Location: MECLABS Research LibraryResearch Partner: [Protected]

Experiment: Background

Experiment: Control (unresponsive)Desktop Tablet

*Android devices also included in this test

Experiment: Control (unresponsive)Mobile

*Android devices also included in this test

Desktop

Experiment: Treatment (responsive)

*Android devices also included in this test

TabletDesktop

Experiment: Treatment (responsive)

*Android devices also included in this test

MobileDesktop

Experiment: Side-by-side

ResponsiveUnresponsive

*Android devices also included in this test

Experiment: Results

Design Conversion Rate

Control – Unresponsive 3.71%

Treatment – Responsive 5.80%

% Relative Change: 56.41%

Aggregate increase in conversion 56%The responsive design treatment showed an aggregate statistically significant difference in free trial subscriptions when compared to the control.

*Android devices are also part of these results

Why did the responsive design outperform the unresponsive by so much?

?

Experiment: Results

Design Conversion Rate

Control – Unresponsive 3.71%

Treatment – Responsive 5.80%

% Relative Change: 56.41%

Aggregate increase in conversion 56%The responsive design treatment showed an aggregate statistically significant difference in free trial subscriptions when compared to the control.

*Android devices are also part of these results

Experiment: Results

Aggregate Increase in Conversion 56%The responsive design treatment showed an aggregate statistically significant difference in free trial subscriptions when compared to the control

Treatment – Responsive 5.80%

Design Conversion Rate

Control – Unresponsive 3.71%

+ Desktop %

+ Tablet %

+ Mobile %

+ Desktop %

+ Tablet %

+ Mobile %

Which device makes the greatest contribution to the total difference??

Today, we are going to review test data for each device category to determine its total impact on perceived friction.

!

Desktop PerformanceResponsive design tested

Experiment: Desktop control

Unresponsive

• For most screen sizes, all primary content is accessible without action

• The primary content is framed within a faux pop-up style box over a faded background

Experiment: Desktop treatment

Responsive

• Like the control, all primary content is accessible without action

• The faux pop-up frame, however, is completely eliminated

Experiment: Desktop side-by-side

Unresponsive Responsive

Experiment: Desktop results

Design Conversion Rate

Desktop Control – Unresponsive 2.65%

Desktop Treatment – Responsive 5.43%

% Relative Change: 104.57%

Increase in conversion 105%The responsive design desktop treatment showed a statistically significant difference in free trial subscriptions when compared to the control.

What was actually responsible for the difference in performance? ?

Tablet PerformanceResponsive design tested

Experiment: Tablet controlUnresponsive

iOS: • The majority of the content is

accessible without action (supporting image requires scrolling)

• A portion of the bottom of the page shows the pop-up

Android: • All of the content is accessible

without action; may require zoom for certain devices

• Supporting imagery and aesthetic quality is lost in the page load

Experiment: Tablet treatment

iOS: • The supporting image is

immediately visible and the pop-up background is completely eliminated

Android: • Significant improvement in

aesthetic and page load, though the supporting image is still lost

• Only vertical scrolling required for ancillary content

Responsive

Experiment: Tablet side-by-side

Unresponsive Responsive

Experiment: Tablet results

Design Conversion Rate

Tablet Control – Unresponsive 7.24%

Tablet Treatment – Responsive 8.81%

% Relative Change: 21.6%

No significant difference in conversionThe responsive design tablet treatment showed no statistically significant difference in free trial subscriptions when compared to the control.X

Why did the same treatments perform differently on different devices??

Mobile PerformanceResponsive design tested

Experiment: Mobile control

iOS: • Requires scrolling to read complete

sentences

• Also requires scrolling to see all required fields and the primary call-to-action

Android: • Requires one to two pinches to zoom

movements and read the content

• When zoomed in, requires horizontal scrolling to read complete sentences

Unresponsive

Experiment: Mobile treatment

• Requires vertical scrolling to get to all fields as well as the call-to-action

• Complete sentences are available without action

• Generally a consistent experience across major device platforms

Responsive

Experiment: Mobile side-by-sideUnresponsive Responsive

Experiment: Mobile results

Design Conversion Rate

Mobile Control – Unresponsive 5.14%

Mobile Treatment – Responsive 4.34%

% Relative Change: -15.6%

No significant difference in conversionThe responsive design mobile treatment showed no statistically significant difference in free trial subscriptions when compared to the control.X

Experiment: All results

% Relative Change: 56.41%

Treatment – Responsive 5.80% +56.4%

Design Conversion Rate % Rel. Change

Control – Unresponsive 3.71% -

+ Desktop 2.65% -

+ Tablet 7.24% -

+ Mobile 5.14% -

+ Desktop 5.43% +104.6%

+ Tablet 8.81% +21.6%

+ Mobile 4.34% -15.6%

What is the customer wisdom? Why did the treatment win in aggregate?

?

What you need to understand

C = 4m + 3v + 2(i - f) - 2a©

The MECLABS Conversion Index

The mobile and tablet treatments made significant changes in layout; they did not, however, make significant changes in frictionexperienced by the customer

Friction

What you need to understand

C = 4m + 3v + 2(i - f) - 2a©

The MECLABS Conversion Index

The desktop treatment did something slightly different

The move to responsive design coincidentally reduced both friction and anxiety by eliminating the pop-up appearance of the control

Friction

Anxiety

Responsive design tested

FKey Principles

1. Responsive design, like any new technology or technique, does not necessarily increase conversion. This is because we are not optimizing design, we are optimizing thought sequences.

2. Responsive design is only as effective as the degree of change it affects in the mind of the customer.

3. This requires you to consider not only how responsive design will impact the look of the page, but the customers perception on two levels:• Friction: How much will responsive design decrease perceived effort?

• Anxiety: How much will responsive design decrease perceived concern?

EXCLUSIVE SAVINGS

Save an EXTRA $200 on Email Summit Tickets

Use promo code:

515-ST-1010*Offer expires January 24

Background: A large technology company selling software to small businesses.

Goal: To significantly increase the number of software purchases.

Research Question: Which design will generate the highest rate of software purchases?

Test Design: A/B multifactor split test

Experiment ID: TP1933Record Location: MECLABS Research LibraryResearch Partner: [Protected]

Experiment: Background

Experiment: Side-by-side

COLUMN #1 COLUMN #2COLUMN #1

Control Treatment

Which treatment won?

Live: Jan. 22 at 4:00 p.m. EST

• What is the optimal number of columns for a page?

• How should columns be weighted on a page?• Which is better: a right sidebar or left

sidebar?

Watch live or get the full 35-minute video replay

To see the results …

To watch live or view the full video replay when it is available, register at the link below:

MarketingExperiments.com/Columns

See how you can conduct research with usMECLABS conducts rigorous experiments in the new science of optimization. We apply our discoveries to help leaders optimize the financial performance of their sales and marketing programs.

Learn more about how you may be a fit for a MECLABS Research Partnership:

• Select Research Partnership Opportunities on the post-webinar survey

• Contact us directlyinfo@MECLABS.com1-877-635-0565

x