1
How to Benchmark Your Online Customer
Experience Against Competition
2
Conducting Remote Unmoderated Usability Studies: Part 1
Ann Rochanayon
Sr. Director UX/CX Research
UserZoom
Speaker
Meet the Speakers
Elijah Degen
VP, Head of Digital Marketing Practice
Interakt
Speaker
3
The #1 Platform for Agile Usability Testing & User Experience Analytics
Our all-in-one enterprise solution makes research & testing
cost-effective, agile and easy
4
• Chat box is available if you have any questions
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the slides/recording
Quick Housekeeping
Continue the discussion using #uzwebinar
Continue the discussion using: #uzwebinar
5
UserZoom Retail Benchmark | December 2014
• How CE/UX Benchmarks Add Value to Your Brand
• About Remote Unmoderated User Testing
• Case Study: Retail Competitive Benchmark Study
Study Methodology
Participant Profile
Summary
Brand, Conversion, and NPS Findings
Contents
6
How can CE/UX benchmarks add to value to your brand?
7
Perceived
CredibilityProfitability
Intent to
Return
Intent to
Purchase
User
Satisfaction
Word of
Mouth
A Positive
Experience
A Negative
Experience
Expectations User Experience
Online User experience – the defining momentThe content and quality of each individual visitor’s interaction — the user experience — collectively defines your digital brand, communicates your value proposition, and creates the foundation for future brand interactions — positive and negative.
8
• Traffic, user demographics, media spend, etc…
• Customer Experience & User Experience have varying
definitions between companies
• Usability benchmarks are generally custom
• Competitors vary depending on who you ask
• Info from UX benchmarking is so actionable, it is rarely
shared
Lots of macro benchmarks are out there…
Why are there few benchmarks for CE/UX?
9
Where can UX benchmarking impact decisions?
Area of Discovery Example value
Best Practices in service/UX design Leverage competitors findings and savecosts
Service/UX Design Decisions Justify investments/costs via VOCOr test prototypes
Know your customer Know what keeps them from converting
Quantify to customer feedback Give hard numbers to show impact.
Example Question:For those who considered your brand, what are the top 5 reasons why not converting?
Online Brand Impact Help brand marketers understand the brand's impact in online experience
Eg Brand Perception can impact conversion as much as online task completion.
10
Web design components
• Navigation, content, site performance, interactive
elements.
• Example Metrics: Task Completion, Ease of Use, Time
on Task, etc.
Brand components
• Voice of Customer, Brand affinity/likeability, Attitudes
toward Brand
• Example Metrics: NPS, CSAT, Brand Attributes Matrices
What can be measured?
11
What is needed to benchmark properly?
Area of Discovery Example value
Correct tools • cross device• website independent• In-app• Video & audio & clicks
Correct Audiences • Target Audience• Counter Audiences
eg Lost Opportunity Audience, Control Group
Correct Proctor / Partner • Impartial = Let the numbers speak for themselves
• Specialists w/ Broad Experience
Correct Type of Testing • Moderated In-Person (powerful exploratory / directional tool, but EXPENSIVE)
• Unmoderated Remote Testing(great deep dive analysis and very cost scalable globally)
12
UserZoom Retail Benchmark | December 2014
Remote Unmoderated User Testing
• ‘In-the-wild’ testing (natural context)
• Geographic representation
• No moderation needed
• Software collects data, stores securely in the cloud,
accessible real time
•Test any web-based UI, live or prototype
• Output: Usability + Behavior + Video & Audio +
Satisfaction+NPS+SurveyFeedback
• Fast data collection
• Cost efficient
13
UserZoom Retail Benchmark | December 2014
• Goals
• Determine which retail site presents the best tablet shopping experience
• Assess the impact of the tablet shopping experience on brand perception, Net
Promoter Score and conversion
• Sites evaluated
• Participants tested one of four sites in Nov - Dec 2014 on a tablet
• N=202
• Ages 18+
• Shopped for clothing and accessories online at least once per year
Methodology
14
UserZoom Retail Benchmark | December 2014
• Between-subjects design
• N=202 (~50 per site)
Methodology: Study Design
Screener
Introductory Questions
Macy’s1. Find a jacket
2. Return policy
3. Checkout
Nordstrom1. Find a jacket
2. Return policy
3. Checkout
Urban Outfitter1. Find a jacket
2. Return policy
3. Checkout
Lands’ End1. Find a jacket
2. Return policy
3. Checkout
Final Questions
15
UserZoom Retail Benchmark | December 2014
Participant Profile: Purchase experience
Please indicate your gender.
84%
16%
Female
Male
5%
42%
17%
14%
21%
18 - 25
26 - 35
36 - 45
46 - 55
Over55
Please indicate your age range.
15%
39%
38%
8%
Weekly
Monthly
Several times per year
Once per year
Which of the following best describes how frequently you
purchase clothes and accessories online?
N=202
16
UserZoom Retail Benchmark | December 2014
Participant Profile: Sites Browse / Purchased
• Kohl’s and Macy’s were the top two sites participants visited to purchase
clothing and accessoriesWhich of the following websites have you browsed for clothes and accessories using your tablet? / Which of the
following websites have you purchased clothes and accessories using your tablet? [Check all that apply]
20%
14%
14%
34%
13%
62%
21%
14%
52%
26%
10%
11%
6%
7%
18%
5%
38%
9%
6%
37%
10%
5%
Aeropostle
Banana Republic
Dillard’s
GAP
J. Crew
Kohl’s
L.L.Bean
Land’sEnd
Macy’s
Nordstrom
Urban OutfittersBrowsed Purchased N=202
17
Summary
18
UserZoom Retail Benchmark | December 2014
To assess the overall user experience ranking of the sites, UserZoom captured usability
metrics for three key tasks on each tablet site. Participants were asked to find an item
meeting specific criteria, find return policy information and asked to evaluate the checkout
process. The usability metrics captured include key performance indicators such as task
effectiveness, ease of use, and clarity of information. The rankings were derived according
to how each site performed on each measure.
User Experience Rankings
79
72
70
55
1
2
3
4
User Experience
Score*
*Score based out of 100 possible points
19
UserZoom Retail Benchmark | December 2014
Summary: Task Effectiveness
• Participants were most successful in completing the study tasks on the Lands’
End site
• Urban Outfitters had the lowest overall task success rates
96% 98%
86%88% 90%94%
79%
69%
85%
98% 98% 98%
Task 1: Find a Jacket Task 2: Return Policy Task 3: Checkout
Macy's N=50 Nordstrom N=49 Urban Outfitters N=52 Lands' End N=51
20
UserZoom Retail Benchmark | December 2014
Summary: Task Ease of Use
• Lands’ End performed relatively well overall on task ease of use
• Urban Outfitters had the low ease of use ratings, consistent with the lower task
success rates
72%
48%
68%65% 67%
78%
37% 38%
62%
69%
61%
86%
Task 1: Find a Jacket Task 2: Return Policy Task 3: Checkout
Macy's N=50 Nordstrom N=49 Urban Outfitters N=52 Lands' End N=51
21
UserZoom Retail Benchmark | December 2014
• Lands’ End led all sites with
the highest ratings in the
usability metrics
• Nearly all participants
succeeded in all three tasks,
more than any other site
• Lands’ End also had relatively
high ease of use ratings and
was rated highest on post-
study brand perception
• Participants found the Lands’
End site fast, simple, easy to
use and understand; Lands’
End also provided clear return
policy information
Summary: Lands’ End
• Although Lands’ End rated highest on task success and fewest issues encountered
for the Return Policy task, one area of improvement to consider would be cleaning up
the look and feel / organization of the Return policy page
22
UserZoom Retail Benchmark | December 2014
Summary: Lands’ End
• Study participants appreciated the fast, simple, easy to navigation with clear
information available on Lands’ End
Simple and easy to use
“Very easy to navigate”
“Overall a very easy pleasant experience”
“Very easy, big selection”
“A lot of ways to narrow your search. Easy
checkout process”
“Everything was so simple and easy to do.”
Clear Information, Easy to Understand
“Easy to navigate site. Information clear and easy
to understand. No issues.”
“The site was very easy to navigate, all necessary
information was given, pictures and descriptions
of merchandise was clear”
“Very easy to maneuver around, easy to
understand return policy”
Fast
“Easy and fast to find what I was looking for”
“Your website does not lag at all which made it an
enjoyable experience.”
23
UserZoom Retail Benchmark | December 2014
• Nordstrom edged out Macy’s for second place in the user experience rankings
• A vast majority of Nordstrom participants succeeded in all tasks with 88% or more succeeding
• Task ease of use were fair with 65% of the participants or higher finding the site easy to use
while performing the assigned tasks
• Participants overall found the Nordstrom site easy to navigate and several mentioned the
helpfulness of the filters
• However, a sizeable percentage of participants encountered technical issues on the Nordstrom
site with 27% reporting site slowness and 24% had problems with pages jumping around when
looking for a product
• Another opportunity for improvement for Nordstrom would be providing clear return shipping
cost information; 27% reported frustration not understanding the cost to return an item by mail
Summary: Nordstrom
24
UserZoom Retail Benchmark | December 2014
Summary: Nordstrom
• The ease of navigation and product filters were cited as being the most helpful
aspects of the Nordstrom site
Easy to Navigate
“It was easy to use their website and find the
information I needed to make a purchase”
“Very easy to navigate and checkout...also great
return policy with free returns”
“How relatively easy it was to make purchases
and to be able to ship those items by mail.”
“Easy to navigate. Inviting layout. Easy to add
items to shopping bag and to checkout.”
Easy-to-use Filters
“The website was set up for easy purchases. And
the filters made my shopping very easy and fast.
Very impressed!”
“The whole thing was very smooth and easy to
understand. The filters, the check out, customer
service- it’s all very accessible.”
“The site was pretty easy to navigate with lots of
filtering options. Most of the information I looked
for was easily found”
“Easy to use the filters to narrow a search.”
25
UserZoom Retail Benchmark | December 2014
• Macy’s finished in third place based on the user experience rankings
• Nearly all participants in succeeded in finding a jacket meeting specific criteria and
finding the return policy while 86% succeeded with the checkout task
• Despite the high success rates in finding the return policy, only 48% found the task easy
with 36% reporting difficulty understanding the shipping cost and 18% struggling to find
return policy information
• 68% found it easy to check out on the Macy’s site; technical issues with the site were
the primary problems encountered
• 16% mentioned the pages were too slow during checkout and 12% had problems with
the page jumping around when clicking
Summary: Macy’s
26
UserZoom Retail Benchmark | December 2014
Summary: Macy’s
• Participants found Macy’s easy to navigate, but also mentioned that the site
was fast and offered a good tablet experience
Easy to Navigate
“Fast. Easy to find things and search”
“Ease of the website, clear self explanatory instructions
on how to check out and add things to my bag”
“Site was very easy to navigate, simple links and the
filter/sort feature was very fast.”
Fast Site
“Many products, fast loading”
“The site was easy to navigate, loading time was fast and
almost everything was very easy to locate.”
“Very easy and clear to navigate. Loaded quickly and
responded in a good amount of time.”
Good Tablet Site
“It worked really well on my tablet. I do most of my online
shopping on a tablet and some sites work better then
other. I would say this one was easy to use its not a site
that I would have to switch to another computer to use.”
“I like the mobile version better than desktop”
27
UserZoom Retail Benchmark | December 2014
• Urban Outfitters lagged behind the
other sites in fourth place
• Participants were least successful
finding a jacket, finding return policy
information and checking out
• Ease of use ratings were also lower
than the other sites
• Only 37% found it easy to find a jacket
with 23% struggling with the amount of
scrolling, 17% struggling with the filters
and 17% having problems narrowing
down results among other issues
• With the return policy, 37% had
problems finding it and another 13% felt
there was too much scrolling required
Summary: Urban Outfitters
28
UserZoom Retail Benchmark | December 2014
Summary: Urban Outfitters
• Participants had greater difficulty navigating Urban Outfitters due to the
scrolling required along with technical problems with the site
Difficult to Navigate
“Everything so complicated to find. Too much scrolling to
find an item. Info text too small to read”
“Not easy to navigate thru to see items. Need more
groupings etc., men’s coats, women’s coats, shoes men’s or
women’s.”
“It is hard to maneuver to categories because the pictures
are so big for the categories.”
“Images were too big, I had to scroll around too much”
Unclear Icons
“Also need to identify the BAG icon.”
“Icons are not easily identifiable.”
Technical Issues
“The checkout kept asking for my shipping address and
then would reload the same page. I did this 4 times.”
“The page did not fully load”
“The page would not bring up the items; the page kept going
blank”
29
Brand, Conversion, and NPS
Findings
30
UserZoom Retail Benchmark | December 2014
• After interacting with the site, Lands’ End had the biggest increase in positive
brand perception
Brand Perception
62%
57%
23%
18%
62%
69%
44%
76%
Macy's Nordstroms Urban Outfitters Lands' End
Pre-site evaluation Post-site evaluation
N=37 N=50 N=37 N=49 N=31 N=52 N=39 N=51
Site Delta
Macy’s 0%
Nordstrom 14%
Urban Outfitters 21%
Lands’ End 58%
* Selected 6-7 on a 7-point scale where 1 = Negative and 7 = Positive
31
UserZoom Retail Benchmark | December 2014
Call to Action
• Participants were most likely to make a purchase from Macy’s or Nordstrom
and least likely to to make a purchase on the Urban Outfitters tablet site
52%47%
15%
39%
38%
35%
43%
57%
10%18%
42%
4%
Macy's N=50 Nordstrom N=49 Urban OutfittersN=52
Lands End N=51
Unlikely (1-2)
Neutral (3-5)
Likely (6-7)
Based only on your experience today, how likely are you to
make a purchase on [SITE] on your tablet in the future?
32
UserZoom Retail Benchmark | December 2014
Net Promoter Score
• Participants were most likely to recommend Macy’s, followed by Nordstrom
• Urban Outfitters had far more detractors than promoters with a -56% NPS
10%
-6%
-56%
- 23%
NPS
36%
43%
69%
45%
18%
20%
17%
33%
46%
37%
14%
22%
Macy's N=50
Nordstrom N=49
Urban Outfitters N=52
Lands End N=51
Detractor (0-6) Passive (7-8) Promoter (9-10)
How likely are you to recommend [SITE] to a friend or
colleague?
33
UserZoom Retail Benchmark | December 2014
Post Site Interaction Brand Attributes*
• Nordstrom was associated with ‘Customer-oriented’ and ‘High quality,’ while
Macy’s was most associated with ‘Value’ and ‘Variety’
42%
10%
54%
40%
52%
51%
24%
63%
18%
41%
19%
23%
40%
12%
35%
37%
8%
51%
31%
33%
Customer-oriented
Exclusive
High quality
Value
Variety
Macy's Post N=50
Nordstrom Post N=49
Urban Outfitters Post N=52
Lands End Post N=51
Based only on your experience today, which of the following
attributes do you associate with [COMPANY]?
* Partial list displayed
34
UserZoom Retail Benchmark | December 2014
Change in Brand Attributes*
• Perceptions of ‘Customer-oriented’ and ‘Variety’ increased for all sites from
pre- to post-study
• Macy’s and Nordstrom had decreases in brand association with ‘Exclusive’
and ‘High quality’
10%
-10%
-26%
6%
4%
27%
-15%
-17%
0%
19%
6%
11%
13%
-3%
23%
21%
0%
12%
15%
25%
Macy's N=50
Nordstrom N=49
Urban Outfitters N=50
Lands End N=51
Value
Customer-oriented
Exclusive
High quality
Variety
Which of the following attributes do you associate with
[COMPANY]?
* Partial list displayed
35
Usability Testing
• Web & Mobile Task-based Testing
• UX & Usability Benchmarking
• Competitive Research
• Concept Testing
IA & UX Design Validation
• Navigation Optimization - Card Sorting
• Findability - Tree Testing
• Design Research - Click Testing
• Impression Testing - Timeout Testing
Customer Experience Measurement
• Voice of Customer- Live Intercept surveys
• Customer Feedback & Sat- Online Surveys
• Web Analytics Integration
Suite of tools For
All stages of your product
life cycle
Introduction to UserZoom
36
UserZoom Data | Results
37
• Benchmarking is valuable to investment decisions
• Benchmarking can give you unparalleled insight and a
competitive edge
• Nearly every customer perspective can be measured
• Partner, Tool & Audience selection is key to success
• Retail is audiences are very measurable for online user
experience & offline in-store experience
• Task Metrics are key to conversion improvement
• Brand Metrics are very measurable via Remote Testing
• Lift in Brand Perception by a website can be HUGE
• Know if your website is presenting the desired Brand
Attributes
Summary
38
Questions