usability testing repor t - sjaiswal.bcitwebdeveloper.ca · usability test for the redesign of the...

Usability Testing Report

Jason Enstedt

Sonam Jaiswal

Nancy Kwong

Kyungmi Moon

Executive Summary The purpose of conducting the usability test was to determine and evaluate the current state of the Best Buy (Canada) website ( www.bestbuy.ca ). This report outlines the test plan of the usability test for the redesign of the Best Buy (Canada) website. The goals of the usability test were to determine the baseline user performance, validate user performance, and to identify potential design concerns in an effort to improve the end-user satisfaction thereby driving traffic to the website through productivity and efficiency enhancements. The usability test was conducted in controlled test conditions with representative users. Prior to the usability test, the participants completed a pre-test questionnaire to gain a better understanding of their demographics. The usability test via the UsabilityHub included a 5-Second test, a First Click test, and a preference test. Following these tests, four specific tasks were completed within the website, which were followed by post-task questions probing user experiences while interacting with the website. The testing results indicated that the Best Buy (Canada) website ( www.bestbuy.ca ) design was visually appealing, lent credibility, and fit the theme of the company but could use improvements in navigation. Although the site as deemed user-friendly, the search options was found to be the most difficult to use. Many found it challenging to find correct fields to narrow down searches and were disappointed in the search feature’s inability to handle multi-keyword searches. It is recommended to improve search performance with a range slider for price and size thereby enabling more intuitive product searches. Also recommended is to enable the global search to search for both products and company information such as company services and policies. These recommendations would enhance user interaction with the website and improve end-user satisfaction by reducing user frustrations with finding Best Buy products and services.

Introduction Best Buy is one of Canada’s largest retailers operating as Best Buy ( www.bestbuy.ca ), Best Buy Mobile and Geek Squad ( www.geeksquad.ca ). Best Buy sells electronics, computers, appliances, cell phones, video games, new technology. It offers in-store pick-up and 2-day shipping on thousands of items. This document describes a test plan for conducting a usability test for the Best Buy (Canada) website ( www.bestbuy.ca ). The goals of usability testing include establishing a baseline of user performance, establishing and validating user performance measures, and identifying

3

http://www.bestbuy.ca/



http://www.geeksquad.ca/


potential design concerns to be addressed in order to improve the efficiency, productivity, and end-user satisfaction. The usability test objectives are:

● To determine design inconsistencies and usability problem areas within the user interface and content areas. Potential sources of error may include:

○ Navigation errors – failure to locate functions, excessive keystrokes to complete a function, failure to follow recommended screen flow.

○ Presentation errors – failure to locate and properly act upon desired information in screens, selection errors due to labelling ambiguities.

○ Control usage problems – improper toolbar or entry field usage. ● Exercise the application or web site under controlled test conditions with

representative users. Data will be used to assess whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.

● Establish baseline user performance and user-satisfaction levels of the user interface for future usability evaluations.

User Types The typical audience for Best Buy includes:

● Personal computer users would be looking for desktops, laptops and tablets with recommended configurations in store.

● Video gamers would be looking for the latest hardware and accessories in-store to

upgrade their desktop

● Computer geeks would be looking for the latest hardware in-store to upgrade their systems.

● Household users would be focusing on household items such as microwaves, washing machines, blenders and other electronic items.

● Kids (ages 5 to 17) would be emphasizing more on games such as PS4 and Xbox.

4

● Music lovers would be looking for headphones, soundbars and stereos.

● Entertainment lovers would be looking for high-resolution televisions such as QLED, OLED and 4K.

● Apple product users would be looking at different kinds of laptops, iPads, iPods and iPhones.

● Some users would be looking for Geek squad to repair their computers.

● Some users would wait for Black Friday and Boxing Day sales in-store to save money on large ticket items ($1K +).

● Users will be also checking out Best Buy website from time to time for online shopping .

5

User Persona

6

Methodology | Participants There were five volunteer participants taking part in the usability test conducted by a moderator, two observers, and a technical representative. The characteristics and backgrounds of the participants fell under the Best Buy website’s potential ideal user types. All the participants were visiting the website for the first time. They are expected to know how to use a computer for browsing, live in the Lower Mainland and have made online electronic purchases within the past three years.

| Setting The participants took part in the usability test at the BCIT downtown campus in Vancouver, BC. A Lenovo laptop with the Best Buy (Canada) website ( www.bestbuy.ca ) open in a browser window was used in a typical office environment. Each participant’s interaction with the Best Buy (Canada) website was conducted by the Moderator who was beside the participant. The two Note Takers/Observers were seated behind the participant monitoring the sessions by taking notes. The Key Technical representative was on the other side of the participant setting up the browser interface.

| Process 1. The participant was first greeted by the moderator and introduced to each of the roles

in conducting the usability test. 2. The moderator then briefed the participants on Best Buy (Canada) website

( www.bestbuy.ca ) and instructed the participants that they were evaluating the website and not their ability to answer the questions.

3. Each participant was informed that the test would be composed of four test components:

a. Pre-test questionnaire, b. Three Usability Hub tests, c. Four Task Scenarios, and d. Post-test questionnaire.

4. The participants answered the pre-test screening questionnaire on their own with the first browser window.

5. The participants were then directed to do the second portion of the test which is the Usability Hub’s 5-Second test, First-Click test, and a Preference test.

7



6. The next component of the test was the Task scenario where the “Think Aloud” approach was introduced. Participants were encouraged to speak through their thought processes, expectations, or surprises they encountered while going through each task scenario. Each of the tasks was measured within a 4-minute time window resulting in either a successful or failed attempt in achieving the task at hand. There were a total of four task scenarios.

7. The last component of the test was the post-test questionnaire, which the participant completed on their own in the last browser window. The post questionnaire differs from the pre-test questionnaire in that the questions are subjective towards their experience using the website.

Pre-Test Questionnaire The purpose of the pre-test questionnaire was to get a basic understanding of the demographics represented by the participants. Both the Pre-Test Questionnaire and Post-Test Questionnaire were created in Google Forms with the information stored in a Google Sheet with the Advanced Summary by Awesome Table Google Form Add-on.

| Basic Tester Information | Findings

| Demographics

The Pre-Test Questionnaire confirmed that the participants were the expected demographic for the users of the BestBuy site being: tech-savvy 18-35 years old students who spend a short amount of time online and own about three electronic devices. (See Appendix: Pre-Test Questionnaire Results — Demographics ) | Online Shopping Experience The participants relied primarily on online store reviews and recommendations from friends and family to decide on purchases. The majority (60%) of the participants make online purchases once or twice a month and make purchases as frequently in-store as they do online. Lack of ease finding products or personnel for assistance are the primary challenges faced by the participants. Conversely, with online shopping, the lack of instant product delivery was the primary drawback cited by the participants followed by the inability to touch the actual

product. (See Appendix: Pre-Test Questionnaire Results — Online Shopping Experience )

8

Usability Hub Tests The three usability tests were created on usabilityhub.com to capture the participants selections and feedback.

| 5-Second Test | Background

Many people make judgements about a website within the first five seconds of visiting it. A 5-Second test can help determine with the website makes a good impression on users, and whether key navigation features or call to action (CTA) buttons stand out. One limitation of the 5-Second test is that the interactive elements of the website cannot be tested.

| Test Administered

For the BestBuy website, the 5-Second test was to determine if the current layout with the CTAs were obvious enough for first-time viewers to readily recall. The screenshot of the BestBuy website had multiple banners advertising a phone, a promotion, and a brand (See Appendix: Usability Hub Tests — 5-Second Test ). | Results

The phone of the day was identified by half of the participants. None of the participants could completely identify the promotion. And only a third of the participants could identify the brand of the day (See Appendix: Usability Hub Tests — 5-Second Test ). | Analysis and Recommendations

The use of multiple banners does not enable website visitors to quickly determine which CTA that BestBuy is highlighting as the most important. The smaller banner on top of the larger banner appears to detract from the clarity of the CTA message. Also the font of the description for the promotion may be too small for website visitors to read and absorb the information. Perhaps the promotion description could be shortened and moved closer to the promotion header.

9

https://usabilityhub.com/

| First Click Test | Background

Successful websites take website visitors’ tasks into account upfront. A First Click Test examines what a website visitor would click on first on the interface in order to complete their intended task — it allows the evaluation of the linking structure effectiveness of the website, including the navigation. | Test Administered

For the BestBuy website, the First Click test was to determine if the current layout and navigation were intuitive enough to find out where the flyer information could be found (See Appendix: Usability Hub Tests — First Click Test ). | Results

The majority of the participants clicked on the Deals to find flyers. A few clicked on Services to find flyers. Another clicked on the Stores with the assumption that the flyers would be linked to store location, similarly like Superstore and Shoppers Drug Mart (See Appendix: Usability Hub Tests — First Click Test ). | Analysis and Recommendations

Due to the lack of industry-standard, there was a lack of clarity in locating the flyers. Perhaps there should be a separate link to flyers.

| Preference Test | Background

Preference tests can help choose between design variations, by simply asking website visitors their preference. A Preference test shows a number of design options and a favourite is selected. These tests are commonly used to measure aesthetic appeal, but participants can also be instructed to judge designs based on their trustworthiness, or how well they communicate a specific message or idea. Preference testing produces both quantitative and qualitative feedback, which makes it a very time-effective evaluation tool. | Test Administered

For the BestBuy website, the Preference test was to determine which layout (gradient banner, more white space, or more colours) was more appealing (See Appendix: Usability Hub Tests — Preference Test Results) .

10

| Results

The majority of the participants selected the option with more white space due to the clarity of the banner (See Appendix: Usability Hub Tests — Preference Test Results) . | Analysis and Recommendations

Future consideration of gradients and colour needs to be carefully contemplated prior to its use.

Task Scenarios | Task 1 Scenario You are a freelancer programmer who is looking for a Lenovo Laptop. Find the product which meets every requirement below.

Brand Lenovo CPU i5 processor

Size 15.6” Display RAM 8GB

Price Under $800 Drive SSD

OS Windows 10

| Task 2 Scenario Imagine you bought a vacuum 3 weeks ago, you have opened the box and you notice that it is not the right model. Check the Bestbuy website and check if you can still return it.

| Task 3 Scenario Imagine you bought a product from Bestbuy and now you are having technical problems with it. Start an online chat with a Best Buy Geek Squad agent.

| Task 4 Scenario Find the trade-in value for an iPhone XR 64GB in good condition, working battery, unlocked, serial number visible, no water damage, rogers carrier and find my phone disabled.

11

Metrics & Reporting Results

12

Overall Observations Participants who ordered electronics online from other online retailers (such as Amazon, The Source, London Drugs, etc.) were more familiar with the industry website layout of information. They were more likely to search for non-purchase topics in the footer.

Task 1 Scenario Find a specific laptop

Solution 1

Average Duration 1:40

Completion Rate 100%

Observations Participants were often stymied with the lack of expected search options available. A few participants wanted the ability to modify filters without having the page reload and the filter option staying open.

1 Path to the solution is outlined in the Appendix: Task Scenarios — Task 1 Scenario: Path to Solution

13

Task 2 Scenario Return Policy for opened item

Solution 2



Observations All participants found the return policy easily

Task 3 Scenario Geek Squad Online chat

Solution 3



Observations Two of the five participants found the BestBuy Chat and required a hint to find the Geek Squad Chat

2 Path to the solution is outlined in the Appendix: Task Scenarios — Task 2 Scenario: Path to Solution 3 Path to the solution is outlined in the Appendix: Task Scenarios — Task 3 Scenario: Path to Solution

14

Task 4 Scenario Trade-In Value for specific cellphone

Solution 4



Observation All participants found the trade-in program. One participant required a hint to look in the footer area.

Post-Test Questionnaire The purpose of the post-test questionnaire was to get an in depth understanding of the participants feelings and general thoughts of the website.

4 Path to the solution is outlined in the Appendix: Task Scenarios — Task 4 Scenario: Path to Solution

15

| Tester Thoughts | Findings 5

| Design

The Post-test questionnaire asked several questions about the users thoughts on the design of various aspects of the website. The general consensus was that the design was pleasing to the eye, fit the theme of the company, was credible and evoked the user to want to explore the site further. | Usability The Post-test questionnaire also asked questions about the users feelings toward the usability of the website. Although the general consensus was that the site was user-friendly, and easy to navigate, some thought it was difficult or confusing to go from page to page. | Tasks

The questionnaire asked the user their feelings toward the tasks they were asked to perform. They gave an average score of 7/10 (1 being hard, 10 being easy) for how easy it was to complete the task. With one person giving a 5/10, and another gave a 10/10. Each person found a different task the hardest to complete. | Critiques

One large area that almost all users found difficult to use was the search. Many found it hard to find the correct fields to narrow down their searches. Or were putting many keywords in the search and hoping it would automatically narrow the search.

5 See Appendix: Post-Test Questionnaire Results

16

Heuristics Evaluation

| Findings A Heuristic Analysis is a way to test a website to see if it is easy to use and is user-friendly. Unlike user testing, which is performed by everyday users, a heuristic analysis is done by usability experts. The site is put through a series of tests and given a score based on the results. The Best Buy website overall did very well. In each of the 9 categories it was tested in, it received at least a 90%. Its lowest score was in regards to the search feature of the website. It where it received 90%. This was because of the site’s subpar search engine. The search bar

17

only searches through the products and not the website itself. This results in a frustrated user. It also does not handle multi-keyword searches well.

Recommendations | Recommendation 1 Through task scenarios, several participants had difficulty in facet search. The current Best Buy website does not include price facets, so all of the participants had to visually search to find a product that met the price requirement. Therefore, it is recommended to improve search performance a range slider for price and size should be added so that customers are able to search for products more intuitively.

18

Current Facet Navigation Recommended Change

| Recommendation 2 As shown by the test result, although all of the participants completed the task 4 scenario, 80% of participants’ first action was to type ‘trade in’ or ‘trade-in’ directly into global search input. However, the search result showed the list of products includes ‘trade’ or ‘in’ which made participants confused and frustrated. Due to the frustration with the search, it took participants on average over two minutes to complete the task. The expectation was the search would find company services and policies , but the actual search was for products only. This is why several participants gave negative feedback about the global search. Based on this feedback, it is recommended to develop the global search input to cover more information to include company services and policies.

19

Current Search Input

Recommended Change

20

Appendix

| Test Plan

Participants 5 participants fitting the described persona

Date of test Wednesday, October 30th, 2019

Location of Test Room 300 BCIT Downtown Campus

Session Duration 100 minutes

Study goals From the current state of the website, are participants able to successfully find:

● a specific product ● information about the return policy. ● where an online chat with Geek Squad agent could be initiated ● the product trade-in section

Participant scheduling

● In-house recruitment ● Using this persona description.

Environment setup

● Sessions will take place in our classroom. ● The room will be set-up with a 15” laptop, with resolution set to 1920 x 1080. ● Chrome will be the default browser ● www.bestbuy.ca will be opened in the browser ● A screen-recording tool will be used to record the participants actions. 6

Tasks to perform

● Participants will fill out a Google Survey with pre-test question. They will then be asked to perform the following tasks:

○ Find a laptop which meets every requirement: Lenovo, i5 processor, 15.6” Display, 8GB RAM, SSD, Window 10, Under $800.

6 Nimbus Screenshot & Screen Video Recorder offered by: nimbusweb.me for Chrome

22

http://nimbusweb.me/

○ Find out if a vacuum can be returned which is purchased 3 weeks ago, opened box

○ Start an online chat with a Best Buy Geek Squad agent. ○ Find the trade in value for an iPhone XR 64GB in good condition, working battery,

unlocked, serial number visible, no water damage, rogers carrier and find my phone disabled.

Protocols

● We will ask post-interview questions. ● During the study we will use the think aloud protocol. ● We will have minimal interaction with the participant until the post-session interview

questions.

| Moderator’s Intro Script Hi, my name is Moon , and I’ll be running this session with you today. Sonam and Nancy are my colleagues who’ll be watching what we do today, and here, Jason will take care of technical stuff during the whole session. Thank you for taking the time to participate in this study. The test will take about 15 to 20 minutes. Before we begin, I’m going to give you a brief overview of the test and how it will work. This session is pretty straightforward — First, you’re gonna answer a few basic questions about you. And then, you’ll get 3 usability hub tests, and 4 specific tasks. Finally, you’re gonna get post-test questions about your experience. Remember, we are not testing you or your ability. We are testing the website. Also, I’d like you to “think aloud” as much as possible. Speak your thoughts as often as you can. For example, you may be looking at a page, suddenly see something you didn’t see before and want to click on it. In that case, saying something like “This caught my eye so I’m going to see what it is. I’m scrolling down.” would be very useful! If at any point you have questions, please don’t hesitate to ask. Do you have any questions so far? Ok, Let’s get started.

23

| Pre-Test Questionnaire Results | Demographics

24

| Online Shopping Experience

26

| Usability Hub Tests | 5-Second Test

A 5-second test is run by showing an image to a participant for just five seconds, after which the participant answers questions based on their memory and impression of the design. Q. Look at the interface for 5 seconds and remember as much as you can.

28

Q1. What is the Big 4 Event? Q2. What phone was being advertised?

A1. 4 different brand sale for 4 days A2. Google Pixel 4

Answers from participants

Q3. What was the brand of the day?

A3. LG

Answers from participants

29

| First-Click Test

The aim of first click testing is to verify that the first click a user makes on an interface to carry out a given task is clear and easy. Q. Where would you click to find the weekly flyer? Click on the image to carry out the instructions or complete the set task

First click test results

| Preference Test

A participant is shown a number of design options and is asked to pick a favorite.

30

These tests are commonly used to measure aesthetic appeal, but participants can also be instructed to judge designs based on their trustworthiness, or how well they communicate a specific message or idea.

Enlarged Images Option 1

31

Option 2

Option 3

32

Results

Reasons for selection From the Talk-Aloud observations, Option 2 with the most votes was selected due to the clean layout and use of white space (obvious banner).

33

| Task Scenarios | Task 1 Scenario: Path to Solution

Under the Shop menu, select “Computers, Tablets & Accessories”, then select “Laptops” :

On the Laptops & MacBooks page, select “Windows Laptops”:

34

On the Laptops page, scroll down to the “Shop by screen size” section, select 15 inches:

On the 15 inches Laptops, use the filter options: Brands, Price, Processor Type, RAM, and Hard Drive Type:

35

| Task 2 Scenario: Path to Solution

Go to footer, under Customer Support, select “Returns & Exchanges”:

36

On the Returns and Exchanges page, select “Best Buy Products”:

On the Best Buy Products page, select, “Small Appliances, Personal Care, and Grooming Appliances” link:

37

On the Small Appliances Return & Exchange Policy page, scroll down to the “Cleaning and Home Comfort” section:

38


Go to footer, under Services, select “Geek Squad”:

On the Geek Squad Services page, scroll down to the “Getting Help Is Easy”, under the “Chat with an Agent Now”, click the “Chat with an Agent Now” link:

40


Go to footer, under Services, select “Trade-In Program”:

On the Trade-In Program page, scroll down to the “Estimate your trade-in value” section, select iPhone:

On the Apple Trade-In page, scroll down to the “Estimated trade-in values” section, and select, “Get estimate” under the iPhone image :

41

On the Cell Phone Estimator form, select from the drop-down menus:

Solution

42

| Post-Test Questionnaire Results

43

usability testing repor t - sjaiswal.bcitwebdeveloper.ca · usability test for the redesign of the...

Documents