04/07/2304/07/23 11

Project nameClient

Month Year

How to run usability testing with eye trackingwww.objectivedigital.com

Usability Testing in SydneyEye tracking eCommerce sites Optimising online marketing

04/07/2304/07/23 22

“Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” ISO9241−11

• Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?

• Efficiency: Once users have learned the design, how quickly can they perform tasks?

• Memorability: When users return to the design after a period of not using it, how easily can they re-establish proficiency?

• Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?

• Satisfaction: How pleasant is it to use the design?

Some definitions Usability

Usability Professionals Association http://www.upassoc.org/usability_resources/about_usability/

04/07/2304/07/23 33

Usability testing: an overview

• One on one testing with representative users.

• Users complete a set of tasks on a website.

• A facilitator uses a testing script, and observes the users’ behaviour as they complete the task.

• The focus is on usability.

• Users are interviewed after each task or at the end of the test.

• A combination of retrospective think aloud (RTA) or think aloud (TA) can be used for the tasks.

• A report with observations and usability issues which are translated into actionable recommendations.

04/07/2304/07/23 44

Why conduct usability tests?

• For a website to be successful, it must be usable. Target users must be able to accomplish tasks with little to no difficulty.

• You will discover how real users interact with your website and uncover problems through observation.

• You will almost always learn something about your website which the project team couldn’t foresee.

• It’s fast and relatively cheap.• You can solve disputes.• Saves time and money in redevelopment.• The development team and the client are too close to

the product!• It will improve the bottom line!

04/07/2304/07/23 55

When to test?

• Test early and test often!!!! • The earlier you start with testing the easier and less expensive it

is to change the system. • The more you test, the more confident you can be that the end

product will actually meet the users’ needs and the experience is optimised.

• Any testing is better than no testing.• You can test at any stage of the project.• Iterative testing substantially improves the usability of websites.• One study found iterative testing resulted in 30% more task

completions, 25% less time to complete the tasks, and 67% greater user satisfaction.

– US Department of Health and Human Services Research-Based Web Design & Usability Guidelines http://www.usability.gov/pdfs/guidelines.html

04/07/2304/07/23 66

Test iteratively

Test labelling, visual design, some tasks

Test some tasks

Test labelling, visual design, first click

Test all tasks

Test all tasks

04/07/2304/07/23 77

How many to test?

• For usability testing, 5-6 or 6-8 people per user group is enough to identify key problems with the site.

Think do they have the knowledge, skills and abilities to complete the task? Could they use the same script?

• It depends on the test objectives and the complexity of the site.

– A more complex site, e.g. a finance site, may have more user groups, e.g. Investors, Advisors, Journalists or Experts and Novices

• Our view: We have often uncovered all of the major issues by lunch time!

04/07/2304/07/23 88

Where to test?

• A two -s highly recommended to have clients and other stakeholders viewing the usability testing.

• However, you do not need a formal usability lab to do testing.

• Alternatively, you can do effective usability testing in any of these settings:

– a fixed laboratory having two or three connected rooms outfitted with audio-visual equipment

– a conference room, or the user's home or work space, with portable recording equipment

– a conference room, or the user's home or work space, with no recording equipment, as long as someone is observing the user and taking notes

– remotely, with the user in a different location– remotely, with observer in a different location

04/07/2304/07/23 99

Outcomes of usability testing

Usability testing provides a deep understanding of usability issues, and their root causes. Outcomes are reported as:•Prioritised usability issues incl. number of users experiencing the problem.•Actionable recommendations for correcting the issues.•Issues illustrated with screenshots, videos clips and user quotations.•What users did on the website, where they went, paths they took.•Success rates of tasks completed.•Users’ self report feedback on their experience when using the site, completing tasks.

04/07/2304/07/23 1010

What usability testing is NOT?

• Focus groups are NOT usability testing. Focus groups are about opinions, wants and needs.

• A research study is NOT usability testing. The sample size is typically much smaller.

• A quality assurance test is NOT usability testing. Although the test might uncover bugs on the website, this is not the primary purpose of the test.

04/07/2304/07/23 1111

Usability Principles

04/07/2304/07/23 1212

Usability principles in practice

• A set of usability principles, often described as heuristics, (rules of thumb), are used to evaluate digital products.

• These heuristics have been widely adopted and adapted by the software industry:

1. Visibility of system status 2. Match between system and the real world 3. User control and freedom 4. Consistency and standards 5. Error prevention 6. Recognition rather than recall7. Flexibility and efficiency of use 8. Aesthetic and minimalist design9. Help users recognize, diagnose, and recover from errors 10. Help and documentation

References:• Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces, Proc.

ACM CHI'90 Conf. (Seattle, WA, 1-5 April), 249-256• Nielsen, J http://www.useit.com/papers/heuristic/heuristic_list.html

04/07/2304/07/23 1313

Test Plan + Process

04/07/2304/07/23 1414

Test Plan

The test plan is developed with the client, based on:• Website goals• What the client wants to test• Technology issues• Key users and tasks ( and their availability)

04/07/2304/07/23 1515

Develop the test plan

04/07/2304/07/23 1616

Scripts and Scenarios

04/07/2304/07/23 1717

Typical test flow ---- change graphic, + how many + time

04/07/2304/07/23 1818

Writing scenarios

• Relate each task to test objective.

• Select scenarios that rate high on frequency and/or importance.

• Don’t make them too complicated. Only short instructions can be retained in short term memory.

• Move from general to more specific.– Begin with an exploratory task.

• Avoid using the terms on the website – it leads people.– You want to take out your super NOT You want to consolidate your super

• Make the tasks concrete, use verbs to describe them– Book a hotel in Rome; Find out if your employer paid your super contribution?

04/07/2304/07/23 1919

Writing scenarios

• Try to make a logical flow from the start to finish– e.g. follow a natural customer journey from search, create account, purchase, check order

• For information tasks, instruct the participant to tell the facilitator when they have found the answer.

– Find a cruise which interests you. Tell the facilitator when you have found it.

• Utilise other communications. – E.g. You saw a TV ad for XYZ company. Find out more about their product.

04/07/2304/07/23 2020

Exploratory scenarios

eCommerce• Christmas is coming up. You want to find some Christmas presents for your

children/nieces/nephews.

Insurance website• You have heard about XYZ on the radio. Browse around the website and find

something that interests you. Tell the facilitator when you have found it.

04/07/2304/07/23 2121

Specific scenarios

Asthma not-for-profit• Your (friend’s) child has been recently diagnosed with Asthma and you want to

learn about it. Find information on what to do if the child is having an asthma attack.

eCommerce• You want to do some more Christmas shopping. Buy a Rose print dress for your

sister and a Thomas and Friends Boys Tricycle for your nephew. Have them delivered to your sister’s address.

04/07/2304/07/23 2222

Conducting the test

04/07/2304/07/23 2323

When facilitating a test…

• Practice the test!

• Ensure the participant is comfortable with the technology you are asking them to use

• Ask the participant if have any questions before they begin• Explain the flow of the interview to the participant• Get a feel for their experience and level of comfort with the internet• Ask the participant to read out the task to ensure that they are attending to the

task details and understand what is being asked of them• Depending on whether is it a RTA or TA task, your involvement (and how much

you talk and when you talk) in the task will vary • Don’t explain the interface the participant is working with to avoid getting biased

feedback• Keep the participant focused on the task

04/07/2304/07/23 2424

When facilitating a test…

• Leave the participant to make mistakes without helping them out but investigate any issues/mistakes later in the session

• Answer questions with questions to elicit more indepth feedback• Probe expectations and use less direct questioning (if possible avoid ‘why’), to

gather less defensive responses – ‘Is this what normally happens when you go on an X website?’– ‘You seemed surprised/bored/puzzled by that feature….’ – ‘Can you please explain what is happening now’ – ‘You seem to be moving the mouse to the right all the time….’– ‘I notice you don’t scroll much…, I notice you keep going up and down the page, what are you

looking for?’• It would be ideal if you are able to have an observer take notes

04/07/2304/07/23 2525

When facilitating a test…

• Think Aloud – Think aloud A is widely used in usability research. During a task, the user is verbalising

their actions and thoughts out loud whilst navigating the site. – The moderators role is then to probe feedback during the test – This technique gives immediate feedback during a task and means the participant is not

relying on memory to provide feedback on things they experienced.

• Retrospective Think Aloud– With eyetracking hardware and software participants are asked to complete a task

silently and on their own. Once the task is complete, the moderator plays back the dynamic eye tracking video while asking the user about their underlying motivation. This allows the participant to get on with their task without being distracted during the task. The feedback is often more thought through when replaying the eye tracking video.

04/07/2304/07/23 2626

Usability Testing Mistakes

• Strategic errors – time it right– Don’t test too early – you just uncover bugs; Don’t test too late – nothing can change

• Inadequate planning – make sure you have a run through– Check your test materials against the thing you are testing, especially if it’s a prototype

• Poor task design– Ensure your scenarios test core functionality and any areas you have identified as

potentially problematic.• Giving too many clues

– Don’t give away the answer in the scenario e.g. Register for the site• Unprofessional demeanor

– Words like Good and Well done may give the impression that you are testing the user, not the site

– Reference: Information and Design Usability Test Mistakes http://www.infodesign.com.au/usabilityresources/usabilitytestingmistakes

04/07/2304/07/23 2727

Usability Testing Mistakes

• Not knowing why you are testing• Testing for too much for too long• Not bringing the team together• Not recruiting the right participants• Not designing the right tasks• Not facilitating the test effectively• Not planning how you will distribute the results• Not iterating to test potential solutions

04/07/2304/07/23 2828

Analysis and Reporting

04/07/2304/07/23 2929

What to report?

Outcomes are reported as:

•a prioritised list of usability issues •recommendations for correcting the issues•Each issue supported by data – what you observed •general feedback on the users’ behaviour when interacting on the website

04/07/2304/07/23 3030

Consider for reporting

• Which tasks which were difficult to complete? • Which tasks took a long time to complete?• What mistakes were consistently made? • How did they complete the tasks? Was it as expected?• How easily did users find the key functions? Register, Login, Add to cart, Checkout

• Where did users first click on the home page?• What interested the users most?• How did the behaviour of the the user groups differ? • What did they do which was unexpected?• What did users notice?• What did users miss?

04/07/2304/07/23 3131

How is usability measured?

• The most common factors measured in usability testing include:– Effectiveness: A user's ability to successfully use a Web site to find information

and accomplish tasks.– Efficiency: A user's ability to quickly accomplish tasks with ease and without

frustration.– Satisfaction: How much a user enjoys using the Web site.– Error frequency and severity - How often do users make errors while using the

system, how serious are these errors, and how do users recover from these errors?

– Memorability - If a user has used the system before, can he or she remember enough to use it effectively the next time or does the user have to start over again learning everything?

• There are two types of usability metrics that can be captured during a usability test. These metrics include:

– Performance data (what actually happened)– Preference data (what participants thought)

04/07/2304/07/23 3232

Usability metrics

• There are several metrics that you will want to collect and that you identified in the usability test plan:

– Successful Task Completion

– Critical & Non-Critical Errors

– Error-Free Rate

– Time On Task

– Subjective Measures

– Likes, Dislikes and Recommendations

04/07/2304/07/23 3333

Recommended reading

• Krug, Steve Don’t make me think New Riders / 2005 (2nd Edition)

• Nielsen, J Ten Usability Heuristics http://www.useit.com/papers/heuristic/heuristic_list.html

• Wroblewski, Luke Web Form Design: Filling in the Blanks (Rosenfeld Media) 2008

04/07/2304/07/23 3434

• Testing with users identifies any usability issues, and allows you to watch people as they interact with the site

• This provides an understanding of why any issues arise

• A representative sample of each user type completes a series of tasks on the website

• The use of Eye Tracking allows the users to undertake the test on their own, without distraction

• Immediately following the test the participant’s eye movements video is replayed from the session and they are asked to comment on their experience

• How eye tracking works• http://www.youtube.com/watch?v=xgRIjrlK1mA

• http://www.youtube.com/user/TobiiEyeTracking#p/u/14/tpLUkKN3AWE

Strategy

Build + TestEye Tracking Usability Testing

04/07/2304/07/23 3535

Eye tracking started over 100 years ago

Raymond Dodge’s Photochronograph(1871-1942)

Delabarre’s Eye Tracker (1898)

Edmund Huey’s Eye Tracker (1898)

04/07/2304/07/23 3636

Eye tracking has come a long way....And has often been very difficult to use….

04/07/2304/07/23 3737

04/07/2304/07/23 3838

From being VERY intrusive and complicated ...(examples of other Eye Trackers used even today ! )

04/07/2304/07/23 3939

Tobii:ing VERY Simple and Easy!!!

04/07/2304/07/23 4040

Underlying TechnologyCorneal reflection system, basic operating principles

• Tobii eye trackers use near infrared diodes to generate reflection patterns on the corneas of the eyes of the user.

• These reflection patterns, together with other visual information about the person, are collected by image sensors.

• Together with very sophisticated image analysis and mathematics a gaze point on the screen can be calculated,i.e. where the user is looking

Pupil Corneal reflection

04/07/2304/07/23 4141

...Fixations and Saccades

Page 41

A fixation describes points where the eye is relatively still and concentrating directly on a subject.

A saccade describes the rapid movements between fixations.

A series of fixations and saccades is known as a scan path. In Tobii Studio fixations appear as spots and saccades appear as the lines between fixations

04/07/2304/07/23 4242

...Fixations and Saccades

Page 42

• FIXATIONS :

• Fixations vary from about 100-600ms

• The majority of information about a scene is acquired during fixations

• The length of a fixation is usually an indication of cognitive processing

• When we fixate only 1-2 degrees of our visual field is seen with 100% accuracy (roughly the size of your thumbnail at arms length)

• The dwell time, or length of a fixation varies depending on the type of stimuli and our familiarity with its content

• Longer dwell times can indicate good, solid engagement

• Lot’s of short sporadic fixations can illustrate confusion, random searching or a lack of content deemed interesting or useful

• SACCADES :

• The average length of a saccade is 20-40ms

• Vision is largely suppressed during saccades

• Regressive saccades can reveal confusion or problems with understanding

• During saccadic movement our eyes can be drawn to areas of interest, contrast, shape and colour where we pause, engage and begin to fixate.

04/07/2304/07/23 4343

...Your Memory

04/07/2304/07/23 4444

Personas and Testing Environment

• Eye tracking is effective because it is non invasive however this massive advantage to studies is often negated by forcing people to be someone they aren’t! • The same thing can apply to test environments and test set-ups.

04/07/2304/07/23 4545

What Does That Gaze Plot Show?

Traditional search strategy on Google – displaying the ‘golden triangle’ and then still clicking on the first result – even after a brief foray below the fold – but with no sign of interest or detailed interaction.

More ‘golden triangle’ style behaviour but this time the participant fails to go below the fold and chooses a video link as their answer.

Very detailed search behaviour with 5 times the number of fixations and each result inspected, the participant then moved onto page two of the results.

04/07/2304/07/23 4646

Mouse Markers

• A “mouse marker” (see example opposite) can be a useful tool when doing web usability testing as it encourages the participant to park their mouse and click on a certain point.

• The benefit of this is that you are controlling the initial ‘start’ point for any visual outputs, allowing you to clip data effectively.

• You also reduce the risk of polluting data (especially statistical data) with false ‘hits’ as people search for the mouse. If you look below at the examples showing a pair of gaze plots with and without a marker prior to the webpage being displayed.

Without a mouse marker

With a mouse marker

04/07/2304/07/23 4747

Fixation Crosses

• A fixation cross (a basic image as shown opposite) between static stimuli is another way of normalising your data, and is particularly useful when doing print ad testing, or when you may be randomising stimuli.

• Naturally when someone reads text copy, see’s an advert or follows a user journey there will be a point where their gaze is carried over to the next stimuli as the image changes. By ‘resetting’ each participants vision to the centre of the screen.

• When randomising images this also allows you to remove any random legacy gaze points which could pollute the data sets.

• As everyone's gaze will be central you can then ‘clip’ the first 20-30ms to remove the forced hot spot of data, the can also ensure interaction elsewhere on the image is not biased due to an unrealistic peak of interaction... As shown below.

Page 47

Data including legacy data from fixation cross

Data shown with 30ms clipped from the initial exposure

04/07/2304/07/23 4848

Heat Maps – Different Metrics

Page 48

This heat map was created with the fixation count metric applied. This means that the hot spots indicate where the highest number of fixations were. In this case this does not necessarily gauge the level of engagements on the page, just the level of attraction.

The gaze opacity map option has the same functionality but reverses the way the data is displayed. The transparent areas are where the largest numbers of fixations were. Remember to take into account peoples peripheral vision, as the output illustrates the fixations only.

•Heat Maps are very commonly used outputs from eye tracking but it is important to know that there are three very different metrics that can be applied to the data, and each is suitable for different types of stimuli, methodology and testing.

•The three available metrics are :•Fixation count : Here the data is collated around the total number of fixations totalled across all the selected recordings, regardless of the duration of each fixation. For example the red areas may indicate 25 fixations.•Absolute duration : With this filter the data represents the total amount of time spent fixating in an area, across all of the selected recordings. For example the hot spot areas may indicate 9.234 seconds. This metric is useful when each person has seen the stimuli for the same exposure period.•Relative duration : The filter is ideal for web based testing, or if each participant sees a stimuli for a variable amount of time. Each persons interaction is looked at individually and the data normalised across all the recordings. The results are shown as a percentage of time spent on that stimuli.

04/07/2304/07/23 4949

Testing eCommerce

04/07/2304/07/23 5050

Retrospective Think Aloud vs Think AloudThink aloud testing slows down the users processes due to the additional cognitive workload. Here the user was tasked with searching for a house, and while doing this the moderator asked about how he found the site, the search engine and if he had noticed the advertising. The participant felt obliged to go below the page fold, then they interacted with the advertising after the moderator had mentioned it. You can clearly see the long fixations, and the extensive journey to basically complete a task that was literally right in front of them from the outset. Think aloud also allows users time to find the next step in a process or journey, potentially further damaging findings from testing. We can fixate 5 times or more a second, potentially over 300 fixations in the one minute this user took to complete this page – far more than we can verbalise and rationalise.By allowing the user to complete their task and interview them retrospectively this user (of a similar PC literacy to the previous participant) got on with the task in hand. They completed the task in 32 seconds, just over half the duration of our first example. More tellingly their interactions show that they didn’t go below the page, the didn’t interact with the advertising and after a very brief scan around the page they determined that their first choice of action was the correct one and searched for their potential new home. When speaking to them after the event during an RTA interview we initially start with the gaze and mouse data hidden – asking them their first impressions and what they thought they did, and then reveal their actual actions and discuss this further with them. The result is far more relevant, realistic and valid data.

04/07/2304/07/23 5151

Testing Mobile Devices

•For users with a T or 1750/2150 series eye tracker mobile device testing is possible using the scene camera function in Tobii Studio, and placing a camera under a table as seen in the image opposite. The user then interacts physically with the phone under the table – and views their actions on the screen and is eye tracked as normal. For obvious reasons this is not the most natural way of testing mobile devices!

•Utilising a X series eye tracker, inverted you can test mobile phone devices in a much more relaxed and realistic way. The eye tracker is inverted to adjust the viewing angle of the unit from 0 to 35 degrees into 0 to -35 degrees. This compensates for the fact you are looking underneath the tracker, not over it as you would with a screen based test. The calibration is done on a level plane with the tracker and there is an amount of movement allowed of the handset by the participant. A manual calibration is required, and a strict procedure to invert the calibration needs to be followed.

04/07/2304/07/23 5252

Testing Physical World Objects

Scene Camera For Physical Stimuli

•To further add to the flexibility of the X series tracker it can also be used with an external ‘scene’ camera to test physical objects such as mobile phones, printed material, toys, hand held devices and so on.

•As there is no screen to base a calibration on a grid is printed or drawn onto card or similar and this is then mapped within the software to ensure the eye tracked data is accurate.

•The camera is connected to the PC via a video capture board and live viewing is possible.

04/07/2304/07/23 5353

Online advertising effectiveness - iMotions

04/07/2304/07/23 5454

Contact details

Objective Digital Pty LtdABN 98 123 747 188Tel 1300 85 80 15Web ObjectiveDigital.comBlog UsableWorld.com.au

James Breeze, CEOEmail jbreeze@objectivedigital.comMob 0410 410 494

Level 10, 220 George StreetSydney NSW 2000

Click to Contact James

for a quote today!