statistical user experience measurement 1

User Experience Measurement

Prototype Comparison

Sharon Harper - Taxonomist & User Experience Consultant

Sharon Harper - Taxonomist & User Experience Consultant

User Experience Measurement

Taxonomy Structure, Organizational, Navigational Difference In Prototype Designs

Sharon Harper - Taxonomist & User Experience Consultant

Statistical Measurement Strategy Compare Means - task completion time

3 or more prototypes

2 prototypes

Across different participants

(independent samples)

Within same group of

participants(paired samples)

< 30 participants

t-test

≥ 30 participants

z-test

Sharon Harper - Taxonomist & User Experience Consultant

1. Null Hypothesis . there is no difference in the usability (performance & satisfaction) of the new user interface and the previous interface. Neither the taxonomy integration, organization or navigation efforts have improved the user experience.

1.(New user interface) – (Earlier user interface) = 0

2. Measure. What difference in the values of the means of two interfaces can be obtained that would disprove this hypothesis? The difference of the paired samples means by the same participants is relative to the spread or variability of their scores (time on successful task completion between prototype interfaces). This is the t value

T – test: Paired Two Samples for Means Comparison

Sharon Harper - Taxonomist & User Experience Consultant

2. Compare this result, the t-value, to determine whether the result is statistically significant. Assign risk factor of 5% (α= 0.05)to ensure 95% significance.

3. Number of paired measurements (participants) minus 1 is the degree of freedom, df.

4. With the acceptable risk factor of 5% or 95% significance, df, and t-value the hypothesis is tested.

5. The resulting p value is less than 0.05 then the null hypothesis is NOT TRUE The time on task for the two interfaces is significantly different, that is the new interface is better

T – test: Paired Two Samples for Means Comparison (cont.)

Sharon Harper - Taxonomist & User Experience Consultant

Excel™ Data Analysis Add In

Other software packages available : SPSS, SAS , MiniLab, http://usablestats.com …

Explanatory resources: The Cartoon Guide to Statistics by Gonick & SmithIntroductory Statistics by Ross

Web Center for Social Research Methodshttp://www.socialresearchmethods.net/kb/stat_t.php

Sharon Harper - Taxonomist & User Experience Consultant

Install Excel Data Analysis Add In

Sharon Harper - Taxonomist & User Experience Consultant

Time on Task t-Test paired sample for means

Sharon Harper - Taxonomist & User Experience Consultant

Result

Sharon Harper - Taxonomist & User Experience Consultant

Misinformation about statistical measurement is strongly ingrained, ‘math anxiety’

‘school’ marketing presenters, anticipate KIS(s) , use examples from others domains: medicine, linguistics, UX development

NOT pages of formulas, jargon

Do not show p-value = .001293051, use .00129 or .001

Usability analysts should have rudimentary statistical knowledge and able

to read power tables identifying the minimum sample size and acceptable

risk from that sample size.

Other methods should accompany quantitative methods

Report/Presentation Elements

Sharon Harper - Taxonomist & User Experience Consultant

UX Resources

Tullis & Albert. (2008) Measuring The User Experience: Collecting, Analyzing, and Presenting Usability Metrics.

Sauro. A Practical Guide to Measuring Usability: 72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software

Sauro & Lewis. (2012) Practical Statistics for User Research.

http://www.measuringusability.com

Questions?

Sharon Harper - Taxonomist & User Experience Consultant

Questions?