a brief overview of ux research

A brief intro to UX Research

Dr. Panagiotis Zaharias

13/5/2014 @ CoLab Athens

The UX Mantra

To measure is to know.

If you can’t measure it,

you can’t improve it…

Overview of the talk

Examples of UX evaluation studies

Current trends in UX evaluation

UX Research: Main Concepts

Main Concepts in UX Research

First things first

The Ancestor: Human Computer Interaction

Human – Computer Interaction, is “a discipline concerned

with the design, evaluation and implementation of interactive

computing systems for human use and the study of major

phenomena surrounding them’’.

User friendly? Usable? …

Once upon a time we were talking about “user friendly” systems…

… so general and vague

Then usability emerged and was originally derived from the term ‘‘user friendly’’

… but usability is still a general and vague term

Many definitions and different approaches to how usability should be measured

What is usability anyway? The ISO definition (9241-11)

Usability is defined as: ‘‘the extent to which a product can be

used by specified users to achieve specified goals with

effectiveness; the extent to which the intended goals of use are

achieved,

efficiency; the resources that have to be expended to achieve the

intended goals and

satisfaction; the extent to which the user finds the use of the

product acceptable,

in a specified context of use’’.

Examples of usability measures (1)

Measures of effectiveness: Task completion (number of correct tasks, number of tasks

where users failed to finish within a set time, number of tasks where users gave up)

Accuracy measures (quantify the number of errors users make during the process of completing tasks)

Recall measures (how much information users can recall after the use of an interface)

Quality of outcome (attempts to measure the outcome of the tasks, measures of understanding e.g. tests of what has been learned from an e-learning system)


Measures of efficiency Time (refers to measures of how long users take to

complete tasks)

Input rate (e.g. words per minute)

Mental effort (mental resources spend on interaction, e.g. heart rate variability, subjective time estimation)

Usage patterns (number of times a certain action has been performed, how much information users access when solving tasks, deviation from the optimal solution etc.)


Measures of satisfaction Preference measures capture which interface users prefer

using

Specific attitudes towards the interface (liking, fun, annoyance etc.), toward the content of interface (quality of information, organization of information etc.)

Perception of outcomes (users’ assessment of their performance, users’ perception of learning, users’ confidence in the solution to tasks)

From Usability to User Experience (1)

UX - A relatively new field - Lack of unique definition…

But let’s take for example the ISO definition on UX:

“A person's perceptions and responses that result from the use

or anticipated use of a product, system or service”

From Usability to User Experience (2)

Usability

• Focused on optimizing task efficiency

• Removing problems in the user interface

• Studying interactive products mainly used for work

UX

• Dynamic context-dependent and subjective

• Focuses also on hedonic qualities of use (identification, stimulation, attractiveness etc.)

Current trends in UX evaluation

As we shift to User Experience…

Challenge#1: Need for new approaches and methods

Existing methods are somewhat unable -it is claimed- to

capture experience

Challenge#2: The choice of UX evaluation method

Qualitative or quantitative methods?

Only few studies combine qualitative and quantitative methods

What Dimensions of Experience Are

Studied?

Usability

• Effectiveness

• Efficiency

• Satisfaction

UX

• Emotions and affect

• Enjoyment

• Aesthetics

• And many more…

What Dimensions of Experience Are

Studied?

Source: Bargas- Avila and Hornbaek, 2012

How do we measure UX?

What are the methods applied in UX

Qualitative (50 %), quantitative (33%), combination (17%)

How UX data are collected?

Questionnaires are still the dominant UX evaluation method

(53%).


Popular UX evaluation methods


When to use which UX evaluation methods

Decision

Qualitative vs.

Quantitative

Attitudinal vs.

Behavioral

Context of system/

product use

Source: http://www.nngroup.com/articles/which-ux-research-methods/

The Attitudinal vs. Behavioral Dimension

to understand, measure, or inform change of people's stated beliefs

e.g. self-reported methods such as surveys, interviews, focus groups, etc.

Attitudinal to understand "what people do" with minimal interference from the method itself

e.g. A/B testing, observation and video recording, first click testing, eyetracking, etc.

Behavioral


The Qualitative vs. Quantitative Dimension

- the data is usually being gathered directly

- they are much better suited for answering question about why or how to fix a problem

Qualitative - the data is gathered indirectly, through an instrument, such as a survey or a web server log etc.

- they do a much better job answering how many and how much type of questions.

Quantitative


The Context of Use Dimension

This dimension can be described by the following options:

Natural or near-natural use of the system/product

Scripted use of the system/product

Not using the system/product during the study

A hybrid of the above


What about the use of UX evaluation

methods across the development phases ?

UX evaluation methods across development

phases

Product Development Phase

Strategize Optimize Assess

Goal: Inspire, explore and

choose new

directions and

opportunities

Inform and optimize

designs in order to

reduce risk and

improve usability

Measure product

performance against

itself or its

competition

Approach: Qualitative and

Quantitative

Mainly Qualitative

(formative)

Mainly Quantitative

(summative)

Typical

methods:

Ethnographic field

studies, focus groups,

diary studies,

surveys, data mining

or analytics etc.

e.g. cardsorting, field

studies, observation,

paper prototype and

usability studies, etc.

e.g. Usability

benchmarking,

surveys, A/B testing,

etc.


Some examples from UX research

studies

Walls of Nicosia The context:

"The Walls of Nicosia” is an interactive application that runs on a multi-touch table at the Leventeio Museum, Nicosia

Students from elementary schools as museum visitors

Goal: To conduct a comparative

summative evaluation

Methods used: Multiple choice knowledge tests

UX questionnaires (FunToolkit)

Observation

Indicative results: Mainly quantitative analysis (pre and post test analysis, t-tests)

UX questionnaire for children: the

FunToolkit (Read, 2007)

The Again-Again table

The Smileyometer

Source: http://hrast.pef.uni-lj.si/docs/research/Serious%20games/evaluation_paper_Janet.pdf

Long term UX evaluation of League of

Legends (1)

The context:

The well known MMOG “League of Legends”

Players were students at TEI of Kavala

Goal: to identify how UX of players

changed over time

Methods used:

An adaptation of UX Curve

Self reported method by drawing curves that depict changes in

UX over time

Long term UX evaluation of League of

Legends (2)

Indicative results:

Reason Categories Positive Negative

General UX 43 14

Ease of use 41 26

Immersion 43 10

Social interaction 38 12

Engagement 32 10

Total 197 72

General UX Curve for each player

Number of reasons for UX dimensions

Heuristic Evaluation of LMS

The context:

A well known cloud-based Learning Management System platform

Goal: to identify usability problems and possible areas for UX improvement

Methods used:

Heuristic evaluation

Heuristic evaluation protocol adapted by Mehlenbacher et al. (2005)

Indicative results: An evaluation report that contains the identified usability and

UX problems, their severity, and relevant redesign suggestions

Typical structure of the report of heuristic

evaluation

A numeric identifier of the problem A short description of the problem (and likely difficulties

for the user) Specific context (defined as location of the problem in the

interface): we identified this generically in terms of affecting the: Structure of the system, including layout, navigation, and basic help

functions such as print, help, etc. Content, including actual content for study, exploration, examples,

exercises, and so on Global issues, i.e., affecting both structure and learning content

The heuristic(s) used The severity rate: severe, moderate, and minor

First Click testing with mockups

The context:

A project for the design of new MOOC interfaces

Goal: to create, inform and optimize MOOC interface

design

Methods used:

Interviews and first click testing with users

First Click testing - indicative results

First Click testing - indicative results

0

2

4

6

8

10

12

Ραβδόγραμμα εργασιών

Σωστές εργασίες Λανθασμένες Εργασίες

Some UX resources:

World’s premier magazine for both academics and industry:

http://interactions.acm.org/

A comprehensive list of UX methods:

http://www.allaboutux.org/

An indicative list of UX tools:

http://www.smashingmagazine.com/2011/10/20/comprehensive

-review-usability-user-experience-testing-tools/

Other sources on the web:

http://uxmag.com/

http://www.usability.gov/

http://www.uxmatters.com/index.php









http://www.smashingmagazine.com/2011/10/20/comprehensive-review-usability-user-experience-testing-tools/






















Key takeaways:

You can’t afford NOT to evaluate UX

Always use a combination of methods

Don’t forget Usability

Do it properly…

Thank you!

Let’s keep in touch:

Web: about.me/panagiotis_zaharias

Email: [email protected]

LinkedIn: gr.linkedin.com/pub/panagiotis-zaharias/0/bb6/a91

Twitter: twitter.com/pan_sugarman

a brief overview of ux research

Design

ux evaluation

indicative

iso definition

focus groups

user experience

heuristic

intended goals

click testing