chi2012 analysis in practical usability evaluation web

Analysis in Practical Usability Evaluation:A Survey Study

Asbjørn Følstad, SINTEFEffie Lai-Chong Law, University of LeicesterKasper Hornbæk, University of Copenhagen

CHI 2012

https://www.uninett.no/sites/drupal.uninett.no.uninett/files/r2d2-logo.jpg

http://www.google.no/imgres?hl=en&sa=X&biw=1024&bih=482&tbm=isch&prmd=imvns&tbnid=2K5y6_jtz2OcyM:&imgrefurl=http://twintide.org/&docid=l0KCny8mo2uYkM&imgurl=http://www.irit.fr/recherches/ICS/projects/twintide/resources/tt_banner.jpg&w=1058&h=159&ei=6E17T5eYDcLm4QT8842IBA&zoom=1&iact=hc&vpx=2&vpy=280&dur=6422&hovh=87&hovw=580&tx=239&ty=50&sig=110134407136566683847&page=1&tbnh=23&tbnw=151&start=0&ndsp=10&ved=1t:429,r:5,s:0

A. Følstad, E. L-C. Law, K. Hornbæk: Analysis in practical usability evaluation: a survey study. 2

What is analysis?




Analysis superficially treated in textbooks

7,5 %Share of pages in Dumas & Redish’ a practical guide to usability testing treating Tabulating and analyzing data + Recommending changes.

In contrast, preparations for a usability test is covered in 46 % of the same book.




Few - if any - studies on how practitioners do analysis

Analysis frameworks and processes

Problem description formats

Analysis tools

User action framework (Andre, Hartson, Williges, 2001)

SUPEX – structured UP extraction(Cockton, Lavery, 1999)

Instant data analysis(Kjeldskov, Skov, Stage, 2004)

Templates(Lavery, Cockton, Atkinson, 1997)

Guidelines(Capra, 2006)

Usability problem inspector(Andre, Hartson, Williges, 2003)

Morae plugin for problem description and grouping(Howarth, Smith-Jackson, Hartson, 2009)

However, the research community has developed:




Few - if any - studies on how practitioners do analysis… and the research and standards communities have discussed the relation between evaluation and design

ISO 9241-210 – Human-centred design for interactive systems

Evaluation

Design

User reqirements

Context analysis



A. Følstad, E. L-C. Law, K. Hornbæk: Analysis in practical usability evaluation: a survey study.

A researcher perspective on analysis

The main output of an analysis in (formative) usability evaluation is a usability problem list to inform later design work

Research-based processes, methods and tools are needed to support rigorous analysis

6




But is the research user-centred?

Do we really know the practitioners?

Do we know how they do usability evaluation in general – and analysis of evaluation data in particular?




Survey to explore analysis practices

155 participants, mainly recruited through SIGCHI and UPA chapters

Experienced participants: Median usability work experience 5 years

Questions concerned the participants' latest usability evaluation

Latest evaluation within last 6 months- 62% within last 2 months

Target both usability testing and inspection – questionnaires adapted- 112 usability testing- 43 usability inspection




Research questions

1. How is analysis supported?

2. How are usability problems identified?

3. How do usability practitioners collaborate in analysis?

4. How is redesign integrated into the evaluation process? ?




Findings

Working hours – for the entire evaluation

Analysis resources

Tools used for analysis

Structured formats for problem description




4. How is redesign integrated into the evaluation process?




Findings

Working hours - for the entire evaluation?

Usability testing: 48 (median)

Usability inspection : 24 (median)








Findings

Analysis resources?

UT UI_

Heuristics / guidelines 60% 76%

Design patterns 41% 54%

Test participant opinion:

… usability problems 64%

… redesign suggestions 48%








FindingsTools used during analysis?(free text)

Screen recording and analysis software, e.g. Morae (11- all UT)

Drawing and prototyping tools, e.g. Balsamiq, Axure (8 – UT and UI)

Plain screen recording, e.g.

Camtasia, SnagIt (5 – all UT)

Web analytics, e.g. Google analytics,

Seevolution (5 - all UI)

(Where are the tools from the last 20 years of research?)








FindingsStructured formats for problem description?

55%: Problems described according to our own format

41%: No format, just plain prose

4%: Formats from standards or literature








Findings





Evaluation deliverables containing redesign suggestions?

Time of making redesign suggestions

Sources of redesign suggestions

Means of redesign presentation




Findings





Deliverable characterized as …

UT UI_A set of redesign suggestions motiva- 51% 53%ted from UPs

A set of UPs with some redesign suggestions 46% 43%

A set of UPs with noredesign suggestions 4% 5%




Findings





When are redesign suggestions made?

49%: First all UPs were identified, then redesign suggestions were made

46%: Some (or all) redesign suggestions were made immedeately upon identfying a UP




Findings





Sources of redesign suggestions

UT UI_Response to UPs 94% 74%

Non-optimal solution, even though no UP 38% 47% was observed




Findings





Means of redesign presentation

UT UI_Textual descriptions 68% 71%

Annotated screen shots 50% 55%

UI digital mock-ups 32% 47%

Sketching 29% 21%




A researcher perspective on analysis - revisited

The main output of an analysis in (formative) usability evaluation is a usability problem list to inform later design work


20





The main output of an analysis in (formative) usability evaluation is a usability problem list and a set of redesign suggestions – the latter often visually presented.


21

Researchers need to learn from practitioners how evaluation and design are related – not vice versa





The main output of an analysis in (formative) usability evaluation is a usability problem list and a set of redesign suggestions – the latter often visually presented.

Research-based processes, methods and tools need to be developed in response to practitioners needs – as seen from the practitioner perspective

22

Should we rather support home-growing of analysis support, and align with commercial tools?



23

Conclusion


A fast-paced presentation of a selection of the survey findings

If what you heard interests you: There is more to be found in the paper :-)

Thank you for your attention!



chi2012 analysis in practical usability evaluation web

Technology

practical usability

usability practitioners

analysis of evaluation

usability problems

formative usability

analysis resources

usability inspectiona

evaluation process