DECIDE:DECIDE: An evaluation frameworkAn evaluation framework

DECIDE: DECIDE: a framework to guide evaluationa framework to guide evaluation

DDetermine the goals.EExplore the questions.CChoose the evaluation approach and methods.

IIdentify the practical issues.DDecide how to deal with the ethical issues.

EEvaluate, analyze, interpret and present the data.

Determine the goalsDetermine the goalsWhat are the high-level goals of the evaluation?

Who wants it and why?The goals influence the approach used for the study.

Some examples of goals: Identify the best metaphor on which to base the design.

Check to ensure that the final interface is consistent.

Investigate how technology affects working practices.

Improve the usability of an existing product .

Explore the questionsExplore the questionsAll evaluations need goals & questions to guide them.

E.g., the goal of finding out why customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions:◦What are customers’ attitudes to these new tickets? ◦Are they concerned about security?◦Is the interface for obtaining them different, poor, new?

What questions are you asking about the design of your project?

Choose the evaluation approach & Choose the evaluation approach & methodsmethods

The evaluation approach shapes the methods used, and in turn, how data is collected, analyzed and presented.

E.g. field studies typically:◦Involve observation and interviews.◦Do not involve controlled tests in a laboratory.

◦Produce qualitative data.

TriangulationTriangulation

Using different methods (or “angles”) to answer the same or similar research questions, to verify or refine results

E.g. ◦Questionnaire◦Interview◦Experimental observation

Identify practical issues Identify practical issues

For example, how to:

Select usersStay on budgetStay on scheduleFind evaluatorsSelect equipment

Decide about ethical issuesDecide about ethical issuesDevelop an informed consent form

Participants have a right to:- Know the goals of the study;- Know what will happen to the findings;- Privacy of personal information;- Leave when they wish; - Be treated politely.

Evaluate, Evaluate, interpret interpret & present& present datadata

The approach and methods shape the evaluation, interpretation, and presentation of the data.

The following need to be considered:- Reliability: can the study be replicated?- Validity: is it measuring what you expected?- Biases: how is the process reflecting biases?- Scope: can the findings be generalized?- Ecological validity: is the environment influencing the findings? - i.e. Hawthorn effect.

Key pointsKey points

Issues to consider before conducting an evaluation study: the goals of the study the approaches and methods to use practical issues ethical issues how the data will be collected, analyzed and presented.

The DECIDE framework provides a simple checklist for planning an evaluation study.

http://usableweb.com This site was created by Keith Instone. It provides many links to sites that deal with different aspects of web usability. There are links to guidelines, examples of poor design and other information that will help you to select which methods to use in your own evaluations.

http://usableweb.com/topics/000862-0-0.html

provides an alternative way of classifying evaluation methods.

http://jthom.best.vwh.net/usability/

The Usability Methods Toolbox site, developed by James Hom,

last updated in 1998 but it contains many useful links to sites that discuss usability evaluation methods

offers an alternative framework for their classification.

http://www.otal.umd.edu/hci-rm/index.html

This site offers a classification and descriptions of some key techniques.

Graduate Students at University of Maryland developed this site.

There is also a section that discusses the role of theory in HCI design and evaluation.

http://universalusability.org/ This site groups information about different methods under three headings:

testing inspection inquiryThe information that it provides is very mixed in quality so be particularly discerning.

Much of the info comes from a limited range of sources.

Good parts include a link to Gary Perlman's site on questionnaires and good descriptions of inspection techniques.