alan dix, janet finlay, gregory d. abowd, russell beale...

ALAN DIX, JANET FINLAY,GREGORY D. ABOWD, RUSSELL BEALE

THIRD EDITION

THIRDEDITION

HUMAN–COMPUTERINTERACTION

Much has changed since the first edition ofHuman–Computer Interaction was published. Ubiquitouscomputing and rich sensor-filled environments arefinding their way out of the laboratory, not just intomovies but also into our workplaces and homes. Thecomputer has broken out of its plastic and glassbounds providing us with networked societies wherepersonal computing devices from mobile phones tosmart cards fill our pockets and electronic devicessurround us at home and work. The web too has grownfrom a largely academic network into the hub ofbusiness and everyday lives. As the distinctions betweenthe physical and the digital, and between work andleisure start to break down, human–computerinteraction is also changing radically.

The excitement of these changes is captured in this newedition, which also looks forward to other emergingtechnologies. However, the book is firmly rooted instrong principles and models independent of thepassing technologies of the day: these foundations willbe the means by which today’s students willunderstand tomorrow’s technology.

The third edition of Human–Computer Interaction can beused for introductory and advanced courses on HCI,Interaction Design, Usability or Interactive SystemsDesign. It will also prove an invaluable reference forprofessionals wishing to design usable computingdevices.

Accompanying the text is a comprehensive websitecontaining a broad range of material for instructors,students and practitioners, a full text search facility forthe book, links to many sites of additional interest andmuch more: go to www.hcibook.com

Alan Dix is Professor in the Department of Computing, Lancaster, UK. Janet Finlay isProfessor in the School of Computing, Leeds Metropolitan University, UK. Gregory D. Abowd isAssociate Professor in the College of Computing and GVU Center at Georgia Tech, USA.Russell Beale is lecturer at the School of Computer Science, University ofBirmingham, UK.

Cover illustration by Peter Gudynas

New to this edition:

� A revised structure, reflecting the

growth of HCI as a discipline,

separates out basic material suitable

for introductory courses from more

detailed models and theories.

� New chapter on interaction designadds material on scenarios and basic

navigation design.

� New chapter on universal design,

substantially extending the coverage

of this material in the book.

� Updated and extended treatment of

socio/contextual issues.

� Extended and new material on novelinteraction, including updated

ubicomp material, designing

experience, physical sensors and a

new chapter on rich interaction.

� Updated material about the webincluding dynamic content.

� Relaunched website including case

studies, WAP access and search.

www.pearson-books.com

HUM

AN–CO

MPU

TERIN

TERACTION

DIXFINLAYABOWDBEALE

Human–Computer Interaction

We work with leading authors to develop the strongest educational materials in computing, bringing cutting-edge thinking and best learning practice to a global market.

Under a range of well-known imprints, including Prentice Hall, we craft high quality print and electronic publications which help readers to understand and apply their content, whether studying or at work.

To find out more about the complete range of our publishing, please visit us on the world wide web at: www.pearsoned.co.uk

Human–Computer Interaction

Third Edition

Alan Dix, Lancaster University

Janet Finlay, Leeds Metropolitan University

Gregory D. Abowd, Georgia Institute of Technology

Russell Beale, University of Birmingham

Pearson Education LimitedEdinburgh GateHarlowEssex CM20 2JEEngland

and Associated Companies throughout the world

Visit us on the world wide web at:www.pearsoned.co.uk

First published 1993Second edition published 1998Third edition published 2004

© Prentice-Hall Europe 1993, 1998© Pearson Education Limited 2004

The rights of Alan Dix, Janet E. Finlay, Gregory D. Abowd and Russell Beale to be identified as authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic,mechanical, photocopying, recording or otherwise, without either the prior written permission of the publisher or a licence permitting restricted copying in the United Kingdom issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP.

All trademarks used herein are the property of their respective owners. The use of any trademark in this text does not vest in the author or publisher any trademarkownership rights in such trademarks, nor does the use of such trademarks imply anyaffiliation with or endorsement of this book by such owners.

ISBN-13: 978-0-13-046109-4ISBN-10: 0-13-046109-1

British Library Cataloguing-in-Publication DataA catalogue record for this book is available from the British Library

10 9 8 7 6 5 4 310 09 08 07 06

Typeset in 10/121/2pt Minion by 35Printed and bound by Scotprint, Haddington

BRIEF CONTENTS

Guided tour xivForeword xviPreface to the third edition xixPublisher’s acknowledgements xxiiiIntroduction 1

FOUNDATIONS 9

Chapter 1 The human 11Chapter 2 The computer 59Chapter 3 The interaction 123Chapter 4 Paradigms 164

DESIGN PROCESS 189

Chapter 5 Interaction design basics 191Chapter 6 HCI in the software process 225Chapter 7 Design rules 258Chapter 8 Implementation support 289Chapter 9 Evaluation techniques 318Chapter 10 Universal design 365Chapter 11 User support 395

MODELS AND THEORIES 417

Chapter 12 Cognitive models 419Chapter 13 Socio-organizational issues and stakeholder requirements 450

Part 3

Part 2

Part 1

EVALUATION TECHNIQUES

OV E RV I E W

n Evaluation tests the usability, functionality andacceptability of an interactive system.

n Evaluation may take place:– in the laboratory– in the field.

n Some approaches are based on expert evaluation:– analytic methods– review methods– model-based methods.

n Some approaches involve users:– experimental methods– observational methods– query methods.

n An evaluation method must be chosen carefully andmust be suitable for the job.

9

9.2 Goals of evaluation 319

WHAT IS EVALUATION?

In previous chapters we have discussed a design process to support the design ofusable interactive systems. However, even if such a process is used, we still need toassess our designs and test our systems to ensure that they actually behave as weexpect and meet user requirements. This is the role of evaluation.

Evaluation should not be thought of as a single phase in the design process (stillless as an activity tacked on the end of the process if time permits). Ideally, evalu-ation should occur throughout the design life cycle, with the results of the evaluationfeeding back into modifications to the design. Clearly, it is not usually possible toperform extensive experimental testing continuously throughout the design, butanalytic and informal techniques can and should be used. In this respect, there is aclose link between evaluation and the principles and prototyping techniques we havealready discussed – such techniques help to ensure that the design is assessed con-tinually. This has the advantage that problems can be ironed out before considerableeffort and resources have been expended on the implementation itself: it is mucheasier to change a design in the early stages of development than in the later stages.We can make a broad distinction between evaluation by the designer or a usabilityexpert, without direct involvement by users, and evaluation that studies actual use of the system. The former is particularly useful for assessing early designs and prototypes; the latter normally requires a working prototype or implementation.However, this is a broad distinction and, in practice, the user may be involved inassessing early design ideas (for example, through focus groups), and expert-basedanalysis can be performed on completed systems, as a cheap and quick usabilityassessment. We will consider evaluation techniques under two broad headings:expert analysis and user participation.

Before looking at specific techniques, however, we will consider why we do evalu-ation and what we are trying to achieve.

GOALS OF EVALUATION

Evaluation has three main goals: to assess the extent and accessibility of the system’sfunctionality, to assess users’ experience of the interaction, and to identify anyspecific problems with the system.

The system’s functionality is important in that it must accord with the user’srequirements. In other words, the design of the system should enable users to per-form their intended tasks more easily. This includes not only making the appro-priate functionality available within the system, but making it clearly reachable bythe user in terms of the actions that the user needs to take to perform the task. It alsoinvolves matching the use of the system to the user’s expectations of the task. Forexample, if a filing clerk is used to retrieving a customer’s file by the postal address,

9.2

9.1

320 Chapter 9 n Evaluation techniques

the same capability (at least) should be provided in the computerized file system.Evaluation at this level may also include measuring the user’s performance with thesystem, to assess the effectiveness of the system in supporting the task.

In addition to evaluating the system design in terms of its functional capabilities,it is important to assess the user’s experience of the interaction and its impact uponhim. This includes considering aspects such as how easy the system is to learn, itsusability and the user’s satisfaction with it. It may also include his enjoyment andemotional response, particularly in the case of systems that are aimed at leisure orentertainment. It is important to identify areas of the design that overload the userin some way, perhaps by requiring an excessive amount of information to be remem-bered, for example. A fuller classification of principles that can be used as evaluationcriteria is provided in Chapter 7. Much evaluation is aimed at measuring featuressuch as these.

The final goal of evaluation is to identify specific problems with the design. Thesemay be aspects of the design which, when used in their intended context, cause unexpected results, or confusion amongst users. This is, of course, related to both thefunctionality and usability of the design (depending on the cause of the problem).However, it is specifically concerned with identifying trouble-spots which can thenbe rectified.

EVALUATION THROUGH EXPERT ANALYSIS

As we have noted, evaluation should occur throughout the design process. In particular, the first evaluation of a system should ideally be performed before anyimplementation work has started. If the design itself can be evaluated, expensive mis-takes can be avoided, since the design can be altered prior to any major resourcecommitments. Typically, the later in the design process that an error is discovered,the more costly it is to put right and, therefore, the less likely it is to be rectified.However, it can be expensive to carry out user testing at regular intervals during thedesign process, and it can be difficult to get an accurate assessment of the experienceof interaction from incomplete designs and prototypes. Consequently, a number ofmethods have been proposed to evaluate interactive systems through expert analysis.These depend upon the designer, or a human factors expert, taking the design andassessing the impact that it will have upon a typical user. The basic intention is to identify any areas that are likely to cause difficulties because they violate knowncognitive principles, or ignore accepted empirical results. These methods can be used at any stage in the development process from a design specification, throughstoryboards and prototypes, to full implementations, making them flexible evalu-ation approaches. They are also relatively cheap, since they do not require userinvolvement. However, they do not assess actual use of the system, only whether ornot a system upholds accepted usability principles.

We will consider four approaches to expert analysis: cognitive walkthrough,heuristic evaluation, the use of models and use of previous work.

9.3

9.3 Evaluation through expert analysis 321

9.3.1 Cognitive walkthrough

Cognitive walkthrough was originally proposed and later revised by Polson and col-leagues [294, 376] as an attempt to introduce psychological theory into the informaland subjective walkthrough technique.

The origin of the cognitive walkthrough approach to evaluation is the code walk-through familiar in software engineering. Walkthroughs require a detailed review ofa sequence of actions. In the code walkthrough, the sequence represents a segmentof the program code that is stepped through by the reviewers to check certain char-acteristics (for example, that coding style is adhered to, conventions for spelling variables versus procedure calls, and to check that system-wide invariants are notviolated). In the cognitive walkthrough, the sequence of actions refers to the stepsthat an interface will require a user to perform in order to accomplish some knowntask. The evaluators then ‘step through’ that action sequence to check it for poten-tial usability problems. Usually, the main focus of the cognitive walkthrough is toestablish how easy a system is to learn. More specifically, the focus is on learningthrough exploration. Experience shows that many users prefer to learn how to use asystem by exploring its functionality hands on, and not after sufficient training orexamination of a user’s manual. So the checks that are made during the walkthroughask questions that address this exploratory learning. To do this, the evaluators gothrough each step in the task and provide a ‘story’ about why that step is or is notgood for a new user. To do a walkthrough (the term walkthrough from now on refersto the cognitive walkthrough, and not to any other kind of walkthrough), you needfour things:

1. A specification or prototype of the system. It doesn’t have to be complete, but itshould be fairly detailed. Details such as the location and wording for a menu canmake a big difference.

2. A description of the task the user is to perform on the system. This should be arepresentative task that most users will want to do.

3. A complete, written list of the actions needed to complete the task with the pro-posed system.

4. An indication of who the users are and what kind of experience and knowledgethe evaluators can assume about them.

Given this information, the evaluators step through the action sequence(identified in item 3 above) to critique the system and tell a believable story about itsusability. To do this, for each action, the evaluators try to answer the following fourquestions for each step in the action sequence.

1. Is the effect of the action the same as the user’s goal at that point? Each useraction will have a specific effect within the system. Is this effect the same as whatthe user is trying to achieve at this point? For example, if the effect of the actionis to save a document, is ‘saving a document’ what the user wants to do?

2. Will users see that the action is available? Will users see the button or menu item,for example, that is used to produce the action? This is not asking whether theywill recognize that the button is the one they want. This is merely asking whether


it is visible to them at the time when they will need to use it. Instances where the answer to this question might be ‘no’ are, for example, where a VCR remotecontrol has a covered panel of buttons or where a menu item is hidden away in asubmenu.

3. Once users have found the correct action, will they know it is the one they need?This complements the previous question. It is one thing for a button or menuitem to be visible, but will the user recognize that it is the one he is looking for tocomplete his task? Where the previous question was about the visibility of theaction, this one is about whether its meaning and effect is clear.

4. After the action is taken, will users understand the feedback they get? If you nowassume that the user did manage to achieve the correct action, will he know thathe has done so? Will the feedback given be sufficient confirmation of what hasactually happened? This is the completion of the execution–evaluation interac-tion cycle (see Chapter 3). In order to determine if they have accomplished theirgoal, users need appropriate feedback.

It is vital to document the cognitive walkthrough to keep a record of what is goodand what needs improvement in the design. It is therefore a good idea to producesome standard evaluation forms for the walkthrough. The cover form would list theinformation in items 1–4 in our first list above, as well as identifying the date andtime of the walkthrough and the names of the evaluators. Then for each action (fromitem 3 on the cover form), a separate standard form is filled out that answers each ofthe four questions in our second list above. Any negative answer for any of the ques-tions for any particular action should be documented on a separate usability prob-lem report sheet. This problem report sheet should indicate the system being built(the version, if necessary), the date, the evaluators and a detailed description of theusability problem. It is also useful to indicate the severity of the problem, that iswhether the evaluators think this problem will occur often, and how serious it willbe for the users. This information will help the designers to decide priorities for correcting the design, since it is not always possible to fix every problem.

Example: programming a video recorder by remote control

We can illustrate how the walkthrough method works using a simple example. Imagine weare designing a remote control for a video recorder (VCR) and are interested in the task of pro-gramming the VCR to do timed recordings. Our initial design is shown in Figure 9.1. The pictureon the left illustrates the handset in normal use, the picture on the right after the timed recordbutton has been pressed. The VCR allows the user to program up to three timed recordings indifferent ‘streams’. The next available stream number is automatically assigned. We want to knowwhether our design supports the user’s task. We begin by identifying a representative task.

Program the video to time-record a program starting at 18.00 and finishing at 19.15 on channel4 on 24 February 2005.


We will assume that the user is familiar with VCRs but not with this particular design.

The next step in the walkthrough is to identify the action sequence for this task. We specify thisin terms of the user’s action (UA) and the system’s display or response (SD). The initial display isas the left-hand picture in Figure 9.1.

UA 1: Press the ‘timed record’ buttonSD 1: Display moves to timer mode. Flashing cursor appears after ‘start:’UA 2: Press digits 1 8 0 0SD 2: Each digit is displayed as typed and flashing cursor moves to next positionUA 3: Press the ‘timed record’ buttonSD 3: Flashing cursor moves to ‘end:’UA 4: Press digits 1 9 1 5SD 4: Each digit is displayed as typed and flashing cursor moves to next positionUA 5: Press the ‘timed record’ buttonSD 5: Flashing cursor moves to ‘channel:’UA 6: Press digit 4SD 6: Digit is displayed as typed and flashing cursor moves to next positionUA 7: Press the ‘timed record’ buttonSD 7: Flashing cursor moves to ‘date:’UA 8: Press digits 2 4 0 2 0 5SD 8: Each digit is displayed as typed and flashing cursor moves to next positionUA 9: Press the ‘timed record’ buttonSD 9: Stream number in top right-hand corner of display flashesUA 10: Press the ‘transmit’ buttonSD 10: Details are transmitted to video player and display returns to normal mode

Figure 9.1 An initial remote control design


9.3.2 Heuristic evaluation

A heuristic is a guideline or general principle or rule of thumb that can guide a designdecision or be used to critique a decision that has already been made. Heuristicevaluation, developed by Jakob Nielsen and Rolf Molich, is a method for structuringthe critique of a system using a set of relatively simple and general heuristics.Heuristic evaluation can be performed on a design specification so it is useful forevaluating early design. But it can also be used on prototypes, storyboards and fullyfunctioning systems. It is therefore a flexible, relatively cheap approach. Hence it isoften considered a discount usability technique.

The general idea behind heuristic evaluation is that several evaluators inde-pendently critique a system to come up with potential usability problems. It isimportant that there be several of these evaluators and that the evaluations be doneindependently. Nielsen’s experience indicates that between three and five evaluatorsis sufficient, with five usually resulting in about 75% of the overall usability problemsbeing discovered.

Having determined our action list we are in a position to proceed with the walkthrough. For eachaction (1–10) we must answer the four questions and tell a story about the usability of the system.Beginning with UA 1:

UA 1: Press the ‘timed record’ buttonQuestion 1: Is the effect of the action the same as the user’s goal at that point?The timed record button initiates timer programming. It is reasonable to assume that a userfamiliar with VCRs would be trying to do this as his first goal.Question 2: Will users see that the action is available?The ‘timed record’ button is visible on the remote control.Question 3: Once users have found the correct action, will they know it is the one they need?It is not clear which button is the ‘timed record’ button. The icon of a clock (fourth buttondown on the right) is a possible candidate but this could be interpreted as a button to changethe time. Other possible candidates might be the fourth button down on the left or the filledcircle (associated with record). In fact, the icon of the clock is the correct choice but it is quitepossible that the user would fail at this point. This identifies a potential usability problem.Question 4: After the action is taken, will users understand the feedback they get?Once the action is taken the display changes to the timed record mode and shows familiar head-ings (start, end, channel, date). It is reasonable to assume that the user would recognize theseas indicating successful completion of the first action.

So we find we have a potential usability problem relating to the icon used on the ‘timed record’button. We would now have to establish whether our target user group could correctly distin-guish this icon from others on the remote.

The analysis proceeds in this fashion, with a walkthrough form completed for each action. We willleave the rest of the walkthrough for you to complete as an exercise. What other usability prob-lems can you identify with this design?


To aid the evaluators in discovering usability problems, a set of 10 heuristics areprovided. The heuristics are related to principles and guidelines (see Chapter 7). Thesecan be supplemented where required by heuristics that are specific to the particulardomain. So, for example, if the system is for synchronous group communication, onemight add ‘awareness of other users’ as a heuristic. Although Nielsen recommendsthe use of these 10 as providing the most effective coverage of the most common us-ability problems, other rules, such as those discussed in Chapter 7, could also be used.

Each evaluator assesses the system and notes violations of any of these heuristicsthat would indicate a potential usability problem. The evaluator also assesses theseverity of each usability problem, based on four factors: how common is the prob-lem, how easy is it for the user to overcome, will it be a one-off problem or a persist-ent one, and how seriously will the problem be perceived? These can be combinedinto an overall severity rating on a scale of 0–4:

0 = I don’t agree that this is a usability problem at all1 = Cosmetic problem only: need not be fixed unless extra time is available on project2 = Minor usability problem: fixing this should be given low priority3 = Major usability problem: important to fix, so should be given high priority4 = Usability catastrophe: imperative to fix this before product can be released (Nielsen)

Nielsen’s ten heuristics are:

1. Visibility of system status Always keep users informed about what is going on,through appropriate feedback within reasonable time. For example, if a systemoperation will take some time, give an indication of how long and how much iscomplete.

2. Match between system and the real world The system should speak the user’slanguage, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making informationappear in natural and logical order.

3. User control and freedom Users often choose system functions by mistake and need a clearly marked ‘emergency exit’ to leave the unwanted state withouthaving to go through an extended dialog. Support undo and redo.

4. Consistency and standards Users should not have to wonder whether words,situations or actions mean the same thing in different contexts. Follow platformconventions and accepted standards.

5. Error prevention Make it difficult to make errors. Even better than good errormessages is a careful design that prevents a problem from occurring in the firstplace.

6. Recognition rather than recall Make objects, actions and options visible. Theuser should not have to remember information from one part of the dialog toanother. Instructions for use of the system should be visible or easily retrievablewhenever appropriate.

7. Flexibility and efficiency of use Allow users to tailor frequent actions.Accelerators – unseen by the novice user – may often speed up the interactionfor the expert user to such an extent that the system can cater to both inexperi-enced and experienced users.


8. Aesthetic and minimalist design Dialogs should not contain information thatis irrelevant or rarely needed. Every extra unit of information in a dialog competeswith the relevant units of information and diminishes their relative visibility.

9. Help users recognize, diagnose and recover from errors Error messages shouldbe expressed in plain language (no codes), precisely indicate the problem, andconstructively suggest a solution.

10. Help and documentation Few systems can be used with no instructions so itmay be necessary to provide help and documentation. Any such informationshould be easy to search, focussed on the user’s task, list concrete steps to be carried out, and not be too large.

Once each evaluator has completed their separate assessment, all of the problemsare collected and the mean severity ratings calculated. The design team will thendetermine the ones that are the most important and will receive attention first.

9.3.3 Model-based evaluation

A third expert-based approach is the use of models. Certain cognitive and designmodels provide a means of combining design specification and evaluation into thesame framework. These are discussed in detail in Chapter 12. For example, theGOMS (goals, operators, methods and selection) model predicts user performancewith a particular interface and can be used to filter particular design options.Similarly, lower-level modeling techniques such as the keystroke-level model pro-vide predictions of the time users will take to perform low-level physical tasks.

Design methodologies, such as design rationale (see Chapter 6), also have a role toplay in evaluation at the design stage. Design rationale provides a framework inwhich design options can be evaluated. By examining the criteria that are associatedwith each option in the design, and the evidence that is provided to support these criteria, informed judgments can be made in the design.

Dialog models can also be used to evaluate dialog sequences for problems, such asunreachable states, circular dialogs and complexity. Models such as state transitionnetworks are useful for evaluating dialog designs prior to implementation. These arediscussed in detail in Chapter 16.

9.3.4 Using previous studies in evaluation

Experimental psychology and human–computer interaction between them possess a wealth of experimental results and empirical evidence. Some of this is specific to aparticular domain, but much deals with more generic issues and applies in a varietyof situations. Examples of such issues are the usability of different menu types, therecall of command names, and the choice of icons.

A final approach to expert evaluation exploits this inheritance, using previousresults as evidence to support (or refute) aspects of the design. It is expensive torepeat experiments continually and an expert review of relevant literature can avoid

9.4 Evaluation through user participation 327

the need to do so. It should be noted that experimental results cannot be expected to hold arbitrarily across contexts. The reviewer must therefore select evidence care-fully, noting the experimental design chosen, the population of participants used, theanalyses performed and the assumptions made. For example, an experiment testingthe usability of a particular style of help system using novice participants may notprovide accurate evaluation of a help system designed for expert users. The reviewshould therefore take account of both the similarities and the differences between the experimental context and the design under consideration. This is why this is an expert review: expertise in the area is required to ensure that correct assumptionsare made.

EVALUATION THROUGH USER PARTICIPATION

The techniques we have considered so far concentrate on evaluating a design or system through analysis by the designer, or an expert evaluator, rather than testingwith actual users. However, useful as these techniques are for filtering and refiningthe design, they are not a replacement for actual usability testing with the people for whom the system is intended: the users. In this section we will look at a numberof different approaches to evaluation through user participation. These includeempirical or experimental methods, observational methods, query techniques, andmethods that use physiological monitoring, such as eye tracking and measures ofheart rate and skin conductance.

User participation in evaluation tends to occur in the later stages of developmentwhen there is at least a working prototype of the system in place. This may rangefrom a simulation of the system’s interactive capabilities, without its underlyingfunctionality (for example, the Wizard of Oz technique, which is discussed inChapter 6, through a basic functional prototype to a fully implemented system.However, some of the methods discussed can also contribute to the earlier designstages, such as requirements capture, where observation and surveying users areimportant (see Chapter 13).

9.4.1 Styles of evaluation

Before we consider some of the techniques that are available for evaluation withusers, we will distinguish between two distinct evaluation styles: those performedunder laboratory conditions and those conducted in the work environment or ‘in thefield’.

Laboratory studies

In the first type of evaluation studies, users are taken out of their normal work envir-onment to take part in controlled tests, often in a specialist usability laboratory

9.4


(although the ‘lab’ may simply be a quiet room). This approach has a number ofbenefits and disadvantages.

A well-equipped usability laboratory may contain sophisticated audio/visualrecording and analysis facilities, two-way mirrors, instrumented computers and thelike, which cannot be replicated in the work environment. In addition, the particip-ant operates in an interruption-free environment. However, the lack of context – forexample, filing cabinets, wall calendars, books or interruptions – and the unnaturalsituation may mean that one accurately records a situation that never arises in thereal world. It is especially difficult to observe several people cooperating on a task ina laboratory situation, as interpersonal communication is so heavily dependent oncontext (see Section 9.4.2).

There are, however, some situations where laboratory observation is the onlyoption, for example, if the system is to be located in a dangerous or remote loca-tion, such as a space station. Also some very constrained single-user tasks may beadequately performed in a laboratory. Finally, and perhaps most commonly, we maydeliberately want to manipulate the context in order to uncover problems or observeless used procedures, or we may want to compare alternative designs within a con-trolled context. For these types of evaluation, laboratory studies are appropriate.

Field studies

The second type of evaluation takes the designer or evaluator out into the user’swork environment in order to observe the system in action. Again this approach hasits pros and cons.

High levels of ambient noise, greater levels of movement and constant inter-ruptions, such as phone calls, all make field observation difficult. However, the very ‘open’ nature of the situation means that you will observe interactions betweensystems and between individuals that would have been missed in a laboratory study.The context is retained and you are seeing the user in his ‘natural environment’. In addition, some activities, such as those taking days or months, are impossible tostudy in the laboratory (though difficult even in the field).

On balance, field observation is to be preferred to laboratory studies as it allows usto study the interaction as it occurs in actual use. Even interruptions are importantas these will expose behaviors such as saving and restoring state during a task.However, we should remember that even in field observations the participants arelikely to be influenced by the presence of the analyst and/or recording equipment, sowe always operate at a slight remove from the natural situation, a sort of Heisenberguncertainty principle.

This is, of course, a generalization: there are circumstances, as we have noted, inwhich laboratory testing is necessary and desirable. In particular, controlled experi-ments can be useful for evaluation of specific interface features, and must normallybe conducted under laboratory conditions. From an economic angle, we need toweigh the costs of establishing recording equipment in the field, and possibly dis-rupting the actual work situation, with the costs of taking one or more participants


away from their jobs into the laboratory. This balance is not at all obvious and anystudy must weigh the loss of contextual information against the increased costs anddifficulty of field studies.

9.4.2 Empirical methods: experimental evaluation

One of the most powerful methods of evaluating a design or an aspect of a design is to use a controlled experiment. This provides empirical evidence to support a particular claim or hypothesis. It can be used to study a wide range of different issuesat different levels of detail.

Any experiment has the same basic form. The evaluator chooses a hypothesis totest, which can be determined by measuring some attribute of participant behavior.A number of experimental conditions are considered which differ only in the valuesof certain controlled variables. Any changes in the behavioral measures are attributedto the different conditions. Within this basic form there are a number of factors thatare important to the overall reliability of the experiment, which must be consideredcarefully in experimental design. These include the participants chosen, the variablestested and manipulated, and the hypothesis tested.

Participants

The choice of participants is vital to the success of any experiment. In evaluationexperiments, participants should be chosen to match the expected user populationas closely as possible. Ideally, this will involve experimental testing with the actualusers but this is not always possible. If participants are not actual users, they shouldbe chosen to be of a similar age and level of education as the intended user group.Their experience with computers in general, and with systems related to that beingtested, should be similar, as should their experience or knowledge of the taskdomain. It is no good testing an interface designed to be used by the general publicon a participant set made up of computer science undergraduates: they are simplynot representative of the intended user population.

A second issue relating to the participant set is the sample size chosen. Often this is something that is determined by pragmatic considerations: the availability ofparticipants is limited or resources are scarce. However, the sample size must be largeenough to be considered to be representative of the population, taking into accountthe design of the experiment and the statistical methods chosen.

Nielsen and Landauer [264] suggest that usability testing with a single participantwill find about a third of the usability problems, and that there is little to be gainedfrom testing with more than five. While this may be true of observational studieswhere the aim is simply to uncover usability issues, it is not possible to discovermuch about the extent of usability problems from such small numbers. Certainly, ifthe intention is to run a controlled experiment and perform statistical analysis on theresults, at least twice this number is recommended.


Variables

Experiments manipulate and measure variables under controlled conditions, inorder to test the hypothesis. There are two main types of variable: those that are‘manipulated’ or changed (known as the independent variables) and those that aremeasured (the dependent variables).

Independent variables are those elements of the experiment that are manipu-lated to produce different conditions for comparison. Examples of independent variables in evaluation experiments are interface style, level of help, number of menu items and icon design. Each of these variables can be given a number of dif-ferent values; each value that is used in an experiment is known as a level of the variable. So, for example, an experiment that wants to test whether search speedimproves as the number of menu items decreases may consider menus with five,seven, and ten items. Here the independent variable, number of menu items, hasthree levels.

More complex experiments may have more than one independent variable. Forexample, in the above experiment, we may suspect that the speed of the user’sresponse depends not only on the number of menu items but also on the choice ofcommands used on the menu. In this case there are two independent variables. If there were two sets of command names (that is, two levels), we would require six experimental conditions to investigate all the possibilities (three levels of menusize × two levels of command names).

Dependent variables, on the other hand, are the variables that can be measured inthe experiment, their value is ‘dependent’ on the changes made to the independentvariable. In the example given above, this would be the speed of menu selection. The dependent variable must be measurable in some way, it must be affected by theindependent variable, and, as far as possible, unaffected by other factors. Commonchoices of dependent variable in evaluation experiments are the time taken to com-plete a task, the number of errors made, user preference and the quality of the user’sperformance. Obviously, some of these are easier to measure objectively than others.However, the more subjective measures can be applied against predetermined scales,and can be very important factors to consider.

Hypotheses

A hypothesis is a prediction of the outcome of an experiment. It is framed in termsof the independent and dependent variables, stating that a variation in the inde-pendent variable will cause a difference in the dependent variable. The aim of theexperiment is to show that this prediction is correct. This is done by disproving thenull hypothesis, which states that there is no difference in the dependent variablebetween the levels of the independent variable. The statistical measures describedbelow produce values that can be compared with various levels of significance. If a result is significant it shows, at the given level of certainty, that the differences measured would not have occurred by chance (that is, that the null hypothesis isincorrect).


Experimental design

In order to produce reliable and generalizable results, an experiment must be care-fully designed. We have already looked at a number of the factors that the experi-menter must consider in the design, namely the participants, the independent anddependent variables, and the hypothesis. The first phase in experimental design thenis to choose the hypothesis: to decide exactly what it is you are trying to demonstrate.In doing this you are likely to clarify the independent and dependent variables, inthat you will have identified what you are going to manipulate and what change youexpect. If your hypothesis does not clearly identify these variables then you need torethink it. At this stage you should also consider your participants: how many areavailable and are they representative of the user group?

The next step is to decide on the experimental method that you will use. There are two main methods: between-subjects and within-subjects. In a between-subjects(or randomized) design, each participant is assigned to a different condition. There are at least two conditions: the experimental condition (in which the variable hasbeen manipulated) and the control, which is identical to the experimental conditionexcept for this manipulation. This control serves to ensure that it is the manipulationthat is responsible for any differences that are measured. There may, of course, bemore than two groups, depending on the number of independent variables and thenumber of levels that each variable can take.

The advantage of a between-subjects design is that any learning effect resultingfrom the user performing in one condition and then the other is controlled: eachuser performs under only one condition. The disadvantages are that a greater num-ber of participants are required, and that significant variation between the groupscan negate any results. Also, individual differences between users can bias the results. These problems can be handled by a careful selection of participants, ensur-ing that all are representative of the population and by matching participantsbetween groups.

The second experimental design is within-subjects (or repeated measures). Hereeach user performs under each different condition. This design can suffer fromtransfer of learning effects, but this can be lessened if the order in which the condi-tions are tackled is varied between users, for example, group A do first condition fol-lowed by second and group B do second condition followed by first. Within-subjectsis less costly than between-subjects, since fewer users are required, and it can be particularly effective where learning is involved. There is also less chance of effectsfrom variation between participants.

The choice of experimental method will depend on the resources available, howfar learning transfer is likely or can be controlled, and how representative the particip-ant group is considered to be. A popular compromise, in cases where there is morethan one independent variable, is to devise a mixed design where one variable isplaced between-groups and one within-groups. So, returning to our example of themenu design, the participants would be split into two groups, one for each commandset, but each group would perform in three conditions, corresponding to the threepossible levels of the number of menu items.


Once we have determined the hypothesis we are trying to test, the variables we arestudying, the participants at our disposal, and the design that is most appropriate, we have to decide how we are going to analyze the results we record. There are anumber of statistical tests available, and the choice of test is vital to the success of theexperiment. Different tests make different assumptions about the data and if aninappropriate test is chosen, the results can be invalid. The next subsection discussesthe factors to consider in choosing a statistical test and surveys the most commonstatistical measures available.

Statistical measures

The first two rules of statistical analysis are to look at the data and to save the data. It is easy to carry out statistical tests blindly when a glance at a graph, histogram or table of results would be more instructive. In particular, looking at the data can expose outliers, single data items that are very different from the rest. Outliers are often the result of a transcription error or a freak event not connected to theexperiment. For example, we notice that one participant took three times as long aseveryone else to do a task. We investigate and discover that the participant had beensuffering from flu on the day of the experiment. Clearly, if the participant’s data wereincluded it would bias the results.

Saving the data is important, as we may later want to try a different analysismethod. It is all too common for an experimenter to take some averages or other-wise tabulate results, and then throw away the original data. At worst, the remainingstatistics can be useless for statistical purposes, and, at best, we have lost the abilityto trace back odd results to the original data, as, for example, we want to do for outliers.

Our choice of statistical analysis depends on the type of data and the questions wewant to answer. It is worth having important results checked by an experiencedstatistician, but in many situations standard tests can be used.

Variables can be classified as either discrete variables or continuous variables. A discrete variable can only take a finite number of values or levels, for example, ascreen color that can be red, green or blue. A continuous variable can take any value(although it may have an upper or lower limit), for example a person’s height or thetime taken to complete a task. A special case of continuous data is when they are pos-itive, for example a response time cannot be negative. A continuous variable can berendered discrete by clumping it into classes, for example we could divide heightsinto short (<5 ft (1.5 m)), medium (5–6 ft (1.5–1.8 m)) and tall (>6 ft (1.8 m)). Inmany interface experiments we will be testing one design against another. In thesecases the independent variable is usually discrete.

The dependent variable is the measured one and subject to random experimentalvariation. In the case when this variable is continuous, the random variation maytake a special form. If the form of the data follows a known distribution then specialand more powerful statistical tests can be used. Such tests are called parametric testsand the most common of these are used when the variation follows the normal distribution. This means that if we plot a histogram of the random errors, they will


form the well-known bell-shaped graph (Figure 9.2). Happily, many of these tests arefairly robust, that is they give reasonable results even when the data are not preciselynormal. This means that you need not worry too much about checking normalityduring early analysis.

There are ways of checking whether data are really normal, but for these the reader should consult a statistics book, or a professional statistician. However, as a general rule, if data can be seen as the sum or average of many small independ-ent effects they are likely to be normal. For example, the time taken to complete acomplex task is the sum of the times of all the minor tasks of which it is composed.On the other hand, a subjective rating of the usability of an interface will not be normal. Occasionally data can be transformed to become approximately normal. The most common is the log-transformation, which is used for positive data with near-zero values. As a log-transformation has little effect when the data are clustered well away from zero, many experimenters habitually log-transform.However, this practice makes the results more difficult to interpret and is not recommended.

When we cannot assume that data are normally distributed, we must often resortto non-parametric tests. These are statistical tests that make no assumptions aboutthe particular distribution and are usually based purely on the ranking of the data.That is, each item of a data set (for example, 57, 32, 61, 49) is reduced to its rank (3, 1, 4, 2), before analysis begins. Because non-parametric tests make fewer assump-tions about the data than parametric tests, and are more resistant to outliers, there is less danger of getting spurious results. However, they are less powerful than thecorresponding parametric tests. This means that, given the same set of data, a para-metric test might detect a difference that the non-parametric test would miss.

A third sort of test is the contingency table, where we classify data by several discrete attributes and then count the number of data items with each attribute combination.

Table 9.1 lists some of the standard tests categorized by the form of independentand dependent variables (discrete/continuous/normal). Normality is not an issue

Figure 9.2 Histogram of normally distributed errors


for the independent variable, but a special case is when it is discrete with only two values, for example comparing two systems. We cannot describe all the tech-niques here; for this you should use a standard statistics text, such as one of thoserecommended in the reading list. The table is only intended to guide you in yourchoice of test.

An extensive and accurate analysis is no use if it answers the wrong question.Examples of questions one might ask about the data are as follows:

Is there a difference? For example, is one system better than another? Techniquesthat address this are called hypothesis testing. The answers to this question are notsimply yes/no, but of the form: ‘we are 99% certain that selection from menus offive items is faster than that from menus of seven items’.

How big is the difference? For example, ‘selection from five items is 260 ms faster than from seven items’. This is called point estimation, often obtained byaverages.

How accurate is the estimate? For example, ‘selection is faster by 260 ± 30 ms’.Statistical answers to this are in the form of either measures of variation such asthe standard deviation of the estimate, or confidence intervals. Again, the answersone obtains are probabilistic: ‘we are 95% certain that the difference in responsetime is between 230 and 290 ms’.

The experimental design issues we have discussed have been principally addressed at the first question. However, most of the statistical techniques listed above, bothparametric and non-parametric, give some answer to one or both of the other questions.

Table 9.1 Choosing a statistical technique

Independent Dependent variable variable

ParametricTwo valued Normal Student’s t test on difference of meansDiscrete Normal ANOVA (ANalysis Of VAriance)Continuous Normal Linear (or non-linear) regression factor analysis

Non-parametricTwo valued Continuous Wilcoxon (or Mann–Whitney) rank-sum testDiscrete Continuous Rank-sum versions of ANOVAContinuous Continuous Spearman’s rank correlation

Contingency testsTwo valued Discrete No special test, see next entryDiscrete Discrete Contingency table and chi-squared testContinuous Discrete (Rare) Group independent variable and then as above


An example: evaluating icon designs

Imagine you are designing a new interface to a document-processing package, whichis to use icons for presentation. You are considering two styles of icon design and youwish to know which design will be easier for users to remember. One set of icons usesnaturalistic images (based on a paper document metaphor), the other uses abstractimages (see Figure 9.3). How might you design an experiment to help you decidewhich style to use?

The first thing you need to do is form a hypothesis: what do you consider to be thelikely outcome? In this case, you might expect the natural icons to be easier to recallsince they are more familiar to users. We can therefore form the following hypothesis:

Example of non-parametric statistics

We will not see an example of the use of non-parametric statistics later, so we will gothrough a small example here. Imagine we had the following data for response times under twoconditions:

condition A: 33, 42, 25, 79, 52condition B: 87, 65, 92, 93, 91, 55

We gather the data together and sort them into order: 25, 33, 42, . . . , 92, 93. We then substitutefor each value its rank in the list: 25 becomes 1, 33 becomes 2, etc. The transformed data are then

condition A: 2, 3, 1, 7, 4condition B: 8, 6, 10, 11, 9, 5

Tests are then carried out on the data. For example, to test whether there is any differencebetween the two conditions we can use the Wilcoxon test. To do this, we take each condition andcalculate the sum of ranks, and subtract the least value it could have (that is, 1 + 2 + 3 + 4 + 5 = 15for condition A, 1 + 2 + 3 + 4 + 5 + 6 = 21 for condition B), giving the statistic U:

rank sum least Ucondition A: (2 + 3 + 1 + 7 + 4) − 15 = 2condition B: (8 + 6 + 10 + 11 + 9 + 5) − 21 = 28

In fact, the sum of these two U statistics, 2 + 28 = 30, is the product of the number of data valuesin each condition 5 × 6. This will always happen and so one can always get away with calculatingonly one of the U. Finally, we then take the smaller of two U values and compare it with a set ofcritical values in a book of statistical tables, to see if it is unusually small. The table is laid out depend-ent on the number of data values in each condition (five and six). The critical value at the 5% levelturns out to be 3. As the smallest statistic is smaller than this, we can reject the null hypothesis andconclude that there is likely to be a difference between the conditions. To be precise, it says thatthere is only a 1 in 20 (5%) chance that the data happened by chance. In fact the test is right – theauthors constructed random data in the range 1–100 and then subtracted 10 from each of the values in condition A.


Users will remember the natural icons more easily than the abstract ones.

The null hypothesis in this case is that there will be no difference between recall ofthe icon types.

This hypothesis clearly identifies the independent variable for our experiment: weare varying the style of icon. The independent variable has two levels: natural andabstract. However, when we come to consider the dependent variable, things are notso obvious. We have expressed our hypothesis in terms of users being able to remem-ber more easily. How can we measure this? First we need to clarify exactly what wemean by the phrase more easily: are we concerned with the user’s performance interms of accurate recall or in terms of speed, for example, or are we looking at moresubjective measures like user preference? In this example, we will assume that thespeed at which a user can accurately select an icon is an indication of how easily it is remembered. Our dependent variables are therefore the number of mistakes inselection and the time taken to select an icon.

Of course, we need to control the experiment so that any differences we observeare clearly attributable to the independent variable, and so that our measurements ofthe dependent variables are comparable. To do this, we provide an interface that is identical in every way except for the icon design, and a selection task that can berepeated for each condition. The latter could be either a naturalistic task (such as producing a document) or a more artificial task in which the user has to select the appropriate icon to a given prompt. The second task has the advantage that it is more controlled (there is little variation between users as to how they will performthe task) and it can be varied to avoid transfer of learning. Before performing theselection task, the users will be allowed to learn the icons in controlled conditions:for example, they may be given a fixed amount of time to learn the icon meanings.

The next stage is to decide upon an experimental method. This may depend on theparticipants that are available, but in this case we will assume that we have sufficientparticipants from the intended user group. A between-subjects experiment wouldremove any learning effect for individual participants, but it would be more difficult

Figure 9.3 Abstract and concrete icons for file operations


to control for variation in learning style between participants. On balance, therefore,a within-subjects design is preferred, with order of presentation controlled.

So all that remains is to finalize the details of our experiment, given the constraintsimposed by these choices. We devise two interfaces composed of blocks of icons, onefor each condition. The user is presented with a task (say ‘delete a document’) and isrequired to select the appropriate icon. The selection task comprises a set of such pre-sentations. In order to avoid learning effects from icon position, the placing of iconsin the block can be randomly varied on each presentation. Each user performs the selec-tion task under each condition. In order to avoid transfer of learning, the users aredivided into two groups with each group taking a different starting condition. For eachuser, we measure the time taken to complete the task and the number of errors made.

Finally, we must analyze our results. Table 9.2 shows a possible set of results for ten participants.1 The first five had the abstract icons presented first (order AN), andthe last five had the natural icons presented first (order NA). Columns (1) and (2) inthe table show the completion times for the task using natural and abstract iconsrespectively. As the times are the result of lots of presentations, we will assume thatthey are normally distributed. The main independent variable, the icon type, is twovalued, suggesting we can use a simple difference of means with Student’s t test(Table 9.1). In fact, because we have used a within-subjects design, there is anotherindependent variable we have to take into account – the participant. This means we

Table 9.2 Example experimental results – completion times

(1) (2) (3) (4) (5)Participant Presentation Natural Abstract Participant Natural Abstractnumber order (s) (s) mean (1)–(3) (2)–(3)

1 AN 656 702 679 −23 232 AN 259 339 299 −40 403 AN 612 658 635 −23 234 AN 609 645 627 −18 185 AN 1049 1129 1089 −40 406 NA 1135 1179 1157 −22 227 NA 542 604 573 −31 318 NA 495 551 523 −28 289 NA 905 893 899 6 −6

10 NA 715 803 759 −44 44mean (μ) 698 750 724 −26 26s.d. (σ) 265 259 262 14 14

s.e.d. 117 s.e. 4.55Student’s t 0.32 (n.s.) 5.78 (p<1%, two tailed)

1 Note that these are fabricated results for the purposes of exposition and this is a rather small sample setfor real purposes.


have more than one discrete independent variable, and referring again to Table 9.1,we see that this implies we should use analysis of variance (ANOVA). A full analysisof variance is quite complex, and is ideally done with the aid of a statistics package.However, this experiment is particularly simple, so we can use a simplified analysis.

Look at columns (2) and (3) of Table 9.2. The completion times range from lessthan 5 minutes (participant 2) to nearly 20 minutes (participant 6), showing a widevariation between individuals. This wide variation emphasizes the importance of thewithin-subjects design. To see how this affects the results, we will first try to analyzethem ignoring the fact that each participant performed under each condition. At the end of the table, the mean and standard deviation have been calculated for eachcondition. These means can then be compared using Student’s t test. The differencebetween the means is 52 seconds, but the standard error of the difference (s.e.d.) is 117. This is calculated as follows:

where σN and σA are the standard deviations (s.d.) of the two conditions, and nN

and nA are the number of data items in each condition (10 in each). The s.e.d. is a measure of the expected variability of the difference between the means, and as we see the actual difference is well within this random variation. Testing the ratio52/117 against tables of Student’s t distribution indeed shows that this is notsignificant.

However, if we glance down the table, we see that in almost every case the timetaken with the abstract icons is greater than the time taken for the natural icons. Thatis, the data seem to support our claim that natural icons are better than abstract ones,but the wide variation between individuals has hidden the effect.

A more sophisticated analysis, a special case of ANOVA, can expose the difference.Looking back at the table, column (3) shows, for each participant, the average of the time they took under the two conditions. This participant mean is then sub-tracted from the data for each condition, yielding columns (4) and (5). Thesecolumns show the effect of the icon design once the differences between participantshave been removed. The two columns are redundant as they always add up to zero.They show that in all but one case (participant 9) the natural icons are faster than theabstract ones.

Even a non-parametric test would show this as a significant difference at the 5%level, but the use of a t test is more precise. We can take either column and see that the column average 26 is much greater than the standard error (14.4/ ). Theratio (mean/s.e.) is compared with the Student’s t table (in statistical tables) usingnine degrees of freedom (10 values minus 1 for the mean), and is indeed far greaterthan the 1% level (3.250); that is, the likelihood of getting our results by chance isless than 1 in 100. So, we reject the null hypothesis that there is no difference andconclude that natural icons are more easily remembered than abstract ones.

In fact, the last statement is not quite correct. What we have shown is that in this experiment natural icons are more rapidly remembered. Possibly, if we go on to

10

s.e.d. N

N

A

A

.= + = + =σ σ2 2 2 2265

10

259

10117 2

n n


analyze the errors, these may present a different story. If these error figures werequite large (say 15 errors or more per condition), then we may be able to assumethese are normal and use ANOVA. If not, we can either use non-parametric tests, ormake use of special tests based on the binomial distribution. We will not performthese analyses here. Possibly, looking at the errors we may find that the natural icons have more errors – it could well be that they are more rapidly, but less accurately,remembered. It is always worth keeping in mind the difference between the intendedpurpose of the experiment (to see which is better remembered) and the actual meas-urements (speed and accuracy).

Finally, one ought to look carefully at the experimental results to see whether there is any other effect that might confuse the results. The graphical presentation of results will help with this, possibly highlighting odd clumps in the data or otherirregularities. In this experiment we may want to check to see if there has been anysignificant transfer effect between the first and second condition for each participant.The second set may be faster as the participants are more practiced, or possibly thesecond set may be slower as learning a second set of icons may be confusing. This willnot matter if the effect is uniform – say they always are 15 seconds slower on the sec-ond test. But there may be systematic effects. For example, seeing the natural iconsfirst might make it more difficult to learn the abstract ones, but not vice versa. If thiswere the case, our observed effect may be about the interference between the iconsets, rather than that one is better than the other.

Worked exercise Design an experiment to test whether adding color coding to an interface will improve accur-acy. Identify your hypothesis, participant group, dependent and independent variables, experi-mental design, task and analysis approach.

Answer The following is only an example of the type of experiment that might be devised.

Participants Taken from user population.

Hypothesis Color coding will make selection more accurate.

IV (Independent Variable) Color coding.

DV (Dependent Variable) Accuracy measured as number of errors.

Design Between-groups to ensure no transfer of learning (or within-groups withappropriate safeguards if participants are scarce).

Task The interfaces are identical in each of the conditions, except that, in the second,color is added to indicate related menu items. Participants are presented with ascreen of menu choices (ordered randomly) and verbally told what they have toselect. Selection must be done within a strict time limit when the screen clears.Failure to select the correct item is deemed an error. Each presentation places itemsin new positions. Participants perform in one of the two conditions.

Analysis t test.


Studies of groups of users

So far we have considered the experimental evaluation of single-user systems.Experiments to evaluate elements of group systems bring additional problems. Giventhe complexities of human–human communication and group working, it is hardlysurprising that experimental studies of groups and of groupware are more difficultthan the corresponding single-user experiments already considered. For the purposeof discussion, let us assume that we are evaluating a shared application with videoconnections between the participants and consider some of the problems we willencounter.

The participant groups To organize, say, 10 experiments of a single-user systemrequires 10 participants. For an experiment involving groups of three, we will, ofcourse, need 30 participants for the same number of experiments. In addition, experi-ments in group working are often longer than the single-user equivalents as we mustallow time for the group to ‘settle down’ and some rapport to develop. This all meansmore disruption for participants and possibly more expense payments.

Arranging a mutually convenient slot when both participants and the equipmentare available is no mean feat. Often the workstations being used in the experimentwill be colleagues’ personal systems, so we are trying to accommodate at least sixpeople, not to mention the experimenters themselves.

Not surprisingly, many reports of group working involve only three or four groups.This is obviously a problem for statistical purposes, but not the primary obstacle.

The experimental task Choosing a suitable task is also difficult. We may want to test a variety of different task types: creative, structured, information passing, and so on. Also, the tasks must encourage active cooperation, either because the task requires consensus, or because information and control is distributed amongthe participants. Obviously, the task also depends on the nature of the groupware system: if it has several available channels, we want to encourage broad use. Forexample, in the case of shared application with video, it should not be possible (or atleast not easy) to perform the task without using the application, otherwise we aresimply investigating video conferencing.

Creative tasks such as ‘write a short report on . . .’ or ‘write a research proposal’are often effective, in that the participants must reach agreement, and can be askedto produce their final report using the shared application. Design tasks are also used.For instance, in one experiment, users of the York Conferencer system (see Fig-ure 14.2 in Section 14.4) were asked to redesign a bank layout. A picture of the current layout was used as a background for the spatially arranged electronic pin-board, and the participants made use of this to arrange comments and suggestionsclose to the features they referred to.

Decision games, as used in management courses, are designed to test and traincooperative activity. They often rely for their success on group coordination, notindividual ability. An example of this is the desert survival task, where the particip-ants are told that they have crashed in the desert. They are given a list of items to rank


in order of importance for their survival: knife, plastic sheet, etc. The participantsmust produce one list between them, a single knowledgeable participant cannot ‘goit alone’. A computerized version of the game of Diplomacy has also been used (seeFigure 14.5 in Section 14.4) as it includes aspects of conflict as well as cooperation.

Finally, time-critical simulated process control tasks force a higher pace of inter-action as the participants control different parts of the model. An example of this is ARKola [147], a simulated bottling plant, which was used at Xerox PARC to invest-igate the importance of background noise in complex cooperative control tasks.

Often the chosen task will require extra implementation effort, and in the case of games this may be extensive. This is obviously a strong factor in the choice of asuitable task.

Data gathering Even in a single-user experiment we may well use several videocameras as well as direct logging of the application. In a group setting this is replic-ated for each participant. So for a three-person group, we are trying to synchronizethe recording of six or more video sources and three keystroke logs. To compoundmatters, these may be spread over different offices, or even different sites. The tech-nical problems are clearly enormous. Four-into-one video recording is possible,storing a different image in each quadrant of the screen, but even this is insufficientfor the number of channels we would like.

One way round this is to focus on the participants individually, recording, for eachone, the video images that are being relayed as part of the system (assuming there is a video connection) and the sounds that the participant hears. These can then be synchronized with the particular participant’s keystrokes and additional videoobservations. Thus, we can recreate the situation as it appeared to the participant.From this recording, we may not be able to interpret the other participants’ actions,but at least we have a complete record for one.

Given sufficient recording equipment, this can be repeated for each participant.Happily, the level of synchronization required between participants is not as great asthat required for each one individually. One can simply start the recorders’ clocks atthe same time, but not worry about sub-second accuracy between participants. Theimportant thing is that we can, as it were, relive the experience for each individual.

Analysis In true experimental tradition, we would like to see statistical differencesbetween experimental conditions. We saw earlier that individual differences madethis difficult in single-user experiments. If anything, group variation is moreextreme. Given randomly mixed groups, one group will act in a democratic fashion;in another, a particular pair will dominate discussion; in a third, one of the parti-cipants will act as coordinator, filtering the others’ contributions. The level of variation is such that even catastrophic failures under one condition and fabuloussuccesses in another may not always lead to statistically significant results.

As an example of this, imagine we have some quantitative measure of quality ofoutput. We will almost certainly have to use non-parametric tests, so imagine wehave found that all the groups under one condition obtained higher scores than anygroup under the other condition. We would need at least four in each condition to


obtain even 5% significance (one tailed). If our results were only slightly less good,say one of the generally better groups performed poorly, we would then require atleast five in each condition.

Now this example only considered one condition, and assumed the best possibleresults. In general, we would expect that the spread between groups within condi-tions would be greater, and we may want to test more conditions at once. Our 10groups will have to increase rapidly to stand any chance of statistically significantresults. However, we saw above that even gathering 10 experimental groups is asignificant problem.

There are three possible solutions to this problem. First, one can use within-groupexperiments, having each group work under several conditions. We have, of course,the normal problems of such analysis, transfer effects and the like, but we also havemore chance of cancelling out the group effect. Secondly, we can look to a micro-analysis of features like gaps between utterances. Such measures are more likely to fit a standard distribution, and thus one can use more powerful parametric tests. Inaddition, they may be more robust to the large-scale social differences between groups.

The third solution is to opt for a more anecdotal analysis, looking for critical incidents – for example, interesting events or breakdowns – in the data. The conceptsand methods for analyzing conversation in Chapter 14 can be used to drive such an analysis. The advantage of this approach is that instead of regarding group differ-ences as a ‘problem’, they can be included in the analysis. That is, we can begin to look for the systematic ways in which different group structures interact with thecommunications media and applications they use.

Of course, experiments can be analyzed using both quantitative and qualitativemethods. Indeed, any detailed anecdotal analysis of the logs will indicate fruitfulmeasures for statistical analysis. However, if the number of experimental groups islimited, attempts at controlled experiments may not be productive, and may effect-ively ‘waste’ the groups used in the control. Given the high costs of group-workingexperiments, one must choose conditions that are likely to give interesting results,even if statistical analysis proves impossible.

Field studies with groups There are, of course, problems with taking groups ofusers and putting them in an experimental situation. If the groups are randomlymixed, then we are effectively examining the process of group formation, rather thanthat of a normal working group. Even where a pre-existent group is used, excludingpeople from their normal working environment can completely alter their workingpatterns. For a new system, there may be no ‘normal’ workplace and all we can do isproduce an artificial environment. However, even with a new system we have thechoice of producing a ‘good’ experiment or a naturalistic setting. The traditions of experimental psychology are at odds with those of more qualitative sociologicalanalysis.

It can be argued that group work can only be studied in context. Moving out ofthe real situation will alter the very nature of the work that is studied. Alternativeapproaches from the social sciences, such as ethnography, have therefore becomepopular, particularly in relation to studying group interaction. Ethnography involves


very detailed recording of the interactions between people, their environment and each other. The ethnographer attempts to remain outside the situation beingstudied and does not impose a particular viewpoint on what is observed. This is verydifferent from the experimental perspective with its hypothesis testing. Ethnographyis discussed in more detail in Chapter 13.

9.4.3 Observational techniques

A popular way to gather information about actual use of a system is to observe usersinteracting with it. Usually they are asked to complete a set of predetermined tasks,although, if observation is being carried out in their place of work, they may beobserved going about their normal duties. The evaluator watches and records theusers’ actions (using a variety of techniques – see below). Simple observation is seldom sufficient to determine how well the system meets the users’ requirementssince it does not always give insight into the their decision processes or attitude.Consequently users are asked to elaborate their actions by ‘thinking aloud’. In thissection we consider some of the techniques used to evaluate systems by observinguser behavior.

Think aloud and cooperative evaluation

Think aloud is a form of observation where the user is asked to talk through what he is doing as he is being observed; for example, describing what he believes is happening, why he takes an action, what he is trying to do.

Think aloud has the advantage of simplicity; it requires little expertise to perform(though can be tricky to analyze fully) and can provide useful insight into problemswith an interface. It can also be employed to observe how the system is actually used.It can be used for evaluation throughout the design process, using paper or simu-lated mock-ups for the earlier stages. However, the information provided is oftensubjective and may be selective, depending on the tasks provided. The process ofobservation can alter the way that people perform tasks and so provide a biased view.The very act of describing what you are doing often changes the way you do it – likethe joke about the centipede who was asked how he walked . . .

A variation on think aloud is known as cooperative evaluation [240] in which theuser is encouraged to see himself as a collaborator in the evaluation and not simplyas an experimental participant. As well as asking the user to think aloud at the begin-ning of the session, the evaluator can ask the user questions (typically of the ‘why?’or ‘what-if ?’ type) if his behavior is unclear, and the user can ask the evaluator forclarification if a problem arises. This more relaxed view of the think aloud processhas a number of advantages:

n the process is less constrained and therefore easier to learn to use by the evaluatorn the user is encouraged to criticize the systemn the evaluator can clarify points of confusion at the time they occur and so maxim-

ize the effectiveness of the approach for identifying problem areas.


The usefulness of think aloud, cooperative evaluation and observation in generalis largely dependent on the effectiveness of the recording method and subsequentanalysis. The record of an evaluation session of this type is known as a protocol, andthere are a number of methods from which to choose.

Protocol analysis

Methods for recording user actions include the following:

Paper and pencil This is primitive, but cheap, and allows the analyst to note inter-pretations and extraneous events as they occur. However, it is hard to get detailedinformation, as it is limited by the analyst’s writing speed. Coding schemes forfrequent activities, developed during preliminary studies, can improve the rate of recording substantially, but can take some time to develop. A variation of paper and pencil is the use of a notebook computer for direct entry, but then oneis limited to the analyst’s typing speed, and one loses the flexibility of paper for writing styles, quick diagrams and spatial layout. If this is the only record-ing facility available then a specific note-taker, separate from the evaluator, is recommended.

Audio recording This is useful if the user is actively ‘thinking aloud’. However, itmay be difficult to record sufficient information to identify exact actions in lateranalysis, and it can be difficult to match an audio recording to some other formof protocol (such as a handwritten script).

Video recording This has the advantage that we can see what the participant isdoing (as long as the participant stays within the range of the camera). Choosingsuitable camera positions and viewing angles so that you get sufficient detail and yet keep the participant in view is difficult. Alternatively, one has to ask theparticipant not to move, which may not be appropriate for studying normalbehavior! For single-user computer-based tasks, one typically uses two videocameras, one looking at the computer screen and one with a wider focus includ-ing the user’s face and hands. The former camera may not be necessary if thecomputer system is being logged.

Computer logging It is relatively easy to get a system automatically to record useractions at a keystroke level, particularly if this facility has been considered early inthe design. It can be more difficult with proprietary software where source codeis not available (although some software now provides built-in logging and play-back facilities). Obviously, computer logging only tells us what the user is doingon the system, but this may be sufficient for some purposes. Keystroke data arealso ‘semantics free’ in that they only tell us about the lowest-level actions, notwhy they were performed or how they are structured (although slight pauses andgaps can give clues). Direct logging has the advantages that it is cheap (except in terms of disk storage), unobtrusive and can be used for longitudinal studies,where we look at one or more users over periods of weeks or months. Technical


problems with it are that the sheer volume of data can become unmanageablewithout automatic analysis, and that one often has to be careful to restore thestate of the system (file contents, etc.) before replaying the logs.

User notebooks The participants themselves can be asked to keep logs of activity/problems. This will obviously be at a very coarse level – at most, records every few minutes and, more likely, hourly or less. It also gives us ‘interpreted’ records,which have advantages and problems. The technique is especially useful in longit-udinal studies, and also where we want a log of unusual or infrequent tasks andproblems.

In practice, one uses a mixture of recording methods as they complement oneanother. For instance, we may keep a paper note of special events and circum-stances, even when we have more sophisticated audio/visual recording. Similarly, we may use separate audio recording, even where a video recorder is used, as thequality of specialist audio recording is better than most built-in video micro-phones. In addition, we may use stereo audio recording, which helps us to locateout-of-screen noises. If one is using a collection of different sources, say audio, video (×2) and keystroke logging, there is considerable difficulty in synchronizingthem during play-back. Most video recorders can superimpose an on-screen clock, which can help, but ideally one uses specialized equipment that can automaticallysynchronize the different sources, possibly merging several video displays onto a single screen. Unfortunately, this sort of equipment is often only available in spe-cialized laboratories.

With both audio and video recording, a major problem is transcription. Typing a transcript from a tape is not the same as taped dictation. The conversation will typically consist of part or broken sentences, mumbled words and inarticulatednoises. In addition, the transcript will need annotating with the different voices(which may only be clear from context) and with non-verbal items such as pauses, emphases, equipment noises, phones ringing, etc. A good audio-typist will be accustomed to completing mumbled words and correcting ungrammatical sentences – typing exactly what is recorded may prove difficult. Some practitionerssay that the use of typists is not good practice anyway as the analyst will miss many nuances that are lost in the written transcript. However, if you wish to pro-duce your own typed transcripts from tape, a course in touch-typing is highly recommended.

For video transcription, professional typists are not an option; there is no standardway of annotating video recordings, and the analyst must invent notations to suit theparticular circumstances. The scale of this task is not to be underestimated. It is com-mon to talk to practitioners who have tens or hundreds of hours of video recording,but have only analyzed tiny fragments in detail. Of course, the fragments will havebeen chosen after more extensive perusal of the material, but it certainly removes anyidea of comprehensive coverage.

Coding can be introduced to indicate particular events but it is sometimes difficultto determine a suitable coding scheme and to use this consistently, particularly if


more than one person is doing the coding. A range of transcribers should thereforetest coding schemes to ensure that they are being interpreted appropriately for a particular data set.

Automatic protocol analysis tools

Analyzing protocols, whether video, audio or system logs, is time consuming andtedious by hand. It is made harder if there is more than one stream of data to synchronize. One solution to this problem is to provide automatic analysis tools tosupport the task. These offer a means of editing and annotating video, audio and system logs and synchronizing these for detailed analysis.

EVA (Experimental Video Annotator) is a system that runs on a multimedia work-station with a direct link to a video recorder [220]. The evaluator can devise a set ofbuttons indicating different events. These may include timestamps and snapshots, aswell as notes of expected events and errors. The buttons are used within a recordingsession by the evaluator to annotate the video with notes. During the session the userworks at a workstation and is recorded, using video and perhaps audio and systemlogging as well. The evaluator uses the multimedia workstation running EVA. On thescreen is the live video record and a view of the user’s screen (see Figure 9.4). Theevaluator can use the buttons to tag interesting events as they occur and can recordadditional notes using a text editor. After the session, the evaluator can ask to reviewthe tagged segments and can then use these and standard video controls to search the information. Links can be made with other types of record such as audio and system logs. A system such as EVA alleviates the burden of video analysis but it is not without its problems. The act of tagging and annotating events can prevent theevaluator from actually concentrating on the events themselves. This may mean thatevents are missed or tagged late.

Commercial systems such as Observer Pro from Noldus have similar functionalityto EVA; portable versions are now available for use in field studies (www.noldus.com).

Figure 9.4 EVA: an automatic protocol analysis tool. Source: Wendy Mackay


The Workplace project at Xerox PARC [348] also includes a system to aid pro-tocol analysis. The main emphasis here is to support the analysis of synchronizedinformation from different data streams, such as video, audio, notes and dia-grams. Each data stream is viewed in an aligned display so that it is possible to compare the records of each for a given point in the interaction. The alignment may be based on timestamps or on an event or action and is implemented usinghypertext links.

A third example is DRUM [223], which also provides video annotation and tag-ging facilities. DRUM is part of the MUSiC (Measuring the Usability of Systems inContext/Metrics for Usability Standards in Computing) toolkit, which supports a complete methodology for evaluation, based upon the application of usability metrics on analytic metrics, cognitive workload, performance and user satisfaction.DRUM is concerned particularly with measuring performance. The methodologyprovides a range of tools as well as DRUM, including manuals, questionnaires, analysis software and databases.

Systems such as these are extremely important as evaluation tools since they offera means of handling the data that are collected in observational studies and allowinga more systematic approach to the analysis. The evaluator’s task is facilitated and itis likely that more valuable observations will emerge as a result.

Post-task walkthroughs

Often data obtained via direct observation lack interpretation. We have the basicactions that were performed, but little knowledge as to why. Even where the particip-ant has been encouraged to think aloud through the task, the information may be atthe wrong level. For example, the participant may say ‘and now I’m selecting theundo menu’, but not tell us what was wrong to make undo necessary. In addition, a think aloud does not include information such as alternative, but not pursued,actions.

A walkthrough attempts to alleviate these problems, by reflecting the participants’actions back to them after the event. The transcript, whether written or recorded, is replayed to the participant who is invited to comment, or is directly questioned by the analyst. This may be done straightaway, when the participant may actuallyremember why certain actions were performed, or after an interval, when theanswers are more likely to be the participant’s post hoc interpretation. (In fact, inter-pretation is likely even in the former case.) The advantage of a delayed walkthroughis that the analyst has had time to frame suitable questions and focus on specific incidents. The disadvantage is a loss of freshness.

There are some circumstances when the participant cannot be expected to talkduring the actual observation, for instance during a critical task, or when the task is too intensive. In these circumstances, the post-task walkthrough is the only way to obtain a subjective viewpoint on the user’s behavior. There is also an argumentthat it is preferable to minimize non-task-related talk during direct observation inorder to get as natural a performance as possible. Again this makes the walkthroughessential.


9.4.4 Query techniques

Another set of evaluation techniques relies on asking the user about the interfacedirectly. Query techniques can be useful in eliciting detail of the user’s view of a system. They embody the philosophy that states that the best way to find out how asystem meets user requirements is to ‘ask the user’. They can be used in evaluationand more widely to collect information about user requirements and tasks. Theadvantage of such methods is that they get the user’s viewpoint directly and mayreveal issues that have not been considered by the designer. In addition, they are relatively simple and cheap to administer. However, the information gained is neces-sarily subjective, and may be a ‘rationalized’ account of events rather than a whollyaccurate one. Also, it may be difficult to get accurate feedback about alternativedesigns if the user has not experienced them, which limits the scope of the informa-tion that can be gleaned. However, the methods provide useful supplementary mater-ial to other methods. There are two main types of query technique: interviews andquestionnaires.

Interviews

Interviewing users about their experience with an interactive system provides adirect and structured way of gathering information. Interviews have the advantagesthat the level of questioning can be varied to suit the context and that the evaluatorcan probe the user more deeply on interesting issues as they arise. An interview willusually follow a top-down approach, starting with a general question about a taskand progressing to more leading questions (often of the form ‘why?’ or ‘what if ?’) toelaborate aspects of the user’s response.

Interviews can be effective for high-level evaluation, particularly in eliciting information about user preferences, impressions and attitudes. They may also revealproblems that have not been anticipated by the designer or that have not occurredunder observation. When used in conjunction with observation they are a usefulmeans of clarifying an event (compare the post-task walkthrough).

In order to be as effective as possible, the interview should be planned in advance,with a set of central questions prepared. Each interview is then structured aroundthese questions. This helps to focus the purpose of the interview, which may, forinstance, be to probe a particular aspect of the interaction. It also helps to ensure abase of consistency between the interviews of different users. That said, the evalu-ator may, of course, choose to adapt the interview form to each user in order to get the most benefit: the interview is not intended to be a controlled experimentaltechnique.

Questionnaires

An alternative method of querying the user is to administer a questionnaire. This isclearly less flexible than the interview technique, since questions are fixed in advance,


and it is likely that the questions will be less probing. However, it can be used toreach a wider participant group, it takes less time to administer, and it can be ana-lyzed more rigorously. It can also be administered at various points in the designprocess, including during requirements capture, task analysis and evaluation, inorder to get information on the user’s needs, preferences and experience.

Given that the evaluator is not likely to be directly involved in the completion ofthe questionnaire, it is vital that it is well designed. The first thing that the evaluatormust establish is the purpose of the questionnaire: what information is sought? It isalso useful to decide at this stage how the questionnaire responses are to be analyzed.For example, do you want specific, measurable feedback on particular interface fea-tures, or do you want the user’s impression of using the interface?

There are a number of styles of question that can be included in the questionnaire.These include the following:

General These are questions that help to establish the background of the user and his place within the user population. They include questions about age, sex,occupation, place of residence, and so on. They may also include questions onprevious experience with computers, which may be phrased as open-ended,multi-choice or scalar questions (see below).

Open-ended These ask the user to provide his own unprompted opinion on aquestion, for example ‘Can you suggest any improvements to the interface?’. Theyare useful for gathering general subjective information but are difficult to analyzein any rigorous way, or to compare, and can only be viewed as supplementary.They are also most likely to be missed out by time-conscious respondents!However, they may identify errors or make suggestions that have not been con-sidered by the designer. A special case of this type is where the user is asked forfactual information, for example how many commands were used.

Scalar These ask the user to judge a specific statement on a numeric scale, usuallycorresponding to a measure of agreement or disagreement with the statement.For example,

It is easy to recover from mistakes.Disagree 1 2 3 4 5 Agree

The granularity of the scale varies: a coarse scale (say, from 1 to 3) gives a clearindication of the meaning of the numbers (disagree, neutral and agree). However,it gives no room for varying levels of agreement, and users may therefore betempted to give neutral responses to statements that they do not feel stronglyabout but with which they mildly disagree or agree. A very fine scale (say 1 to 10)suffers from the opposite problem: the numbers become difficult to interpret in a consistent way. One user will undoubtedly interpret the scale differently from another. A middle ground is therefore advisable. Scales of 1 to 5 or 1 to 7have been used effectively. They are fine enough to allow users to differentiateadequately but still retain clarity in meaning. It can help to provide an indication


of the meaning of intermediate scalar values. Odd-numbered scales are used mostoften but it is possible to use even-numbered scales (e.g. 1–6) if the ‘neutral’option is not wanted. This does not allow for fence sitting – except decisively byselecting 31/2!).

Multi-choice Here the respondent is offered a choice of explicit responses, andmay be asked to select only one of these, or as many as apply. For example,

How do you most often get help with the system (tick one)?Online manual qContextual help system qCommand prompt qAsk a colleague q

Which types of software have you used (tick all that apply)?Word processor qDatabase qSpreadsheet qExpert system qOnline help system qCompiler q

These are particularly useful for gathering information on a user’s previous experi-ence. A special case of this type is where the offered choices are ‘yes’ or ‘no’.

Ranked These place an ordering on items in a list and are useful to indicate a user’spreferences. For example,

Please rank the usefulness of these methods of issuing a command (1 most useful,2 next, 0 if not used).

Menu selection qCommand line qControl key accelerator q

These question types are all useful for different purposes, as we have noted.However, in order to reduce the burden of effort on the respondent, and so encour-age a high response rate amongst users, it is best to use closed questions, such asscalar, ranked or multi-choice, as much as possible. These provide the user withalternative responses and so reduce the effort required. They also have the advantageof being easier to analyze. Responses can be analyzed in a number of ways, fromdetermining simple percentages for each response, to looking at correlations and factor analysis. For more detail on available methods the reader is referred to the recommended reading list at the end of the chapter.

Whatever type of questionnaire is planned, it is wise to carry out a pilot study. This allows any problems with the questionnaire design to be ironed out before thequestionnaire is distributed to potentially hundreds of users! The questionnaireshould be tested on four or five users to see if the questions are comprehensible andthe results are as expected and can be used in the manner intended. If users seem to


be misunderstanding a particular question, it can then be rephrased (and retested)before the final version is sent out.

Distribution of questionnaires can also be problematic. It is important that therespondents are representative of the user population but you also need to ensurethat you are able to reach as many potential respondents as possible. Return rate for questionnaires is quite low (often 25–30%) so many more need to be sent out to get a reasonable return. Questionnaires should ideally be distributed to a randomsubset of the user population. So, for example, if the population is all workers in a company, one may choose to send a questionnaire to every fourth person on analphabetically ordered personnel list. However, questionnaires are now often dis-tributed via the internet, either by email, where potential respondents can be selectedrandomly, or via a website, where the respondents are limited to those who visit the site and who may not be representative. In practice, questionnaire respondentsare self-selecting anyway, in that only those who choose to respond are included inthe study; if the questionnaire is designed to capture demographic informationabout each respondent then the level of representativeness (or otherwise) can bedetermined from the responses.

Worked exercise You have been asked to compare user performance and preferences with two different learn-ing systems, one using hypermedia (see Chapter 21), the other sequential lessons. Design aquestionnaire to find out what the users think of the system. How would you go about com-paring user performance with these two systems?

Answer Assume that all users have used both systems.

QuestionnaireConsider the following questions in designing the questionnaire:

n what information is required?n how is the questionnaire to be analyzed?

You are particularly interested in user preferences so questions should focus on differ-ent aspects of the systems and try to measure levels of satisfaction. The use of scaleswill make responses for each system easier to compare.

Table 9.3 shows an example questionnaire.

To test performance you would design an experiment where two groups of participantslearn the same material using the two systems, and test how well they have learned(using a standard measurable test).

Participants User group

IV (Independent Variable) Style of learning system

DV (Dependent Variable) Performance (measured as test score)

Design Between-subjects design


Table 9.3 Questionnaire to compare two systems

PART I: Repeat for each systemIndicate your agreement or disagreement with the following statements. (1 indicates completedisagreement and 5 complete agreement.)

The system tells me what to do at every point.Disagree 1 2 3 4 5 Agree

It is easy to recover from mistakes.Disagree 1 2 3 4 5 Agree

It is easy to get help when needed.Disagree 1 2 3 4 5 Agree

I always know what the system is doing.Disagree 1 2 3 4 5 Agree

I always know where I am in the training material.Disagree 1 2 3 4 5 Agree

I have learned the material well using the system.Disagree 1 2 3 4 5 Agree

I could have learned the material more effectively using a book.Disagree 1 2 3 4 5 Agree

I always know how well I am doing.Disagree 1 2 3 4 5 Agree

PART II: Comparing both systems:

Which system (choose 1) was most:Helpful to use A BEfficient to use A BEnjoyable to use A B

Please add any comments you have about either system:

9.4.5 Evaluation through monitoring physiological responses

One of the problems with most evaluation techniques is that we are reliant on obser-vation and the users telling us what they are doing and how they are feeling. What ifwe were able to measure these things directly? Interest has grown recently in the useof what is sometimes called objective usability testing, ways of monitoring physi-ological aspects of computer use. Potentially this will allow us not only to see moreclearly exactly what users do when they interact with computers, but also to measurehow they feel. The two areas receiving the most attention to date are eye tracking andphysiological measurement.


Eye tracking for usability evaluation

Eye tracking has been possible for many years, but recent improvements in hard-ware and software have made it more viable as an approach to measuring usability.The original eye trackers required highly invasive procedures where eye caps wereattached to the cornea under anaesthetic. Clearly inappropriate for usability testing!Modern systems vary: some use a head-mounted camera to monitor the eye, but the most sophisticated do not involve any contact between the equipment and the participant, with the camera and light sources mounted in desk units (see Figures 9.5, 9.6) [112].

Furthermore, there have been rapid improvements in the software available bothfor the control of eye-tracking equipment and the analysis and visualization of thelarge volumes of data it produces.

Eye movements are believed to reflect the amount of cognitive processing a dis-play requires and, therefore, how easy or difficult it is to process [150]. So measur-ing not only where people look, but also their patterns of eye movement, may tell uswhich areas of a screen they are finding easy or difficult to understand. Eye move-ment measurements are based on fixations, where the eye retains a stable position for a period of time, and saccades, where there is rapid ballistic eye movement fromone point of interest to another. There are many possible measurements related tousability evaluation including:

Number of fixations The more fixations the less efficient the search strategy.

Fixation duration Longer fixations may indicate difficulty with a display.

Figure 9.5 Eye-tracking equipment. Source: Courtesy of J. A. Renshaw


Scan path indicating areas of interest, search strategy and cognitive load. Movingstraight to a target with a short fixation at the target is the optimal scan path butplotting scan paths and fixations can indicate what people look at, how often andfor how long.

Eye tracking for usability is still very new and equipment is prohibitively expens-ive for everyday use. However, it is a promising technique for providing insights into what really attracts the eye in website design and where problem areas are in system use. More research is needed to interpret accurately the meaning of the vari-ous eye movement measurements, as well as to develop more accessible and robustequipment. But, given the potential for gathering new data measurements relativelyunobtrusively, it is likely that eye tracking will become part of the standard equip-ment for usability laboratories in the coming few years.

Figure 9.6 Calibrating the eye tracker. Source: Courtesy of J. A. Renshaw


Physiological measurements

As we saw in Chapter 1, emotional response is closely tied to physiological changes.These include changes in heart rate, breathing and skin secretions. Measuring thesephysiological responses may therefore be useful in determining a user’s emotionalresponse to an interface [288, 363]. Could we determine which interaction eventsreally cause a user stress or which promote relaxation?

Physiological measurement involves attaching various probes and sensors to theuser (see Figure 9.7). These measure a number of factors:

Figure 9.7 Data Lab Psychophysiology equipment showing some of the sensors(above) and a typical experimental arrangement (below) with sensors attached to theparticipant’s fingers and the monitoring software displayed on the evaluator’s machine.Source: Courtesy of Dr R. D. Ward


Heart activity, indicated by blood pressure, volume and pulse. These may respondto stress or anger.

Activity of the sweat glands, indicated by skin resistance or galvanic skin response(GSR). These are thought to indicate levels of arousal and mental effort.

Electrical activity in muscle, measured by the electromyogram (EMG). These appearto reflect involvement in a task.

Electrical activity in the brain, measured by the electroencephalogram (EEG). Theseare associated with decision making, attention and motivation.

Figure 9.8 illustrates the output obtained from such measurements.One of the problems with applying these measurements to interaction events is

that it is not clear what the relationship between these events and measurementsmight be. For example, if increased pulse rate is observed during an interactive task,does that indicate frustration with the interface or stress at being unable to completethe task? How will physiological changes differ in response to discrete events or tocontinuous interface use? Is it possible to map patterns of physiological measure-ment to specific emotional states?

These are still research questions but the approach is interesting, as it offers apotential means of objectively capturing information about the user’s emotional

Figure 9.8 Output of monitoring pulse rate (above) and skin conductivity (below).Source: Screen shot courtesy of Dr R. D. Ward; frame source: National InstrumentsBioBench software

9.5 Choosing an evaluation method 357

state, which, as we saw in Chapter 1, impacts on our ability to solve problems andperform tasks.

CHOOSING AN EVALUATION METHOD

As we have seen in this chapter, a range of techniques is available for evaluating aninteractive system at all stages in the design process. So how do we decide whichmethods are most appropriate for our needs? There are no hard and fast rules in this– each method has its particular strengths and weaknesses and each is useful ifapplied appropriately. However, there are a number of factors that should be takeninto account when selecting evaluation techniques. These also provide a way of cat-egorizing the different methods so that we can compare and choose between them.In this final section we will consider these factors.

9.5.1 Factors distinguishing evaluation techniques

We can identify at least eight factors that distinguish different evaluation techniquesand therefore help us to make an appropriate choice. These are:

n the stage in the cycle at which the evaluation is carried outn the style of evaluationn the level of subjectivity or objectivity of the techniquen the type of measures providedn the information providedn the immediacy of the responsen the level of interference impliedn the resources required.

Design vs. implementation

The first factor to affect our choice of evaluation method is the stage in the designprocess at which evaluation is required. As we saw earlier in this chapter, it is desir-able to include evaluation of some sort throughout the design process. The main distinction between evaluation of a design and evaluation of an implementation isthat in the latter case a physical artifact exists. This may be anything from a papermock-up to a full implementation, but it is something concrete that can be tested.Evaluation of a design, on the other hand, precedes this stage and seeks instead toprovide information to feed the development of the physical artifact.

Roughly speaking, evaluation at the design stage needs to be quick and cheap so might involve design experts only and be analytic, whereas evaluation of theimplementation needs to be more comprehensive, so brings in users as participants.There are, of course, exceptions to this: participatory design (see Chapter 13)

9.5


involves users throughout the design process, and techniques such as cognitive walk-through are expert based and analytic but can be used to evaluate implementationsas well as designs.

Early evaluation, whether of a design or an early prototype or mock-up, will bringthe greatest pay-off since problems can be easily resolved at this stage. As more commitment is made to a particular design in the implementation, it becomesincreasingly difficult for changes to be made, no matter what the evaluation suggests.Ironically, the most resources are often ploughed into late evaluations. This is lessprofitable and should be avoided, although obviously some evaluation with users isrequired with a complete, or almost complete, system, since some elements (such assystem performance) will only be evident in the working system.

Laboratory vs. field studies

We have already discussed the pros and cons of these two styles of evaluation.Laboratory studies allow controlled experimentation and observation while losingsomething of the naturalness of the user’s environment. Field studies retain the latter but do not allow control over user activity. Ideally the design process shouldinclude both styles of evaluation, probably with laboratory studies dominating theearly stages and field studies conducted with the new implementation.

Subjective vs. objective

Evaluation techniques also vary according to their objectivity – some techniques rely heavily on the interpretation of the evaluator, others would provide similarinformation for anyone correctly carrying out the procedure. The more subjectivetechniques, such as cognitive walkthrough or think aloud, rely to a large extent onthe knowledge and expertise of the evaluator, who must recognize problems andunderstand what the user is doing. They can be powerful if used correctly and willprovide information that may not be available from more objective methods.However, the problem of evaluator bias should be recognized and avoided. One way to decrease the possibility of bias is to use more than one evaluator. Objectivetechniques, on the other hand, should produce repeatable results, which are notdependent on the persuasion of the particular evaluator. Controlled experiments are an example of an objective measure. These avoid bias and provide comparableresults but may not reveal the unexpected problem or give detailed feedback on userexperience. Ideally, both objective and subjective approaches should be used.

The extent to which the results are dependent on the subjective response of the user also varies. At one extreme is asking for the user’s opinions, which is highlysubjective; at the other is measuring physiological changes in the body, which areoutside the user’s control.

Qualitative vs. quantitative measures

The type of measurement provided by the evaluation technique is also an importantconsideration. There are two main types: quantitative measurement and qualitative


measurement. The former is usually numeric and can be easily analyzed using statist-ical techniques. The latter is non-numeric and is therefore more difficult to analyze,but can provide important detail that cannot be determined from numbers. The typeof measure is related to the subjectivity or objectivity of the technique, with subject-ive techniques tending to provide qualitative measures and objective techniques,quantitative measures. This is not a hard and fast rule, however. It is sometimes possible to quantify what is, in fact, qualitative information by mapping it onto ascale or similar measure. A common example of this is in questionnaires where qual-itative information is being sought (for example, user preferences) but a quantitativescale is used. This is also common in experimental design where factors such as thequality of the user’s performance are used as dependent variables, and measured ona quantitative scale.

Information provided

The level of information required from an evaluation may also vary. The informa-tion required by an evaluator at any stage of the design process may range from low-level information to enable a design decision to be made (for example, whichfont is most readable) to higher-level information, such as ‘Is the system usable?’.Some evaluation techniques, such as controlled experiments, are excellent at provid-ing low-level information – an experiment can be designed to measure a particularaspect of the interface. Higher-level information can be gathered using questionnaireand interview techniques, which provide a more general impression of the user’sview of the system.

Immediacy of response

Another factor distinguishing evaluation techniques is the immediacy of theresponse they provide. Some methods, such as think aloud, record the user’s behavior at the time of the interaction itself. Others, such as post-task walkthrough,rely on the user’s recollection of events. Such recollection is liable to suffer from bias in recall and reconstruction, with users interpreting events according to their preconceptions. Recall may also be incomplete. However, immediate techniques canalso be problematic, since the process of measurement can actually alter the way theuser works.

Intrusiveness

Related to the immediacy of the response is the intrusiveness of the technique itself.Certain techniques, particularly those that produce immediate measurements, areobvious to the user during the interaction and therefore run the risk of influencingthe way the user behaves. Sensitive activity on the part of the evaluator can help toreduce this but cannot remove it altogether. Most immediate evaluation techniquesare intrusive, with the exception of automatic system logging. Unfortunately, this islimited in the information that it can provide and is difficult to interpret.


Resources

The final consideration when selecting an evaluation technique is the availability of resources. Resources to consider include equipment, time, money, participants,expertise of evaluator and context. Some decisions are forced by resource limita-tions: it is not possible to produce a video protocol without access to a video camera(and probably editing facilities as well). However, other decisions are not so clearcut. For example, time and money may be limited, forcing a choice between two pos-sible evaluations. In these circumstances, the evaluator must decide which evaluationtactic will produce the most effective and useful information for the system underconsideration. It may be possible to use results from other people’s experiments to avoid having to conduct new experiments.

Some techniques are more reliant on evaluator expertise than others, for examplethe more formal analytic techniques. If evaluator expertise is limited, it may be morepractical to use more simple heuristic methods than methods that require under-standing of user goal structures and so on.

Finally, the context in which evaluation can occur will influence what can be done.For practical reasons it may not be possible to gain access to the intended users of a system (if it is a general system, for example). Or it may not be feasible to test thesystem in its intended environment (for example, a system for a space station or adefence system). In these circumstances simulations must be used.

9.5.2 A classification of evaluation techniques

Using the factors discussed in the previous section we can classify the evaluationtechniques we have considered in this chapter. This allows us to identify the tech-niques that most closely fit our requirements. Table 9.4 shows the classification for

Table 9.4 Classification of analytic evaluation techniques

Cognitive Heuristic Review Modelwalkthrough evaluation based based

Stage Throughout Throughout Design DesignStyle Laboratory Laboratory Laboratory LaboratoryObjective? No No As source NoMeasure Qualitative Qualitative As source QualitativeInformation Low level High level As source Low levelImmediacy N/A N/A As source N/AIntrusive? No No No NoTime Medium Low Low–medium MediumEquipment Low Low Low LowExpertise High Medium Low High


analytic techniques, Table 9.5 for experimental and query techniques, Table 9.6 forobservational and Table 9.7 for monitoring techniques.

The classification is intended as a rough guide only – some of the techniques donot fit easily into such a classification since their use can vary considerably.

Table 9.5 Classification of experimental and query evaluation techniques

Experiment Interviews Questionnaire

Stage Throughout Throughout Throughout

Style Laboratory Lab/field Lab/field

Objective? Yes No No

Measure Quantitative Qualitative/ Qualitative/quantitative quantitative

Information Low/high level High level High level

Immediacy Yes No No

Intrusive? Yes No No

Time High Low Low

Equipment Medium Low Low

Expertise Medium Low Low

Table 9.6 Classification of observational evaluation techniques

Think aloud1 Protocol analysis2 Post-task walkthrough

Stage Implementation Implementation Implementation

Style Lab/field Lab/field Lab/field

Objective? No No No

Measure Qualitative Qualitative Qualitative

Information High/low level High/low level High/low level

Immediacy Yes Yes No

Intrusive? Yes Yes3 No

Time High High Medium

Equipment Low High Low

Expertise Medium High Medium

1 Assuming a simple paper and pencil record2 Including video, audio and system recording3 Except system logs


SUMMARY

Evaluation is an integral part of the design process and should take place through-out the design life cycle. Its aim is to test the functionality and usability of the designand to identify and rectify any problems. It can also try to determine the user’s attitude and response to the system. It can take place in a specialist laboratory or inthe user’s workplace, and may or may not involve active participation on the part ofthe user.

A design can be evaluated before any implementation work has started, to min-imize the cost of early design errors. Most techniques for evaluation at this stage are analytic and involve using an expert to assess the design against cognitive andusability principles. Previous experimental results and modeling approaches can alsoprovide insight at this stage. Once an artifact has been developed (whether a pro-totype or full system), experimental and observational techniques can be used to get both quantitative and qualitative results. Query techniques provide subjectiveinformation from the user. If more objective information is required, physiologicalmonitoring can capture the user’s physical responses to the system.

The choice of evaluation method is largely dependent on what is required of theevaluation. Evaluation methods vary in the stage at which they are commonly usedand where they can be used. Some are more subjective than others and provide qualitative rather than quantitative measures. Some provide immediate informationwhile others get feedback after the event. However, the more immediate methodsalso tend to intrude most seriously on the interaction. Finally, some require moreresources in terms of time, equipment and expertise than others.

9.6

Table 9.7 Classification of monitoring evaluation techniques

Eye tracking Physiological measurement

Stage Implementation ImplementationStyle Lab LabObjective? Yes YesMeasure Quantitative QuantitativeInformation Low level Low levelImmediacy Yes YesIntrusive? No1 YesTime Medium/high Medium/highEquipment High HighExpertise High High

1 If the equipment is not head mounted

Exercises 363

EXERCISES

9.1 In groups or pairs, use the cognitive walkthrough example, and what you know aboutuser psychology (see Chapter 1), to discuss the design of a computer application of your choice(for example, a word processor or a drawing package). (Hint: Focus your discussion on one ortwo specific tasks within the application.)

9.2 What are the benefits and problems of using video in experimentation? If you have access to avideo recorder, attempt to transcribe a piece of action and conversation (it does not have to bean experiment – a soap opera will do!). What problems did you encounter?

9.3 In Section 9.4.2, we saw that the observed results could be the result of interference. Can youthink of alternative designs that may make this less likely? Remember that individual variation wasvery high, so you must retain a within-subjects design, but you may perform more tests on eachparticipant.

9.4 Choose an appropriate evaluation method for each of the following situations. In each case identify:

(i) the participants(ii) the technique used(iii) representative tasks to be examined(iv) measurements that would be appropriate(v) an outline plan for carrying out the evaluation.

(a) You are at an early stage in the design of a spreadsheet package and you wish to test whattype of icons will be easiest to learn.

(b) You have a prototype for a theatre booking system to be used by potential theatre-goers toreduce queues at the box office.

(c) You have designed and implemented a new game system and want to evaluate it beforerelease.

(d) You have developed a group decision support system for a solicitor’s office.(e) You have been asked to develop a system to store and manage student exam results and

would like to test two different designs prior to implementation or prototyping.

9.5 Complete the cognitive walkthrough example for the video remote control design.

9.6 In defining an experimental study, describe:

(a) how you as an experimenter would formulate the hypothesis to be supported or refuted byyour study;

(b) how you would decide between a within-groups or between-groups experimental design withyour subjects.

9.7 What are the factors governing the choice of an appropriate evaluation method for different inter-active systems? Give brief details.

alan dix, janet finlay, gregory d. abowd, russell beale...

Documents