CMC/CC A
Usability Evaluation
Master IK, CIW, MMI
L.M. Bosveld-de Smet
Course 4; mon. 02/10/06; 16.00-18.00
Usability Evaluation
Design of interactionDix et al. (2004)
What is wanted
Analysis
Design
Implementand deploy
Prototype
Usability
Evaluation
Design process of usable interactive systems
Interaction design process Complex Iterative Never complete
User-centered design Within software engineering framework
Design rules Implementation support Evaluation techniques Universally accessible designs Provision of user support
3 related concepts
Usability: measure of success of a product User-Centered Design: design involving user
participation Usability Engineering:
Whole process ensuring usable interactive systems Process aiming at systems fit for the user, the task, the
environment Process implying user participation during software development Process committing itself to design-evaluate-redesign
development cycle In search for objective measures for user interface
Usability: different views
Same intuitions vs. Different approaches Different views on:
Definition of usability Usability attributes
Importance of user-centered design When and how to deal with user participation How to build usable systems How to measure usability
Usability Attributes: Shackel’s view “A usable product is one that users find
satisfactory for the tasks for which it was designed.”
“Good design for usability depends upon achieving successful harmony in the dynamic interplay between user, task, system, and environment.”
Usability: 4 usable criteria Learnability Effectiveness Attitude Flexibility
Usability Attributes: ISO definition
“Usability is the effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in particular environments.” Effectiveness: “accuracy and completeness with which
users achieve specific goals” Efficiency: “accuracy and completeness of goals in
relation to resources expended” Satisfaction: “comfort and acceptability of the system”
An aside: User Interface Standards
Disadvantages: Constrain design Stagnate innovation Describe principles, do not provide solutions Become quickly obsolete
Advantages: Define ‘good practice’ Affect attitudes w.r.t. ‘software development’
User-Centered Design: Karat’s view
Karat (1996) “UCD: Quality or Quackery?”
“UCD is an iterative process whose goal is the development of usable systems, achieved through involvement of potential users of a system in system design”
“I suggest we consider UCD a nice fluffy little catch phrase. It captures a commitment that the usability community supports – that you must involve users in system design – while leaving fairly open how this is accomplished”
UCD: a more objective view
UCD can be accomplished through the application of Usability Engineering
Design should centre on users End-user should be consulted Needs of end-users should be considered
HCI in software process
Software engineering Usability engineering Iterative design practices Design rationale
Waterfall model
Requirements Specification
Architectural design
DetailedDesign
Implementationand Unit testing
Integration andTesting
Operation andMaintenance
Usability Engineering
Usability specification as part of requirements specification
Iteration and testing List of usability measurement requirements
(Whiteside, Bennett and Holtzblatt, 1988) Time to complete a task Ratio of successes to failures Time spent in errors Number of commands used Frequency of help and documentation use …
Iteration in waterfall model
Requirements Specification
Architectural design
DetailedDesign
Implementationand Unit testing
Integration andTesting
Operation andMaintenance
Usability Engineering LifecycleFaulkner (2000)
Know the user Know the task User requirements capture Setting usability goals Design process Apply guidelines, heuristics Prototyping Evaluation with users Redesign and evaluation with users Evaluation with users and report
Role of prototyping
Design
Redesign
Prototype Evaluate Done!OK?
Not OK?
Iterative design and prototyping
Three main approaches to prototyping Throw-away prototyping within requirements
specification Incremental prototyping within the life cycle Evolutionary prototyping throughout the life cycle
Throw-away prototyping
PreliminaryRequirements
BuildPrototype
EvaluatePrototype
FinalRequirements
Adequate?yes
no
Evolutionary prototyping
Build Prototype
Evaluate Prototype
Operation andMaintenance
Req
Arch
Det
Impl
Int
Usability Evaluation (UE)
Methodologies for measuring usability aspects of system’s user interface and identifying specific problems (Dix et al. 1998; Nielsen 1993)
Should occur throughout design life cycle Common activities:
Capture Analysis Critic
There is a wide range of UE techniques Each technique has its own requirements Different techniques uncover different usability problems
Taxonomies of UE (1)
Formative vs. Summative Analytical vs. Empirical Expert analysis vs. User participation
Analytic methods Review methods Model-based methodsvs. Experimental methods Observational methods Query methods
Taxonomies of UE (2)
Automated vs. Non-automated Ivory and Hearst (2001):
Testing Inspection Inquiry Analytical modeling Simulation
Expert Analysis
Controlled Experiment
User participation
Important factors choice UE
Stage in cycle at which UE is carried out Style of UE Level of subjectivity or objectivity of UE
technique Type of measures provided Information provided Immediacy of response Level of interference implied Resources required
Analytic UE techniques
Cognitive Walkthrough
Heuristic Evaluation
Review based Model based
Stage throughout throughout design design
Style laboratory laboratory laboratory laboratory
Objective? no no as source as source
Measure qualitative qualitative as source qualitative
Information low level high level as source low level
Immediacy N/A N/A as source N/A
Intrusive? no no no no
Time medium low low-medium medium
Equipment low low low low
Expertise high medium low high
Experimental and query UE techniques
Experiment Interviews Questionnaire
Stage throughout throughout throughout
Style laboratory lab / field lab / field
Objective? yes no no
Measure quantitative qualitative / quantitative
qualitative / quantitative
Information low/high level high level high level
Immediacy yes no no
Intrusive? yes no no
Time high low low
Equipment medium low low
Expertise medium low low
Observational UE techniques
Think aloud Protocol analysis
Post-task walkthrough
Stage implementation implementation implementation
Style lab / field lab / field lab / field
Objective? no no no
Measure qualitative qualitative qualitative
Information high/low level high/low level high/low level
Immediacy yes yes no
Intrusive? yes yes no
Time high high medium
Equipment low high low
Expertise medium high medium
Monitoring UE techniquesEye tracking Physiological
measurement
Stage implementation implementation
Style laboratory laboratory
Objective? yes yes
Measure quantitative quantitative
Information low level low level
Immediacy yes yes
Intrusive? no yes
Time medium/high medium/high
Equipment high high
Expertise high high
Vocabulary application
Task analysis Feasability study Design representation
Feasibility study
Why is system needed? How will system help to improve user task
performance? Are there critical processes that need to be
supported by system? What are the technical implications? Can system be produced within given budget? Is there a timescale for the development of the
system?
Strategies for representing design
Storyboards State transition diagrams Simulations Scenarios Rapid prototyping Wizard of Oz …
Storyboard
State Transition Diagram