![Page 1: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/1.jpg)
Ch 8: Usability Testing - 1
Yonglei Tao
![Page 2: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/2.jpg)
Usability Learnability Efficiency User retention over time
Memorability Error rate
Error frequency and severity Subjective Satisfaction
![Page 3: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/3.jpg)
ISO Usability Standard 9241 Effectiveness
Can you achieve what you want to? Efficiency
Can you do it without wasting effort? Satisfaction
Do you enjoy the process?
![Page 4: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/4.jpg)
Why Usability Testing Developers and users possess different
models Developer’s intuitions are not always correct
There is no average user It is impossible to predict usability from
appearance Problems found late in the development
process are more difficult and expensive to fix
![Page 5: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/5.jpg)
Common Usability Problems Reported by IBM Usability Experts Ambiguous menus and icons Single-direction movement through a system Lack of white space Annoying distractions Unclear step sequences Input and direct manipulation limits More steps to manage the interface than to
perform tasks Lack of system anticipation and intelligence Inadequate feedback and confirmation Inadequate error messages, help, tutorials,
and documentation
![Page 6: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/6.jpg)
Common Activities in the Process Capture
Collecting usability data such as task completion time, errors, guideline violations, and subjective ratings
Analysis Interpreting data to identify usability problems
in the interface Critique
Suggesting solutions or improvements
![Page 7: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/7.jpg)
Questions to Answer When to perform evaluation
Early, middle, or late What to achieve
Concerns and goals Tasks and scenarios Measurements
Which technique to use Who are involved and where to do How to analyze data and identify problems
![Page 8: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/8.jpg)
When to Evaluate? Formative evaluation
Assessment that occurs during the formation of a design
Summative evaluation An evaluation of the final product
(implementation)
![Page 9: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/9.jpg)
What to Evaluate Features Screen layout Navigational structure Reliability and performance User support Relative to end user tasks and skills
![Page 10: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/10.jpg)
Which Technique to Use? Analytic evaluation
Keystroke-Level Model GOMS State Transition Networks Hick and Fitts’s laws
Expert reviews Cognitive walkthrough Heuristic evaluation Guidelines checklists Consistency inspection
![Page 11: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/11.jpg)
Which Technique to Use? Observational methods
Think aloud protocol Co-discovery method Comparative evaluation Journaled sessions
Query techniques Contextual inquiry Interviews Questionnaires
Controlled experiments Eye tracking Performance measurement
![Page 12: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/12.jpg)
How to Measure Objective measurements
Collect data from users during evaluation Subjective measurements
Users’ opinions, observations, and responses to interview and follow-up questions
![Page 13: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/13.jpg)
Measuring Methods Time to complete a task Percent of task completed (per time unit) Number of user errors Time spent in errors Number of times that a user is not able to
proceed or makes a serious mistake Number of times that help is needed Number of commands used Number of repetitions of failed commands Overall satisfaction with a product User comments and questions
![Page 14: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/14.jpg)
Some Metrics from ISO 9241
Usability Effectiveness Efficiency Satisfactionobjective measures measures measures
Suitability Percentage of Time to Rating scale for the task goals achieved complete a task for satisfaction
Appropriate for Number of power Relative efficiency Rating scale fortrained users features used compared with satisfaction with
an expert user power features
Learnability Percentage of Time to learn Rating scale forfunctions learned criterion ease of
learning
Error tolerance Percentage of Time spent on Rating scale for
errors corrected correcting errors error handling successfully
![Page 15: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/15.jpg)
Ways to Determine if a usability goal is achieved With respect to information on
An existing system or previous version Competitive systems Carrying out the task without use of a computer
system An absolute scale Your own prototype User’s own earlier performance Each component of a system separately A successive split of the difference between best
and worst values observed in user tests
![Page 16: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/16.jpg)
An Example – Usability Specification An electronic meeting calendar
Allows the user to schedule and keep track of future meetings
Replaces a paper-based system Specifying usability goals early on so as
to drive the design activity
![Page 17: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/17.jpg)
Usability Specification - 1 Attribute: Ease of learning
Measuring concept: Ease of first use of system without training
Measuring method: Time to create first entry in the to- do list
Now level: 30 seconds on paper-based system
Worst case: 1 minute Planned level: 45 seconds Best case: 30 seconds (equivalent to now)
![Page 18: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/18.jpg)
Usability Specification - 2 Attribute: Task efficiency
Measuring concept: Scheduling a weekly meeting Measuring method: Time it takes to enter a
weekly meeting appointment
Now level:
Worst case: Planned level: Best case:
![Page 19: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/19.jpg)
Usability Specification - 3 Attribute: Recoverability
Measuring concept: Measuring method:
Now level:
Worst case: Planned level: Best case:
![Page 20: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/20.jpg)
Discussion Imagine you have asked to produce a
prototype for this meeting calendar system. What would be an appropriate prototyping technique to enable you to test the design according to the usability specification given here, and why?
![Page 21: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/21.jpg)
Usability Evaluation Process Determine what to find out Develop the tasks for each experiment
Writing Scripts
Get some users Set up the testing environment with
samples Run the test and collect data Analyze the data
![Page 22: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/22.jpg)
Perform the Test Pre-Test
Have the users fill out a pre-test questionnaire if any
Provide Instructions to them During the Test
Let the users proceed with scripts Find out how they work with the product Collect data on how they are doing
Post-Test Have the users fill out a post-test questionnaire Verbal interview with them Organize and summarize test data
![Page 23: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/23.jpg)
Process the Data Documenting Identify problems
Severity Frequency
Prioritize problems Find out reasons Work out solutions
![Page 24: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/24.jpg)
Case Study Information retrieval tasks for INTUITIVE
Navigation and exploration User’s knowledge about the database is imprecise Need to show what is in the database and allow the
user to select entities for queries Query formation
Translate user’s need, expressed via a graphical notation, to SQL internally
Previewing the retrieved data Allow the user to select a subset of retrieved items
Presentation of retrieved items
![Page 25: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/25.jpg)
Usability Evaluation User testing
Users were videotaped and timed when performing increasingly complex tasks
Captured data including time taken, errors, help accessed, and task steps missed
Expert evaluation Using Nielsen’s heuristic checklist by HCI
expert evaluators
![Page 26: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/26.jpg)
Experiment Results Error count
Experts identified 86 usability problems Users identified 38, not a subset of the above
Cost 33.5 hours for heuristic evaluation 125 hours for end user testing More expensive to hire HCI experts
Ease of problem fixing HCI experts are better at accurate, thorough
reporting and identifying causes
![Page 27: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/27.jpg)
Experiment Results (Cont.) Problems identified by end users, not by HCI
experts 39% of those identified by users Some examples
Repeated resubmission of the same queries Stuck on creating query Cannot make sense of results Difficult in moving windows
![Page 28: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/28.jpg)
Experiment Results (Cont.) Problems identified by both end users and
HCI experts More feature related, rather than performance
related Some examples
No complete view of the query – only one entity at a time
Menu terminology confusing Attribute selection unpredictable
![Page 29: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/29.jpg)
Experiment Results (Cont.) Problems identified by HCI experts, not by
end users 40% of those identified by experts Some examples
No arrows indicating the direction of relations Not able to cancel the operation after a query is
submitted Slow response time to display results There should be a clean up command
![Page 30: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/30.jpg)
Discussions Heuristic evaluation
Identify an interface error by predicting user problems it will cause
Good at finding poor terminology and lack of clarity
Ten heuristics are inadequate as a guide A subjective process
![Page 31: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/31.jpg)
Discussions (Cont.) End user testing
Identify the symptom and infer its cause Good at finding problems while performing real
tasks Task-based May miss features not encountered in tasks Users tend to blame themselves rather than the
interface
![Page 32: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/32.jpg)
Summary Both techniques share the same goals, but the
actual results are quite different End user testing indicates the symptom of a problem Heuristic evaluation identifies its cause
Heuristic evaluation helps analyze observed problems Observation of novices is still vital as many
problems are a consequence of the user’s knowledge, or lack of it
Usability testing is costly and time-consuming Necessary to use a variety of techniques Need to focus on the aspects each one does best
![Page 33: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/33.jpg)
Comparative Study of Heuristic Evaluation and User Testing A study published by National Institute of Health
Heuristic evaluation, cognitive walk-throughs, and usability testing are increasingly used to evaluate and improve the design of clinical applications
Need to find out how they can be used effectively Conducted heuristic evaluation and usability
testing on four major patient record systems which cover 80% of the market in general dentistry
Both methods indicate that there are significant usability problems with those systems
![Page 34: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/34.jpg)
Comparative Study (Cont.) Heuristic evaluation
Evaluators were dentists with significant background in information systems and usability evaluation
Evaluated each of the four dental systems Usability testing
Four groups of users with five novice users in each Each participant used one system and worked through
nine clinical tasks using a think-aloud protocol Two researchers recorded both task outcomes (rate of
completed tasks, incomplete tasks and incorrectly completed tasks) and the types of usability problems for each task
![Page 35: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/35.jpg)
Comparative Study (Cont.) An average of 50% of empirical determined usability
problems were identified by heuristic evaluation Some statements of heuristic violations were specific
enough to precisely identify the actual usability problems
Other violations were less specific, but still revealed usability problems and poor task outcomes
“Heuristic evaluation may, under certain circumstances, be a useful tool to determine design problems early in the development cycle.”
![Page 36: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/36.jpg)
Usability Problems Identified with the Two Usability Methods
![Page 37: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/37.jpg)
“Specific” Heuristic Violations Which Directly Predicated a Usability Problem
![Page 38: Ch 8: Usability Testing - 1 Yonglei Tao. Usability Learnability Efficiency User retention over time Memorability Error rate Error frequency](https://reader036.vdocuments.us/reader036/viewer/2022062322/5697bfef1a28abf838cba0c5/html5/thumbnails/38.jpg)
“General” Heuristic Violations Which Suggested a Usability Problem