evaluation methods - summary. how to chose a method? stage of study – formative, iterative,...
TRANSCRIPT
![Page 1: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/1.jpg)
Evaluation Methods - Summary
![Page 2: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/2.jpg)
How to chose a method?
Stage of study– formative, iterative, summative
Pros & cons Metrics
– depends on what you want to measure Qualitative vs. quantitative Research perspective
– CS vs. psychology vs. sociology
![Page 3: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/3.jpg)
Pros & Cons
Realism Precision Generalizability Time & cost Researcher expertise
![Page 4: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/4.jpg)
Methods
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Experimental simulation GOMS Computer simulation Formal theory
![Page 5: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/5.jpg)
Survey
Pros– Easy to get a large number of responses.– Quick and easy to conduct.– Highly generalizable.
Cons– Self-selection.– Participants often only offer enough information to answer the
question.– Can miss details.– Low in realism and precision.
![Page 6: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/6.jpg)
Interviews
Pros– Quick and easy to conduct.– Gives designer quick feedback on a range of ideas.– Can get a person’s initial reaction to an idea.– Can get detailed information from a person.
Cons– Often takes place away from natural setting.– Question wording or interviewer “body language” can bias
answers.– High probability of false positives and missed problems (e.g., users
may not have a clear idea of how an app will be used).– Can miss details if interviewer doesn’t know what issues to draw
out.
![Page 7: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/7.jpg)
Controlled Lab Experiment
Pros– Provides precise, quantifiable data.– Easier to draw inferences from data.– Relatively quick.– Can get a medium-sized number of participants.
Cons– Short duration of a lab experiment may not be enough to allow
users to become accustomed to an app.– Not a natural setting – interaction may not be normal.
![Page 8: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/8.jpg)
In-lab Observation
Pros– Relatively quick.– Can get a medium-sized number of participants.
Cons– Observations are subjective and error prone.– Short duration of lab observation is not enough time
for user to get accustomed to using the interface.– Not a natural setting – interaction may not be
normal.
![Page 9: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/9.jpg)
Controlled Field Experiment
Pros– Less intrusive than most other evaluation methods.– Provides more precise data than field observation.– Can observe natural behavior of user (though some
part of the system will be controlled/unnatural).
Cons– More intrusive than field observation.– Less natural than field observation.
![Page 10: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/10.jpg)
Field Observation Study
Pros– Only way to observe natural behavior of user & interaction
between user & tools.
Cons– Difficult and time consuming.– Hard to get permission to observe people.– Observations are subjective and error prone.– Cannot make strong interpretations from observations.– Not very generalizable.
![Page 11: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/11.jpg)
Heuristic Evaluation
Pros– Quick and easy.
Cons– Nielson’s heuristics may not be as relevant to non-
GUIs.– Results in false positives in missed problems,
especially when experts are not part of target audience.
![Page 12: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/12.jpg)
Cognitive Walkthrough
Pros– Quick and easy.
Cons– Results in false positives and missed problems
when evaluator is different from target audience.
![Page 13: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/13.jpg)
Experimental Simulation
Pros– Still fairly precise.– More realistic than in-lab experiment.
Cons (same as lab exp.)– Short duration of a lab experiment may not be
enough to allow users to become accustomed to an app.
– Not a natural setting – interaction may not be normal.
![Page 14: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/14.jpg)
GOMS
Pros– Predict human performance before committing to a
specific design in code or running user studies– Many studies have validated the model (it works)
Cons– Assumes error-free, skilled user behavior– No formal recipe for how to perform analysis– Significant time investment
![Page 15: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/15.jpg)
Computer Simulation
Creating a complete & closed system that models the operation of the concrete system without users.
Example:– geophysical process going on in connection with the
eruption of Mount St. Helens– Simulation of pilot during plane crash
![Page 16: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/16.jpg)
Computer Simulation
Pros– Supposedly high in realism (depends on accuracy of
data/system replication)
Cons– Low in precision & generalizability
![Page 17: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/17.jpg)
Formal Theory– This is a dissertation
Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest.
Pros– Relatively generalizable
Cons– Not realistic or precise
![Page 18: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/18.jpg)
How to chose a method?
Stage of study Pros & cons
– Realism– Precision– Generalizability– Time & cost
Researcher expertise Metrics Qualitative vs. quantitative Research perspective
![Page 19: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/19.jpg)
Methods
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 20: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/20.jpg)
Early Stage – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 21: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/21.jpg)
Iterative & Summative Stages – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 22: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/22.jpg)
Realism – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 23: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/23.jpg)
Precision – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 24: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/24.jpg)
Generalizability – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 25: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/25.jpg)
Time & Cost – these are the best
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 26: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/26.jpg)
Researcher Perspective – Whoever is doing will choose whatever they want
Survey Interview Controlled-lab
experiment In-lab observation Controlled field
experiment Field observation study Heuristic Evaluation
Cognitive Walkthrough Automated observation
user study Experimental simulation GOMS Computer simulation Formal theory
![Page 27: Evaluation Methods - Summary. How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to](https://reader036.vdocuments.us/reader036/viewer/2022062519/5697bff21a28abf838cbbcc5/html5/thumbnails/27.jpg)
Metrics: examples
Traditional GUIs:– efficiency (time to complete task)– accuracy (# of errors)– simplicity
Peripheral Displays:– awareness (recall)– distraction (dual-task behavior)– aesthetics