chapter 10: introducing evaluation group 4: tony masi, sam esswein, brian rood, chris troisi
Post on 25-Dec-2015
226 Views
Preview:
TRANSCRIPT
Chapter 10: Introducing Chapter 10: Introducing EvaluationEvaluation
Group 4: Tony Masi, Sam Esswein, Brian Rood, Chris Troisi
Chapter GoalsChapter Goals
Explain the concepts and terms used to Explain the concepts and terms used to discuss evaluation.discuss evaluation.
HutchWorld case study.HutchWorld case study.
Examine how different techniques are Examine how different techniques are used at different stages of development.used at different stages of development.
Discuss how developers cope with real-Discuss how developers cope with real-world constraints.world constraints.
Definition of EvaluationDefinition of Evaluation
Dictionary Definition – “To ascertain or fix Dictionary Definition – “To ascertain or fix the value or worth of.”the value or worth of.”
Book’s Definition – “The process of Book’s Definition – “The process of systematically collecting data that informs systematically collecting data that informs us about what it is like for a particular user us about what it is like for a particular user or group of users to use a product for a or group of users to use a product for a particular task in a certain type of particular task in a certain type of environment.”environment.”
Some Evaluation Companies:Some Evaluation Companies:
META Group, Inc. – META Group, Inc. – www.metagroup.comwww.metagroup.com
Canadian Innovation Centere - Canadian Innovation Centere - http://www.innovationcentre.ca/http://www.innovationcentre.ca/
Hitachi Data Systems - Hitachi Data Systems - http://www.hds.com/http://www.hds.com/
How To Evaluate Your Software - How To Evaluate Your Software - http://www.ageesw.com/how.htmhttp://www.ageesw.com/how.htm
Iterative design & evaluation is a continuous process that Iterative design & evaluation is a continuous process that examines:examines:
Early ideas for conceptual model Early ideas for conceptual model
Early prototypes of the new systemEarly prototypes of the new system
Later, more complete prototypesLater, more complete prototypes
Designers need to check that they understand users’ Designers need to check that they understand users’ requirements.requirements.
What to evaluateWhat to evaluate
Why to evaluateWhy to evaluate
Designers should not presume everyone is Designers should not presume everyone is like them or that following set guidelines like them or that following set guidelines guarantees usability.guarantees usability.
Evaluation is needed to check that users can Evaluation is needed to check that users can use the product and like it.use the product and like it.
Why to evaluateWhy to evaluate
www.AskTog.comwww.AskTog.com– ““Iterative design, with its repeating cycle of Iterative design, with its repeating cycle of
design and testing, is the only validated design and testing, is the only validated methodology in existence that will methodology in existence that will consistently produce successful results.consistently produce successful results. If If you don’t have user-testing as an integral part you don’t have user-testing as an integral part of your design process you are going to throw of your design process you are going to throw buckets of money down the drain.”buckets of money down the drain.”– – Bruce TognazziniBruce Tognazzini
Why to evaluateWhy to evaluate
Tognazzini’s 5 reasons to evaluate:Tognazzini’s 5 reasons to evaluate:– Problems are fixed before the product is shipped, not Problems are fixed before the product is shipped, not
after.after.– The team can concentrate on real problems, not The team can concentrate on real problems, not
imaginary ones.imaginary ones.– Engineers code instead of debatingEngineers code instead of debating– Time to market is sharply reducedTime to market is sharply reduced– Upon first release, sales department has a rock-solid Upon first release, sales department has a rock-solid
design it can sell without having to pepper pitches with design it can sell without having to pepper pitches with how well the NEXT release will work.how well the NEXT release will work.
Why to evaluateWhy to evaluate
USABILITY TESTING involves measuring the USABILITY TESTING involves measuring the performance of typical users on typical tasks.performance of typical users on typical tasks.
SATISFACTION can be evaluated through SATISFACTION can be evaluated through questionnaires and interviews.questionnaires and interviews.
Trends are towards evaluating more Trends are towards evaluating more subjective user-experience goals, like subjective user-experience goals, like emotionally satisfying, motivating, fun, etc.emotionally satisfying, motivating, fun, etc.
When to evaluateWhen to evaluate
Throughout designThroughout design
New productNew product– Use mockups, sketches, and other low-fidelity prototyping Use mockups, sketches, and other low-fidelity prototyping
techniques to elicit users’ opinionstechniques to elicit users’ opinions
– Goal is to asses how well a design fulfills users’ needs and whether Goal is to asses how well a design fulfills users’ needs and whether the users like itthe users like it
Upgrade existing productUpgrade existing product– Evaluations to compare user performance and attitudes toward new Evaluations to compare user performance and attitudes toward new
design with those of the previous versionsdesign with those of the previous versions
Evaluation is a key ingredient for a successful design.Evaluation is a key ingredient for a successful design.
Two main types of evaluationTwo main types of evaluation
Formative evaluationFormative evaluation is done at different stages of is done at different stages of development to check that the product meets users’ needs.development to check that the product meets users’ needs.– Design proceeds through iterative cycles of ‘design-test-Design proceeds through iterative cycles of ‘design-test-
redesign’redesign’– Helps ensure products success upon first arrival in marketHelps ensure products success upon first arrival in market
Summative evaluationSummative evaluation assesses the quality of a finished assesses the quality of a finished product.product.– Satisfy sponsoring agencySatisfy sponsoring agency– Check that standard is being upheldCheck that standard is being upheld
Chapter 10 focuses on Formative Evaluation.Chapter 10 focuses on Formative Evaluation.
Story of the 1984 OMS Story of the 1984 OMS (Box 10.1)(Box 10.1)
Background –Background –– Voice mail system for Olympic Games contestants and Voice mail system for Olympic Games contestants and
their families could send and receive messagestheir families could send and receive messages– Developed by IBMDeveloped by IBM– Could be used from almost any push-button phone Could be used from almost any push-button phone
system around the worldsystem around the world
Reasons for intense evaluation -Reasons for intense evaluation -– IBM’s reputation at stakeIBM’s reputation at stake– Olympics a high-profile eventOlympics a high-profile event
Story of the 1984 OMS Story of the 1984 OMS (Box 10.1)(Box 10.1)
Evaluation activities –Evaluation activities –– Use of printed scenariosUse of printed scenarios
– Iterative testing of user guidesIterative testing of user guides
– Development of early simulationsDevelopment of early simulationsKeypadsKeypads
ReactionsReactions
– Olympian on design teamOlympian on design team
– Interviews with other OlympiansInterviews with other Olympians
– Overseas testsOverseas tests
– Free coffee and donut testsFree coffee and donut tests
– ‘‘Try-and-destroy-it’ with Computer Science studentsTry-and-destroy-it’ with Computer Science students
– Pre-Olympic field testsPre-Olympic field tests
– Heavy traffic testsHeavy traffic tests
HutchWorld Case StudyHutchWorld Case Study
A Virtual CommunityA Virtual CommunityCollaboration:Collaboration:– Microsoft’s Virtual Worlds Research GroupMicrosoft’s Virtual Worlds Research Group– Fred Hutchinson Cancer Research CenterFred Hutchinson Cancer Research Center
Uses:Uses:– chattingchatting– storytellingstorytelling– discussionsdiscussions– emotional supportemotional support
Why? Why? – cancer patient isolation issuescancer patient isolation issues
How to design HutchWorld?How to design HutchWorld?
Needs:Needs:– usefuluseful– engagingengaging– easy-to-useeasy-to-use– emotional satisfactionemotional satisfaction
Early ideas:Early ideas:– what is typical cancer treatment?what is typical cancer treatment?– what resources are available to patients?what resources are available to patients?– what are the specific needs of the users?what are the specific needs of the users?– what kind of “world” should be the model?what kind of “world” should be the model?– how will users interact within the virtual community?how will users interact within the virtual community?– what should it look like?what should it look like?
No stone left unturnedNo stone left unturned
Interviews with patients, caregivers, family, Interviews with patients, caregivers, family, friends, clinicians, social support groups, friends, clinicians, social support groups, former patients, and expertsformer patients, and experts
Reading of latest research literature, and Reading of latest research literature, and HutchWorld web pagesHutchWorld web pages
Visiting Fred Hutch research facilities, and Visiting Fred Hutch research facilities, and the Hutch school for pediatric patients and the Hutch school for pediatric patients and juvenile patient family membersjuvenile patient family members
ProblemProblem
Inadequate non-verbal feedbackInadequate non-verbal feedback– potential for misunderstandingpotential for misunderstanding– no:no:
facial expressionsfacial expressions
body languagebody language
tone of voicetone of voice
ResearchResearch
Studies indicates social support helps Studies indicates social support helps cancer patients cope psychologically with cancer patients cope psychologically with their disease.their disease.
Patients also benefit in their overall Patients also benefit in their overall physical wellbeing.physical wellbeing.
Example: women with breast cancer Example: women with breast cancer receiving therapy lived on average twice receiving therapy lived on average twice as long as those that did not.as long as those that did not.
Features of HutchWorldFeatures of HutchWorld
Availability:Availability:– anytime, day or nightanytime, day or night– regardless of geographic locationregardless of geographic location
Designed to resemble the outpatient facility Designed to resemble the outpatient facility – This real-world metaphor helped users infer the This real-world metaphor helped users infer the
functionality.functionality.
Synchronous chat environment was selected for Synchronous chat environment was selected for realism (vs. asynchronous)realism (vs. asynchronous)
3D photographic avatars (p. 326)3D photographic avatars (p. 326)
Before testingBefore testing
Logistical issues:Logistical issues:– Who would provide training for patients and Who would provide training for patients and
testers?testers?– How many systems were needed for testing?How many systems were needed for testing?– Where should these systems be placed?Where should these systems be placed?
Testing HutchWorldTesting HutchWorld
Test 1:Test 1:– six computers six computers – scaled-back prototypescaled-back prototype– Microsoft specialists trained Hutch volunteersMicrosoft specialists trained Hutch volunteers– events were hosted in the prototypeevents were hosted in the prototype
Test 1 observations:Test 1 observations:– general usage of the prototypegeneral usage of the prototype– usage of the space during unscheduled timesusage of the space during unscheduled times
Testing HutchWorldTesting HutchWorld
Test 1 results:Test 1 results:– small user communitysmall user community– critical mass concept – not enough participants to fill critical mass concept – not enough participants to fill
the chat room for successful conversationthe chat room for successful conversation– lack of interestlack of interest– patient availabilitypatient availability– patients preferred asynchronous communication (via patients preferred asynchronous communication (via
email, journals, etc.)email, journals, etc.)– prototype did not include original computer uses prototype did not include original computer uses
patients played games and searched the internetpatients played games and searched the internet
Redesigning HutchWorldRedesigning HutchWorld
a more “unified” product was desired that a more “unified” product was desired that included a variety of communication, included a variety of communication, information, and entertainment tasksinformation, and entertainment tasks
new support:new support:– more asynchronous communicationmore asynchronous communication– information-retrieval toolsinformation-retrieval tools– email, a bulletin board, text-chatemail, a bulletin board, text-chat– games and other entertainment tasksgames and other entertainment tasks– a web page creation toola web page creation tool– a way to check if anyone is around to chat witha way to check if anyone is around to chat with
Usability TestsUsability Tests
Seven participantsSeven participants– four had used chat roomsfour had used chat rooms– all had browsed the weball had browsed the web– given five minutes to get familiar with softwaregiven five minutes to get familiar with software
A running commentary was given by each during A running commentary was given by each during exploration (what each was looking at, thinking, exploration (what each was looking at, thinking, or confused by)or confused by)After five minutes, a series of structured tasks were After five minutes, a series of structured tasks were given focusing on how the participants (p. 329):given focusing on how the participants (p. 329):– dealt with their virtual identitydealt with their virtual identity– communicated with otherscommunicated with others– retrieved desired informationretrieved desired information– found entertainmentfound entertainment
QuestionnaireQuestionnaire
After the test participants were asked to fill After the test participants were asked to fill out a questionnaire about their experience out a questionnaire about their experience with HutchWorldwith HutchWorld– What did you like about HutchWorld?What did you like about HutchWorld?– What did you not like about HutchWorld?What did you not like about HutchWorld?– What did you find confusing or difficult to use What did you find confusing or difficult to use
in HutchWorld?in HutchWorld?– How would you suggest improving How would you suggest improving
HutchWorld?HutchWorld?
Usability FindingsUsability Findings
The back button did not always work.The back button did not always work.Users ignored navigation buttonsUsers ignored navigation buttons– more prominent buttons were neededmore prominent buttons were needed
Users expected that objects in 3D would do something Users expected that objects in 3D would do something when clicked onwhen clicked on– provide links to web pages when objects are clickedprovide links to web pages when objects are clicked
Users did not realize other real people were interacting Users did not realize other real people were interacting with them in the worldwith them in the world– wording was changed in the overview descriptionwording was changed in the overview description
Users did not notice the chat window and instead Users did not notice the chat window and instead chatted with people on the participation listchatted with people on the participation list– instructions on where to chat were clarifiedinstructions on where to chat were clarified
Follow-upFollow-up
more rounds of observation and testing more rounds of observation and testing were conducted with new subjectswere conducted with new subjects
HutchWorld was installed at the Fred HutchWorld was installed at the Fred Hutchinson CenterHutchinson Center
observations continued on the users observations continued on the users – which parts of the system are being used?which parts of the system are being used?– when are they being used?when are they being used?– why are they being used?why are they being used?
Future of HutchWorldFuture of HutchWorld
evaluation of the effects of the software at the evaluation of the effects of the software at the Fred Hutchinson CenterFred Hutchinson Centerinvestigation will include:investigation will include:– How the computers and software impact the social How the computers and software impact the social
wellbeing of the patients and their caregivers?wellbeing of the patients and their caregivers?– What type of computer-based communication best What type of computer-based communication best
supports this patient community?supports this patient community?– What are the general usage patterns of the system?What are the general usage patterns of the system?– How might any medical facility use computers and How might any medical facility use computers and
software like HutchWorld to provide social support for software like HutchWorld to provide social support for its patients and caregivers?its patients and caregivers?
top related