11

Evaluate—Qualitative Evaluate—Qualitative MethodsMethods

October 2, 2007October 2, 2007

NEEDS

DESIGN

IMPLEMENTEVALUATE

22

EvaluationEvaluation

A little out of sequence due to A little out of sequence due to schedulingscheduling

Will get more implementation over Will get more implementation over next two weeksnext two weeks

Imagine you’ve implemented your Imagine you’ve implemented your applicationapplication

These are techniques you will need These are techniques you will need to design user study (end of to design user study (end of project)project)

33

Methods for evaluating Methods for evaluating system system QualitativeQualitative

– Rich, subjectiveRich, subjective– Exploratoring conceptsExploratoring concepts– More useful for earlier inputMore useful for earlier input

QuantitativeQuantitative– Precise, objective, repeatablePrecise, objective, repeatable– Demonstrating claimsDemonstrating claims– More useful at documenting More useful at documenting

improvementimprovement– Can be expensiveCan be expensive

44

For your projectFor your project

Will require aspects of both qualitative Will require aspects of both qualitative and quantitative methodsand quantitative methods– QualitativeQualitative

How users react to project, perceptions?How users react to project, perceptions?

– QuantitativeQuantitative How users perform on project?How users perform on project?

What would you improve on next What would you improve on next iteration?iteration?– Perhaps users’ perceptions of performance Perhaps users’ perceptions of performance

more important than actual valuesmore important than actual values– Elevator waiting storyElevator waiting story

55

DesignDesign evaluation evaluation methods!methods! Most important aspect of Most important aspect of

evaluation is upfront design!evaluation is upfront design!– Expensive to line up users, collect Expensive to line up users, collect

datadata– Design to collect right informationDesign to collect right information

Pick appropriate method for what Pick appropriate method for what you want to learnyou want to learn

66

Applying an evaluation Applying an evaluation methodmethod Determine the activity to observeDetermine the activity to observe Develop the methodDevelop the method Human subjects review approvalHuman subjects review approval Pilot test the methodPilot test the method Recruit participantsRecruit participants Collect the dataCollect the data Inspect & analyze the dataInspect & analyze the data Draw conclusions to resolve design problems, Draw conclusions to resolve design problems,

reflect on what you learnedreflect on what you learned Redesign and implement the revised interfaceRedesign and implement the revised interface

77

Demographic Demographic informationinformation Demographic dataDemographic data

– Age, gender, cultureAge, gender, culture– Task expertise, experienceTask expertise, experience– MotivationMotivation– Frequency of useFrequency of use– Education, literacy, trainingEducation, literacy, training

No matter what method, collect No matter what method, collect demographic datademographic data

88

Environmental Environmental informationinformation Besides info on the user, may also Besides info on the user, may also

need info on the operating need info on the operating environmentenvironment– Windows, Mac, Linux?Windows, Mac, Linux?– Firefox, Internet Explorer, Safari?Firefox, Internet Explorer, Safari?– Wired ethernet, wireless, modemWired ethernet, wireless, modem– Morning, afternoon, nightMorning, afternoon, night– Office, mobile, homeOffice, mobile, home

99

Qualitative methodsQualitative methods

““Discount” usability methodsDiscount” usability methods– Hueristic EvaluationHueristic Evaluation– Cognitive Walkthrough Cognitive Walkthrough

Questionnaire / SurveyQuestionnaire / Survey Think aloud protocolThink aloud protocol Co-discoveryCo-discovery Semi-structured interviewSemi-structured interview Deploy and observe in useDeploy and observe in use

1010

““Discount” usability Discount” usability methodsmethods Enable evaluation at early stage, Enable evaluation at early stage,

before prototype implementedbefore prototype implemented Conducted quickly, inexpensivelyConducted quickly, inexpensively Early evaluation investment saves Early evaluation investment saves

downstream development costsdownstream development costs– Hueristic evaluationHueristic evaluation– Cognitive walkthroughCognitive walkthrough

1111

Heuristic EvaluationHeuristic Evaluation

Fancy way to describe expert Fancy way to describe expert reviewreview– HCI expertHCI expert– Domain expertDomain expert

Expert review identifies usability Expert review identifies usability issues before implementationissues before implementation

Our grades on your homework are Our grades on your homework are form of heuristic evaluationform of heuristic evaluation

1212

Evaluation hueristicsEvaluation hueristics

Visibility of system statusVisibility of system statusMatch between system and the real worldMatch between system and the real worldUser control and freedomUser control and freedomConsistency and standardsConsistency and standardsError preventionError preventionRecognition rather than recallRecognition rather than recallFlexibility and efficiency of useFlexibility and efficiency of useAesthetic and minimalist designAesthetic and minimalist designHelp users recognize, diagnose, and recover Help users recognize, diagnose, and recover

from errorsfrom errorsHelp and documentation Help and documentation

1313

Heuristic evaluation Heuristic evaluation methodmethod Multiple experts individually review Multiple experts individually review

(around 5 experts get 75% (around 5 experts get 75% problems)problems)

Observer records issues, answers Observer records issues, answers questions, gives hintsquestions, gives hints

Conduct using low fidelity prototype Conduct using low fidelity prototype or task analysis with storyboards and or task analysis with storyboards and scenariosscenarios

Generate list of usability problems Generate list of usability problems according to hueristic compromisedaccording to hueristic compromised

1414

Hueristic Evaluation Hueristic Evaluation analysisanalysis After created list of problemsAfter created list of problems

– Rank severityRank severity– Estimate fixabilityEstimate fixability– Suggest possible fixesSuggest possible fixes

Analysis may involve larger teamAnalysis may involve larger team

1515

Hueristic Evaluation as Hueristic Evaluation as rigorous design reviewrigorous design review You can make a living out of You can make a living out of

doing Hueristic Evaluationdoing Hueristic Evaluation– Substantial consulting market for Substantial consulting market for

conducting Heuristic Evaluationconducting Heuristic Evaluation You may pay a consultant to do a You may pay a consultant to do a

Heuristic EvaluationHeuristic Evaluation– Know what you’re paying forKnow what you’re paying for– Especially the Severity, Fixability, Especially the Severity, Fixability,

Potential Fix aspectsPotential Fix aspects

1616

Learning more about Learning more about Hueristic EvaluationHueristic Evaluation You can learn to do a Hueristic You can learn to do a Hueristic

EvaluationEvaluation– http://www.useit.com/papers/http://www.useit.com/papers/

heuristic/heuristic/

1717

Cognitive WalkthroughCognitive Walkthrough

Have user imagine walking through Have user imagine walking through the process of using systemthe process of using system

Can use low-fidelity prototyping, Can use low-fidelity prototyping, partially implemented prototypepartially implemented prototype

Can use target user rather than Can use target user rather than expertexpert– Pluralistic walkthrough uses experts, Pluralistic walkthrough uses experts,

users, developersusers, developers Like a code walkthroughLike a code walkthroughC. Wharton et. al. "The cognitive walkthrough method: a practitioner's guide"

in J. Nielsen & R. Mack "Usability Inspection Methods" pp. 105-140.

1818

Walkthrough Walkthrough procedureprocedure Give user representation of interface Give user representation of interface

and taskand task– Can they discover how to accomplish goal Can they discover how to accomplish goal

with description of interface?with description of interface?– Can ask “From here, how would you like to Can ask “From here, how would you like to

accomplish…?”accomplish…?” Step through interfaceStep through interface

– User takes action, system provides responseUser takes action, system provides response– Describe actions not depicted in interface Describe actions not depicted in interface

representationrepresentation– Somewhat like Wizard of OzSomewhat like Wizard of Oz

1919

Stepping through Stepping through interfaceinterface Will user try to achieve the right goal? Will user try to achieve the right goal?

– Conceptual model of goals and tasksConceptual model of goals and tasks Will user notice correct action is Will user notice correct action is

available? available? – VisibilityVisibility– Understandability Understandability

Will user associate correct action with Will user associate correct action with the goal to be achieved? the goal to be achieved? – Aligning goals with sequence of actionsAligning goals with sequence of actions

If correct action performed, will user see If correct action performed, will user see progress toward solution? progress toward solution? – FeedbackFeedback

2020

Next assignmentNext assignment

Testing storyboard with one userTesting storyboard with one user– Effectively, this is a cognitive Effectively, this is a cognitive

walkthroughwalkthrough– Create storyboardCreate storyboard– Define taskDefine task– Step through with one userStep through with one user

2121

Questionnaires & Questionnaires & surveyssurveys User responses to specific questionsUser responses to specific questions Preparation is expensive, Preparation is expensive,

administration relatively cheapadministration relatively cheap Oral vs. writtenOral vs. written

– Oral provides interaction, followup, but Oral provides interaction, followup, but takes more timetakes more time

– Written more efficient, can provide Written more efficient, can provide quantitative dataquantitative data

2222

Designing questionsDesigning questions

Design questions with analysis in Design questions with analysis in mindmind– Closed format more precise, easier to Closed format more precise, easier to

analyzeanalyze Convert qualitativeConvert qualitativequantitative measuresquantitative measures You give categories to usersYou give categories to users

– Open-ended questions provide richer Open-ended questions provide richer feedback, longer to analyzefeedback, longer to analyze Users give you categoriesUsers give you categories

2323

Designing survey Designing survey questionsquestions Multiple choiceMultiple choice

– Collecting informationCollecting information Ordinal rankingOrdinal ranking

– Expressing relative preferencesExpressing relative preferences Likert scalesLikert scales

– Expressing personal reactionsExpressing personal reactions

2424

Closed format stylesClosed format styles

facebook

LinkedIn

Orkut

MySpace

Rank frequency of usefrom5 – Most frequent1- Least frequent0 - Unused

___ facebook___ MySpace___ LinkedIn___ Orkut___ Other__________

Which social networkingsystems do you use?

Other_____________

Multiple choice Ordinal ranking

2525

Likert scalesLikert scales

Ask users to rate on a numeric scaleAsk users to rate on a numeric scale Odd number scale allows a neutral Odd number scale allows a neutral

midpoint (5- or 7-point scale)midpoint (5- or 7-point scale) Even number scale forces taking a Even number scale forces taking a

position (4- or 6-point scale)position (4- or 6-point scale) ““Anchors” give examples of points Anchors” give examples of points

along the scalealong the scale

2626

Example questionExample question

How important is the Berkeley-How important is the Berkeley-Stanford Big Game?Stanford Big Game?

Very ImportantNot Important

Most important event this Fall

Could notcare less

Maybe I’ll goif my friends go

2727

Closed FormatClosed Format

AdvantagesAdvantages– Clarify among Clarify among

alternativesalternatives– Easily Easily

quantifiablequantifiable– Eliminate useless Eliminate useless

answersanswers– Relatively quick Relatively quick

to administerto administer

DisadvantagesDisadvantages– Must cover whole Must cover whole

rangerange– All choices should All choices should

be similarly likelybe similarly likely– Don’t get Don’t get

interesting, interesting, “different” reactions“different” reactions

2828

Questions people can Questions people can answer about answer about themselvesthemselves What they doWhat they do How they do itHow they do it Opinions about current activitiesOpinions about current activities Complaints about current activitesComplaints about current activites Comparing one thing with anotherComparing one thing with another How often they have done How often they have done

something in the recent pastsomething in the recent past

2929

Questions people Questions people cannot answer about cannot answer about themselvesthemselves Predicting what they would do / Predicting what they would do /

like / wantlike / want Imagining a hypothetical scenarioImagining a hypothetical scenario Whether they would like a certain Whether they would like a certain

feature or productfeature or product Estimating how often they do Estimating how often they do

thingsthings

3030

What’s most What’s most important?important?

3131

Web-based survey Web-based survey toolstools SurveymonkeySurveymonkey

– http://www.surveymonkey.com/http://www.surveymonkey.com/ ZoomerangZoomerang

– http://info.zoomerang.com/http://info.zoomerang.com/ Allows free basic analysis, more Allows free basic analysis, more

advanced features for feeadvanced features for fee Can extend reach to large Can extend reach to large

number of respondentsnumber of respondents

3232

Thinking aloud Thinking aloud protocolprotocol Have subject “think out loud” Have subject “think out loud”

while performing taskwhile performing task Psychology to elicit cognitionPsychology to elicit cognition Requires training taskRequires training task Facilitator actively prompts if Facilitator actively prompts if

subject falls silent for more then subject falls silent for more then 10 secondss10 secondss– ““What are you thinking now?”What are you thinking now?”– ““So, you are trying to…?”So, you are trying to…?”– ““And now you are…?”And now you are…?”

3333

Exercise: VolunteerExercise: Volunteer

Never used Photoshop beforeNever used Photoshop before

3434

Co-discoveryCo-discovery

Have two people work on a task Have two people work on a task together (even though the task is together (even though the task is normally done by one person)normally done by one person)

Coordination with each other Coordination with each other naturally elicits cognitionnaturally elicits cognition

3535

Exercise: Two Exercise: Two volunteersvolunteers Never used Photoshop beforeNever used Photoshop before

3636

Think aloud and co-Think aloud and co-discoverydiscovery Valuable to evaluate tasks that Valuable to evaluate tasks that

require cognitionrequire cognition Time intensiveTime intensive Rich feedbackRich feedback Think aloud requires trainingThink aloud requires training

3737

Semi-structured Semi-structured interviewsinterviews Interactively asking questions (face-Interactively asking questions (face-

to-face, telephone)to-face, telephone) Give users chance to explain “why” Give users chance to explain “why”

to complement “what” they did, to complement “what” they did, subjective user’s viewpointsubjective user’s viewpoint

Can help with design questionsCan help with design questions– ““What improvements would you What improvements would you

suggest?”suggest?” Can be done individually or in groupsCan be done individually or in groups

3838

Semi-structured Semi-structured interviewsinterviews Begin with list of open-ended Begin with list of open-ended

questionsquestions– Ask all users these questionsAsk all users these questions– Let users elaborateLet users elaborate– Flexibility to ask follow-up questionsFlexibility to ask follow-up questions

Must audio-recordMust audio-record Interviewer should attend to user Interviewer should attend to user

(not notepad or laptop), use audio (not notepad or laptop), use audio record for data (note timestamps)record for data (note timestamps)

3939

Questionnaire IssuesQuestionnaire Issues

LanguageLanguage– Beware terminology, jargonBeware terminology, jargon

ClarityClarity– ““How effective was the system?” How effective was the system?”

(ambiguous)(ambiguous)

Avoid leading questionsAvoid leading questions– Phrase neutrally rather than positive Phrase neutrally rather than positive

or negativeor negative ““How easy or hard was it to accomplish How easy or hard was it to accomplish

the task?”the task?”

4040

Questionnaire Issues Questionnaire Issues (2)(2) Prestige bias Prestige bias

– People answer a certain way because they People answer a certain way because they want you to think that way about themwant you to think that way about them

Embarrassing questionsEmbarrassing questions– ““What did you have the most problems What did you have the most problems

with?”with?” Hypothetical questionsHypothetical questions ““Halo effect”Halo effect”

– When estimate of one feature affects When estimate of one feature affects estimate of another (e.g. intelligence/looks)estimate of another (e.g. intelligence/looks)

– Aesthetics & usability, one example in HCIAesthetics & usability, one example in HCI

4141

InterviewsInterviews

DisadvantagesDisadvantages– Subjective viewSubjective view– Interviewer(s) can bias the interviewInterviewer(s) can bias the interview– Problem of inter-rater or inter-Problem of inter-rater or inter-

experimenter experimenter reliabilityreliability (agreement)(agreement)

– Time-consumingTime-consuming– Hard to quantifyHard to quantify

4242

Pilot test observation Pilot test observation methodmethod Pilot test method with some target Pilot test method with some target

usersusers– Debug the questions, methodsDebug the questions, methods– Also debug logisticsAlso debug logistics– Don’t count pilot data in analysisDon’t count pilot data in analysis

Make changes now before Make changes now before collecting data (want method for collecting data (want method for collecting data to be consistent)collecting data to be consistent)

4343

Methods used in Methods used in combinationcombination Mix of closed format, open-ended Mix of closed format, open-ended

questionsquestions Surveys, questionnaires often Surveys, questionnaires often

used with quantitative used with quantitative performance measures to assess performance measures to assess how users feel about interactionshow users feel about interactions

4444

Mechanics of user Mechanics of user testingtesting Readings give more detailed nuts Readings give more detailed nuts

and boltsand bolts Common sense structuring of the Common sense structuring of the

experience to help it run experience to help it run smoothlysmoothly

4545

Analyzing qualitative Analyzing qualitative datadata Rich, open-ended dataRich, open-ended data Goal: Structure to characterize, Goal: Structure to characterize,

describe, summarize datadescribe, summarize data Sounds harder than it isSounds harder than it is

4646

Analyzing qualitative Analyzing qualitative datadata Exercise to immerse in dataExercise to immerse in data

– Develop categories to countDevelop categories to count RangeRange AverageAverage

– Identify common patternsIdentify common patterns Allows identifying the interesting, Allows identifying the interesting,

unusual, exceptionsunusual, exceptions Also look for correlationsAlso look for correlations

4747

Exercise: Analyzing Exercise: Analyzing conceptual map of conceptual map of BerkeleyBerkeley Example of rich, qualitative dataExample of rich, qualitative data See if we can detect some See if we can detect some

patternspatterns Characterize set of qualitative Characterize set of qualitative

datadata

4848

Berkeley mapBerkeley map

Number of features?Number of features? Format of mapFormat of map Common featuresCommon features

– LandmarksLandmarks– RoadwaysRoadways

Unusual featuresUnusual features AssessmentsAssessments CorrelationsCorrelations

4949

Qualitative analysisQualitative analysis

Start with things you can countStart with things you can count– Average, range, medianAverage, range, median

Look for patterns that are in Look for patterns that are in commoncommon

Recognize features that are Recognize features that are unusual, interestingunusual, interesting

Look for correlationsLook for correlations Reflect on what the data is sayingReflect on what the data is saying

5050

Qualitative study of Qualitative study of your projectyour project What do you want to learnWhat do you want to learn

– User reactions, perceptionsUser reactions, perceptions– Conceptual model problemsConceptual model problems– Areas to improve designAreas to improve design– Does the design work?Does the design work?

5151

Next timeNext time

Quantitative methodsQuantitative methods ReadingsReadings

– ""A face(book) in the crowd: social A face(book) in the crowd: social Searching vs. social browsing" Searching vs. social browsing"

– ""iPod distraction: effects of iPod distraction: effects of portable music-player use on portable music-player use on driver performance"driver performance"

Questions on Project Proposal Questions on Project Proposal assignment?assignment?