stanford hci group / cs147 u 15 november 2007 closing the loop: from analysis to design scott...
Post on 20-Dec-2015
217 Views
Preview:
TRANSCRIPT
stanford hci group / cs147
http://cs147.stanford.edu15 November 2007
Closing the Loop:From Analysis to DesignScott Klemmertas: Marcello Bastea-Forte, Joel Brandt,Neil Patel, Leslie Wu, Mike Cammarano
Analyzing the results Quantitative data, which might include:
success rates time to complete tasks pages visited error rates ratings on a satisfaction questionnaire
Qualitative data, which might include: notes of your observations about the pathways
participants took notes about problems participants had (critical
incidents) notes of what participants said as they worked participants' answers to open-ended questions
Source: Usability.gov
Using the Test Results Summarize the data
make a list of all critical incidents positive & negative
include references back to original data try to judge why each difficulty
occurred What does data tell you?
UI work the way you thought it would? users take approaches you expected?
something missing?
Using the Results (cont.) Update task analysis & rethink design
rate severity & ease of fixing CIs fix both severe problems & make the easy
fixes
Will thinking aloud give the right answers? not always if you ask a question, people will always give
an answer, even it is has nothing to do with facts
try to avoid specific questions
Measuring Bottom-Line Usability
Situations in which numbers are useful time requirements for task completion successful task completion compare two designs on speed or # of errors
Ease of measurement time is easy to record error or successful completion is harder
define in advance what these mean
Do not combine with thinking-aloud. Why? talking can affect speed & accuracy
Analyzing the Numbers
Example: trying to get task time <=30 min. test gives: 20, 15, 40, 90, 10, 5 mean (average) = 30 median (middle) = 17.5 looks good!
Wrong answer, not certain of anything! Factors contributing to our uncertainty
small number of test users (n = 6) results are very variable (standard deviation =
32) std. dev. measures dispersal from the mean
Analyzing the Numbers (cont.)
This is what statistics is for Crank through the procedures and you find
95% certain that typical value is between 5 & 55
Analyzing the Numbers (cont.)
Participant # Time (minutes)1 202 153 404 905 106 5
number of participants 6mean 30.0median 17.5std dev 31.8
standard error of the mean = stddev / sqrt (#samples) 13.0
typical values will be mean +/- 2*standard error --> 4 to 56!
what is plausible? = confidence (alpha=5%, stddev, sample size) 25.4 --> 95% confident between 5 & 56
Web Usability Test Results
Analyzing the Numbers (cont.)
This is what statistics is for Crank through the procedures and you find
95% certain that typical value is between 5 & 55
Usability test data is quite variable need lots to get good estimates of typical values 4 times as many tests will only narrow range by 2x
breadth of range depends on sqrt of # of test users
this is when online methods become useful easy to test w/ large numbers of users
Measuring User Preference How much users like or dislike the system
can ask them to rate on a scale of 1 to 10 or have them choose among statements
“best UI I’ve ever…”, “better than average”… hard to be sure what data will mean
novelty of UI, feelings, not realistic setting … If many give you low ratings -> trouble Can get some useful data by asking
what they liked, disliked, where they had trouble, best part, worst part, etc. (redundant questions are OK)
Reporting the Results Report what you did & what
happened Images & graphs help people get it! Video clips can be quite convincing
Study Goals
1.What individual differences (previous software used, computer use, spatial reasoning ability, etc.) best predict performance on simple modeling tasks?
2.What usage log metrics (e.g. frequency of undo operations, frequency of camera operations, etc.) best predict performance on simple modeling tasks?
3.What specific problem do novice SketchUp users encounter most frequently on simple modeling tasks?
14
n = 54
90% students
35% architecture20% computer science10% mechanical engineering10% civil engineering25% other (art, physics, etc.)
41% never used44% novice15% intermediate
2. How much experience do you have with Google SketchUp? (place an 'X' in the appropriate box):
[ ] Expert (heavy user of inferencing, work with a large tool set, have used SketchUp extensively as part of several large projects)
[ ] Intermediate (know what inferencing is, work with some advanced tools, have used SketchUp to complete at least one project)
[ ] Novice (have played around in SketchUp, but don't claim to know much about it)
[ ] I have not used SketchUp yet. (This does not exclude you from the study!)
Study Design
1.Entry questionnaire (5 min.)
2.Mental rotation test (15 min.)
3.Video tutorials (15 min.)
4.Free exploration (10 min.)
5.Tasks (3 x 15 min.)
6.Exit questionnaire (5 min.)
Data Size
Event log data (450 MB)
Screen capture video (75 GB!)
3D models (100 MB)
Questionnaires (17 KB)
Study Goals
1.What individual differences (previous software used, computer use, spatial reasoning ability, etc.) best predict performance on simple modeling tasks?
2.What usage log metrics (e.g. frequency of undo operations, frequency of camera operations, etc.) best predict performance on simple modeling tasks?
3.What specific problem do novice SketchUp users encounter most frequently on simple modeling tasks?
Developer Hypotheses (Wrong)For the Push/Pull tool:
90% of undo operations are caused by bugs in SketchUp.
10% of undo operations are caused by difficulties with inferencing.
eye to the future Instrumenting Applications
Source: Hartmann, B., Klemmer, S.R., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A., Gee, J. Reflective physical prototyping through integrated design, test, and analysis.Proceedings of UIST 2006, October 2006
d.Tools physical prototyping captures user tests
Challenges (1/8 – from Grudin) Disparity of Work and Benefit
Groupware applications often require additional work from individuals who do not perceive a direct benefit from the use of the application
Challenges (2/8)
Critical Mass and Prisoner’s DilemmaGroupware may not enlist the “critical mass” of users required to be useful, or can fail because it is never to any one individual’s advantage to use it
Challenges (3/8)
Disruption of Social ProcessesGroupware can lead to activity that violates social taboos, threatens existing political structures, or otherwise demotivates users crucial to its success
Challenges (4/8)
Exception HandlingGroupware may not accommodate the wide range of exception handling and improvisation that characterizes much group activity
Challenges (5/8)
Unobtrusive AccessibilityFeatures that support group processes are used relatively infrequently, requiring unobtrusive accessibility and integration with more heavily used features.
Challenges (6/8)
Difficulty of EvaluationThe almost insurmountable obstacles to meaningful, generalizable analysis and evaluation of groupware prevent us from learning from experience
Challenges (7/8)
Failure of IntuitionIntuitions in product development environments are especially poor for multiuser applications, resulting in bad management decisions and error-prone design process.
Challenges (8/8)
The adoption processGroupware requires more careful implementation in the workplace than product developers have confronted
top related