low-fidelity user testing is 485, professor matt thatcher
TRANSCRIPT
Low-Fidelity User Testing
IS 485, Professor Matt Thatcher
2
Agenda Administrivia High-fidelity vs. low-fidelity
prototyping Hand out CD of user testing from past
project
3
What is Fidelity? High-fidelity (Hi-fi)
– prototypes look like their final product » computer-based prototypes (e.g., Palm Pilot prototype)» limited functionality so users can actually interact with
it» more polished and aesthetically pleasing
Low-fidelity (Low-fi)– artists rendition with many details missing
» paper-based prototypes» quick and inexpensive, provide valuable insights, but
don’t really demonstrate functionality» think about a fashion designer or an architect
4
Why Use Low-Fi Prototypes? Traditional methods take too long
– sketches --> hi-fi prototype --> evaluate --> iterate
Can simulate the prototypes– sketches --> evaluate --> iterate– sketches act as prototype– test prototypes in days vs. weeks
Kindergarten implementation skills– allows non-programmers to participate
5
Disadvantages of Hi-Fi Prototypes
Perceptions of the tester/reviewer?– formal representation indicates finished nature– users focus on appearance (color, fonts, etc.)
Time?– encourages precision– specifying details takes more time and results
in fewer design iterations
Creativity?– we tend to fall in love with our software
solutions and our program code
6
Stages of Low-Fi Prototyping Initial rough sketches of UI design ideas
– evaluate and iterate (usually w/i design team)
Storyboards– explicitly conveys the design model– evaluate and iterate (often w/i design team)
Chauffeured prototype– evaluate (with actual users) and iterate
When do you stop?– when you converge on a design model that is stable
enough to commit to software
7
Evaluating Low-Fi Prototypeswith Real Users
Constructing chauffeured low-fis– the materials– faking the interaction (wizard of oz technique)
Conducting a low-fi test with real users– preparing test materials– conducting the test (various roles)– debriefing test users– generating a report
Evaluate the results and iterate based on user feedback
8
The Basic Materials(Chauffeured Prototype)
Large, heavy, white paper (11” x 17”) 5” x 8” index cards Post-it notes of various size Tape, glue stick, correction tape Pens and markers (many colors and
sizes) Overhead transparencies Scissors, X-acto knives, etc. And more…
9
Constructing the Model Set a deadline
– don’t think too long - build it!!– it’s about getting ideas on paper
Draw a window frame on large paper Put different screen regions on cards
– anything that moves, changes, appears/disappears
Ready responses for any user action– e.g., have those pull-down menus already made
Use photocopiers to make many versions
10
Some Examples
Edit Dialog Box
UndoRepeatCutCopyPastePaste SpecialEtc…
File Edit Format Help …..
Edit Submit
NewSaveSave as…Print PreviewPrint…Etc…
File Edit
Record Added
OK
Error -4325
OK
11
12
13
14
15
16
Briefing Room Interactive (BRI)
17
Clarissa’s Embroidery Website (continued on next page)
18
19
A small part of the prototype for the SALT Center (UofA)
20
Low-Fi Computer Screen(pretty cool)
21
“Computer’s” viewTest participant’s view(with a low-fi calculator and scan-o-matic)
And a printer, too!!!!
Scenario materials
22
Constructing the Model With UI Tools / Builders (but do not
code)
23
UI Builder Tools Pen and paper Visio or Smartdraw Software drawing tools
– Microsoft Acess, Paint, PowerPoint, Word, Excel, etc.
If you construct the model with tools– print out the background screens and moving
pieces– the prototype should still be “paper-based” even
if you use software to make it look clean– hand write most of the words
24
Video “Paper Prototyping: A How-To Video”
– Nielsen Norman Group. Get some ideas and take some notes
25
Testing the Model Select your users
– look at user profile for target users– use a questionnaire to get the people you need– don’t use friends or family– get at least 3 users to evaluate the system
Prepare scenarios that are:– typical of the product during actual use– make prototype support these (small, yet broad)– use 3 scenarios of varying difficulty in the evaluation
Practice to avoid “bugs”
26
Conducting a Chauffeured Test Four testers (minimum)
– greeter - puts users at ease & gets data– facilitator - only team member who speaks
» gives instructions to the users» hands the scenarios to be completed to the users» encourages thoughts, opinions (think aloud protocol)
– computer - knows application logic & controls it» always simulates the response, w/o explanation
– observers - take notes & recommendations Typical session is 1 hour
– preparation, the test, debriefing
27
Conducting a Chauffeured Test Greet
– get consent forms filled– reads greeter’s script (or orientation script)
» provides brief background of what test is about, describes what a low-fi prototype is and its importance, assures confidentiality, put’s user at ease (voluntary, quit at any time, testing UI, etc.)
Test– facilitator reads demo script & hands written scenarios to user
» must be clear & detailed– chauffeur the prototypes
» involves user watching while another person “drives” the system– facilitator keeps getting “output” from participant
» Think Aloud Protocol - “what are you thinking right now?”– observe -> no “a-ha”, laugh, gape, etc.– possible to make changes to prototype in “real-time” if you want
28
Conducting a Chauffeured Test Debrief
– fill out post-evaluation questionnaire– discuss prepared list of debriefing topics– ask questions about parts you saw problems– gather impressions– give thanks
29
Video “Paper Prototyping: A How-To Video”
– Nielsen Norman Group. Get some ideas and take some notes
30
Evaluating Results Sort & prioritize observations
– what was important?– were there lots of problems in the same area?– assign a severity rating (0-4 scale) to each
problem identified or uncovered during the testing
Create a written report on findings– gives agenda for meeting on design changes
Make changes & iterate– note that you can make some changes “as you
test”
31
IBM’s Prototype Design Labs (Participatory Design Sessions)
“We start with low fidelity prototypes using a pencil and paper toolkit with paper window templates as well as colored sticky notes. Although our prototyping tools are can create these prototypes as quickly as pencil and paper can, we find that users are more willing to propose changes when the prototypes are low fidelity ones. We then use higher fidelity prototypes typically using IBM VisualAge. Rather than discussing what users would want in team meetings, we simply ask users directly.” (from IBM’s Ease of Use Website)
32
Microsoft Corporation “Prior to specifying the product, teams will perform
contextual inquiry with usability engineers. Early user interface evaluations may take the form of paper prototypes…Much early testing relies heavily on paper prototypes, which are used to iteratively lock in a design or a set of features and characters on the screen.”
(from the article, “Organizing Usability Work to Fit the Full Product Range”)
33
Clothes-Shopper(An Example)
Problem– contemporary approach to shopping is often slow and painful– people often have problems shopping for clothing through
catalogs (on-line or otherwise) because they are unable to visualize how they will look in the clothing
Design solution– design a UI that allow shoppers to quickly and easily visualize
how various combinations of clothing will look on them
Method– use low-fi prototyping to develop a good, general understanding
of the strengths / weaknesses of the initial UI design w/o spending a large amount of time on developing sftwr
34
Three Scenarios to Be Evaluated Moderate scenario
– Joe from Berkeley wants to buy a new shirt and sports coat to be more fashionable in school. He does not have a car today and decides to give the Clothes Shopper a try. Help him navigate through the system, buy the things he wants and leave. Although there is no budget constraint, he wants a white plain shirt and a cool brown jacket.
Moderate - difficult scenario– Sally is a regular user of the Clothes Shopper system. She wants to buy a white
gown for an upcoming prom but only has $150 as her budget. She already has an existing profile and does not wish to reenter the same information. Help Sally buy the gown and get her on a happy date!
Beginner scenario– Joanna is a new user to the Clothes Shopping system. She wants to go on the
tour first and become familiarized with the system before doing any shopping. After the tour she wants to window shop some blue shorts, but doesn't feel like buying anything today. She prefers to try on some shorts casually and leave the mall.
35
Procedure Greet Conduct the test
– facilitator, computer, observers Debrief Evaluate and analyze
– discuss what went well, what went poorly– comment on users’ reactions and comments– compile into master copy and make
suggestions for next iteration
36
37
38
39
Some Raw Notes from Observers (1/3)
Participant #2: Marlon Urias Scenario 1 (Joe from Berkeley):
Moved mouse around at first
"Is Tours clickable?"
Confused at mall entrance
"If you want to buy clothes, do you tour or go shopping?"
Easily finishes login screen
Claims he doesn’t know four of the measurements asked for, one of them "maybe" he knows
Thinks it’s "good" the male measurement screen didn’t block the profile setup screen, but was bottom part of that screen
Would be "helpful" if you knew where to get pants and shirts. Guesses it’s the Navigational Assistant. Clicks on torso.
Spent a lot of time at Navigational Assistant, confused. Pop-up notes should show up as you move around the Navigational Assistant, so you know where to click for clothes
40
After he picks shirts, realizes he needs to choose color, assuming program knows he is choosing color for shirt.
Wanted to see what results you would get so far before you finished choosing your preferences, "Would you click on the red curtain?" Since there are graphics, he hopes/assumes you would see results there. "Wish I could see it" (wanted to see Search Results updated automatically after changing one preference)
Clicked on left page flipper in Search Results window looking for clothes not there yet, before he hit Search button
Finally realized he’s supposed to hit Search button, says the term should change to "Show Product" or "See Product" or "See Progress"
Goes back to change size after seeing Search Results
He hopes preferences are kept track of, it wasn’t clear.
"Does preference change update the Search Results?" To him, Search button doesn’t equal update Search Results
Wishes he could buy the clothes before he sees it on Dressing Room model by clicking on Search Results thumbnails
Some Raw Notes from Observers (2/3)
41
Wants to see a big "Buy" button to buy clothes in Search Results window without opening Dressing Room
Wanted to delete things from Search Results window with right-click
"Can you decide to buy clothes from the thumbnail?"
Wonders what happens if model wears multiple shirts
Thinks Select All Clothes button chooses everything in Search Results
Confused how to make Dressing Room model wear clothes Would rather have right-click menus than "make assumptions"
Doesn't like Search button, "What's the difference between search and narrow down options?" (which is done in preferences)
Says that go and previous buttons are annoying.
Again there is problems with the page tabs, "are the numbers sizes or choices?" Recommended to avoid page tabs because not intuitive.
Had confusion with Select All, "does it buy it?" "Returns it to the model."
After opening the Shopping Cart, has no problem buying the items. "happy, very intuitive shopping cart window."
Some Raw Notes from Observers (3/3)
42
Summary Results (1/2)(or Merged List of Critical Incidents)
User knew how to enter system by clicking on diagram– one user had question about what “Tour” meant
Users got through logon screen pretty easily– comments about how much information the user was able or willing to reveal
(e.g., body weight, exact dimensions of clothes, etc.)
Big confusion at the main menu– did not know what to do to begin shopping (clicked on a lot of areas)– finally noticed “Navigation Assistant” after much searching and
experimentation and learned that they needed to click on it to begin– when selected garment that they wanted, many clicked on preferences
(colors) thinking they would be presented with choices» specifying search parameters was not intuitive and natural in choosing
their garment– began to play with search parameters and figured out how to use it to get
results
43
Summary Results (2/2)
Big confusion at the main menu (continued)– once the user got to the search results, two users were often confused on
how to get the garment of their choice on the model » they just clicked on the garment and expected it to appear on the
model» the design model was to have the user drag the garment to the model
– after getting the garments on the model the users expected to see a “checkout” or “buy” option that would send them to a checkout screen
» the designers intended for the users to “double-click” on the shopping cart to do this function
– after the users figured out how to get to the checkout, they bought the garments and exited the system pretty easily
Final note was that once the user learned how the system worked, they were able to quickly run through the system to perform the other scenarios
44
Discussion and Recommendations(What to Do Next?) (1/3)
Change Begin Tour to Begin Tutorial Login screen was too wordy -- the number of buttons
will be reduced to 3– New User, Use existing profile, Change Current Profile
Profile setup was fine except, for women, bra and chest size were considered redundant (eliminate one)
Navigational Assistant was not intuitive -- maybe add text or have menus automatically pop up instead of requiring a click– also need to add dresses and suit to torso section of assistant
since users asked where these would be– gray out preferences menu initially so user knows to click on
the navigational assistant first
45
Discussion and Recommendations(What to Do Next?) (2/3)
Change preferences menu– use words Show Clothes instead of Search (not very intuitive)
Change some things about the Search Results section– user did not know whether to click or drag the thumbnails to
dress the dressing room model. They decided to just require a click to minimize confusion
– change title from Search Results to Clothes Gallery Dressing room changes?
– users liked the red curtain that concealed the model until a search was performed
– liked the Shopping Cart and Trash Can options– were very confused by the Select All Clothes option
» designers just dropped the idea to reduce confusion
46
Discussion and Recommendations(What to Do Next?) (3/3)
Dressing room changes?– so users can just select clothing on the model and put it in the
shopping cart or trash can as desired– add a Checkout button so the user can go ahead and buy what
s/he has in his / her shopping cart (many users asked if such an option existed)
A user also requested if it were possible to have a counter showing how many pieces of clothing were shown in the Search Results window (soon to be called the Clothes Gallery)– this function will be added
47
High-Fi Revisions
48
49
50
Hints for Low-Fi User Testing(from “Waving Magic Wands”)
Interaction techniques to optimize lo-fi testing– establish a paper world
» e.g., low-fi computer screen, printer, keyboard, etc.
– provide a pointing toy– use input and output interaction– prototype all relevant elements– let test participants design (on the spot)– facilitate fun!
51
Milestones 2 and 3 Milestones 2
– look at initial sketches from Milestone 1, discuss with team, and converge on a design model
– develop low-fi storyboards to support scenarios
Milestone 3– develop a low-fi chauffeured prototype for user testing– identify and recruit 3 evaluators (preferably real users)– prepare and conduct a chauffeured user test– discuss the results (what test users said and did, + or -)– discuss likely changes to design for the next design
iteration
52
Milestone 3 Results
– raw data» what each observer writes down during user testing,
video taping, audio taping, etc.
– merged list of critical incidents (and severity ratings)
» get rid of redundant observations, make clarifications, etc.
– textual summary of results Discussion and recommendations
– specific recommendations to address potential usability problems uncovered during testing
53
Low-Fidelity Prototypes Paper-based prototypes
– focus on high-level aspects of UI– see if the design model matches the user model– not concerned about details (e.g., color, fonts, alignment)
Low-fis help answer certain questions– are the various UI screens organized well?– does the layout of each screen make sense?– is the navigation natural and intuitive?– is the wording/terminology user-centered?– are important tasks missing?– are the metaphors/mappings effective and appropriate?
54
Disadvantages of Low-Fis Paper mock-ups don't go far enough
– not realistic interaction with the interface» how would you show a drag operation?» how will users interact with I/O devices to be used» doesn’t provide realistic response time (how measure
performance)
– has no functionality– evaluators may scoff at low-tech prototypes– little feedback about details (appearance like
colors, fonts, alignments, etc.) of the design– don’t know if design is technically feasible
55
Summary Use prototypes to involve the user Advantages of low-fi prototypes
– cheap and quick to develop and evaluate– requires few technical skills– leads to fast iterations (and more iterations)– users and designers less likely to see design as fixed
56
E-Groceries Case Study