lean usability
DESCRIPTION
Dre and I talk about how usability testing evolved at Meetup, and the techniques we use today.TRANSCRIPT
![Page 1: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/1.jpg)
LEAN USABIILTY @
PRESENTED AT NYC LEAN STARTUP MEETUP FEB 9, 2010
ANDRES GLUSMAN @glusman ANNA DEYOUNG
![Page 2: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/2.jpg)
OVERVIEW OF USABILITY TESTING AT MEETUP
![Page 3: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/3.jpg)
340 TESTS IN LAST 18 MONTHS
TEST 2-3 DAYS A WEEK
EVERYTHING USER-FACING GETS TESTED
ON PACE TO DO 400+ SESSIONS THIS YEAR SUPPORTING 6-8 PROJECT TEAMS
WILL COST ABOUT AS MUCH AS DOING 4 OUTSOURCED USABILITY PROJECTS
![Page 4: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/4.jpg)
WHY USABILITY TESTING MATTERS
![Page 5: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/5.jpg)
PRODUCT DEVELOPMENT IS AN ART
THAT INVOLVES MANY PEOPLE
& SUFFERS FROM THE MALKOVICH BIAS
![Page 6: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/6.jpg)
MALKOVICH BIAS THE TENDENCY TO BELIEVE THAT EVERYONE
USES THE WEB LIKE YOU DO
![Page 7: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/7.jpg)
WATCHING PEOPLE USE THE STUFF YOU BUILD IS THE BEST WAY TO CONFRONT THE
MALKOVICH BIAS
![Page 8: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/8.jpg)
HOW WE GOT HERE….
![Page 9: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/9.jpg)
~ FIRST ITERATION ~ DIY TESTING
![Page 10: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/10.jpg)
![Page 11: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/11.jpg)
DIY EXPERIMENT • RECRUITED OFF OF CRAIGSLIST
• 1 PRODUCT PERSON ENGAGED USERS IN A FEW DIRECTED TASKS THEN SYNTHESIZED FINDINGS INTO A PRESENTATION
VERDICT • LOW COST
• LEARNED INTERESTING FINDINGS • HARD TO REALLY CONVEY USER BEHAVIOR TO REST OF TEAM
• WE DID NOT MAKE INTO ONGOING ROUTINE
![Page 12: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/12.jpg)
~ SECOND ITERATION ~ PROPOSALS FROM OUTSIDE FIRMS
![Page 13: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/13.jpg)
LOOKED AT FORMAL USABILITY TESTS • RECEIVED PROPOSALS FROM SEVERAL USABILITY TESTING FIRMS
• $32K - $53K FOR 2 DAYS (16 TESTS)
• INVOLVED USE OF TESTING FACILITY • PROPOSED DELIVERABLE WAS A REPORT/RECOMMENDATION
VERDICT • HELD OFF
![Page 14: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/14.jpg)
~ THIRD ITERATION ~ ACCIDENTALLY DISCOVERED LEAN USABILITY
![Page 15: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/15.jpg)
“I’LL LET YOU INTO THE BETA PROGRAM IF YOU AGREE TO LET MY TEAM WATCH YOU SET IT UP”
‐CHANCECONVERSATIONTHATSETFOUNDATIONFORCURRENTPROGRAM
![Page 16: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/16.jpg)
Table
Projecteduser’sscreenontowall
user
moderator
developer
developer
developer
compu
ter
CONFERENCEROOMATMEETUP
HEADQUARTERS
![Page 17: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/17.jpg)
“THAT WAS HUMBLING” - A DEVELOPER AFTER THE SESSION
![Page 18: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/18.jpg)
SHIFTED THE CONVERSATION FROM HOW I WOULD USE THE WEBSITE TO HOW THIS
USER USES THE WEBSITE
![Page 19: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/19.jpg)
ACCIDENTAL EXPERIMENT • USER GOT TO JUMP TO FRONT OF LINE IN BETA PROGRAM IN RETURN
FOR PARTICIPATION • HELD TEST IN CONFERENCE ROOM WITH PROJECT TEAM IN THE ROOM
• PROJECTED USER’S SCREEN ON WALL • INFORMALLY MODERATED USER TO VERBALIZE THOUGHT PROCESS
• TEAM DISCUSSION AFTER THE SESSION
VERDICT • LOW COST • USABILITY ISSUES WERE OBVIOUS TO ENTIRE TEAM
• REALIZED THIS APPROACH COULD BE POWERFUL
![Page 20: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/20.jpg)
~ FOUTH ITERATION ~ FIGURE OUT A REPLICABLE ROUTINE
![Page 21: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/21.jpg)
HIRED SOMEONE TO TEACH US TO FISH BUT…
FOCUSED ON MINIMUM VIABLE SOLUTION & STRIPPED OUT AS
MUCH COST AS POSSIBLE
![Page 22: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/22.jpg)
HELD SESSION AT MEETUP’S CONFERENCE ROOM AND HAD TEAM
SIT BEHIND THE TESTER
![Page 23: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/23.jpg)
Table
Projecteduser’sscreenontowall
user
moderator
compu
ter
CONFERENCEROOMATMEETUP
HEADQUARTERS
developer
developer
developer
developer
developer
developer
developer
![Page 24: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/24.jpg)
FREAKED OUT TESTERS, BUT WORKED WELL ENOUGH
![Page 25: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/25.jpg)
~ FIFTH ITERATION ~ COMMITTED TO USABILITY TESTING
& MADE IT ROUTINE
![Page 26: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/26.jpg)
• CTO MANDATED THAT ALL USER-FACING PRODUCT BE TESTED AS PART OF DEVELOPMENT PROCESS
• PURCHASED SOFTWARE TO HELP SHARE/RECORD SESSIONS
• TEAMS VIEWED ALL SESSIONS LIVE
• IMMEDIATELY FOLLOWED UP WITH A DISCUSSION
• NO REPORTS
• DID OUR OWN RECRUITING OUT OF THE COMMUNITY TO TEST MEMBER EXPERIENCE
• USED OUTSIDE RECRUITERS TO TEST EXPERIENCE FOR PEOPLE UNFAMILIAR WITH MEETUP
![Page 27: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/27.jpg)
PURCHASED SOFTWARE TO HELP
• USE MORAE ($1500)
• PC BASED
• ALLOWED US TO BROADCAST PICTURE IN PICTURE
• ALLOWED FOR SESSIONS TO BE RECORDED AND LIVE BROADCASTED TO OTHER ROOM
• ALLOWED US TO MAKE HIGHLIGHT VIDEOS
• LOTS OF FUNCTIONALITY (BUT WE ONLY USE A FRACTION OF IT)
![Page 28: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/28.jpg)
Table
Projecteduser’sscreenonwall
developer
developer
developer
developer
developer
developer
DESK
MOD USER
computer
MAINCONFERENCEROOM
SMALLCONFERENCEROOM
![Page 29: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/29.jpg)
~ 6TH THRU NTH ITERATIONS ~ KEEP EXPERIMENTING WITH RECRUITING,
MODERATING, & SHARING
![Page 30: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/30.jpg)
EXPERIMENT WITH COMMANDO TESTING • TAKE A LAPTOP WITH SILVERBACK SOFTWARE TO A COFFEE SHOP
• GET PERMISSION FROM THE MANAGER
• OFFER FANCY CUP OF COFFEE FOR A TEN MINUTE TEST
VERDICT • CHEAP • DO NOT NEED TO PLAN AHEAD
• FEEL LIKE A BAD ASS …UNTIL YOU GET TO THE COFFEE SHOP
• WASTE A LOT OF TIME WRANGLING PEOPLE
![Page 31: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/31.jpg)
EXPERIMENT WITH SILVERBACK SOFTWARE • $50 (FREE 30-DAY TRIAL) • MAC ONLY
• LIGHTWEIGHT • EDIT VIDEO USING IMOVIE
• WE CURRENTLY USE IT FOR COMMANDO TESTING IN THE FIELD AND IPHONE TESTING
VERDICT • HIGHLY RECOMMEND AS A STARTING POINT,
BUT ONLY TESTS ON MAC OS
![Page 32: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/32.jpg)
OTHER EXPERIMENTS
• LOW COST RECRUITING (FROM THE BUILDING)
• SCREENCASTING WITH CHAT (USING DIMDIM)
• TIVO-IZING / TIME SHIFTING TESTS TO SAVE DEVELOPER’S TIME
![Page 33: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/33.jpg)
TIPS FOR DOING YOUR OWN LEAN USABILITY
![Page 34: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/34.jpg)
9 LEAN USABILITY PRINCIPLES • LEARN TO LOVE ERROR
• LOOK FOR BOULDERS IN THE ROAD
• SUBSTITUTE FREQUENCY FOR PRECISION
• STRIP OUT COSTS WHEREVER POSSIBLE
• THINK MINIMUM VIABLE PRODUCT PROCESS
• IN RECRUITING, BE PREPARED TO TRADE MONEY FOR TIME
• BASIC MODERATION TECHNIQUES CAN GO A LONG WAY
• EXPOSE TEAM TO USERS
• TAKE NOTES & HAVE DISCUSSIONS (DON’T WRITE REPORTS)
![Page 35: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/35.jpg)
Recruiting DO
1. Practice with your friends and family
2. Have 1 person own usability testing for the company
3. Create a screener listing minimum user requirements – completed over phone or online (guidelines available online)
4. Consider using a vetted recruiter if you can afford it or for specialized audiences
5. Allow a week to schedule in-house sessions
6. Follow up with testers the day before 7. Offer incentive
DON’T 1. Rely on the kindness of friends to
volunteer every time you need testing 2. Use the same tester(s) for different
iterations of the same project
3. Explain too much about the purpose of the test when recruiting
4. Bother the participant with too many calls or emails
5. Be afraid to turn them down if they seem “weird”
6. Forget to thank them! (They may become evangelists)
![Page 36: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/36.jpg)
Moderating DO
1. Practice the test ahead of time and write down questions
2. Create a scenario to test an experience, not just a feature
3. Be aware of your assumptions beforehand as not to project them on the participant
4. Spend a few minutes getting to know the participant
5. Ask general, open-ended questions first and then probe for details
6. Let participant explain things in their own words and avoid correcting them
DON’T 1. Start the test before they sign a
consent/release form (example available online)
2. Forget to remind participant that you are testing the product, not the participant
3. Answer their questions or explain things about the project to help them along
4. Use terms from your website or industry jargon when asking questions
5. Accept just “yes” or “no” for an answer
![Page 37: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/37.jpg)
A few great moderating questions
When first looking at project • What are you looking at here?
• What can you do?
As test progresses
• Show me how you would ____________
• What do you expect will happen when you _____?
Wrapping up • What did you find confusing?
• How would you describe what you did here today?
• What, if anything you saw today, could you imagine using at home? How?
![Page 38: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/38.jpg)
Sharing DO
1. Record the sessions 2. Take notes for immediate
discussion afterward 3. Debrief with project teams
the same day 4. Provide notes and video for
those who could not be there
5. Protect the privacy and dignity of the participant
DON’T
1. Assume everyone saw the same things you did in a test
2. Interpret user reactions if you don’t have the data to support it
3. React too strongly to any single test
4. Just cherry-pick your pet issues that might have come up
![Page 39: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/39.jpg)
Testing day checklist (office test) 1. Set up item to be tested 2. Start recording and sharing processes 3. Greet participant and bring them to the private testing room 4. Collect release form, explain test and build rapport 5. Conduct the test, reminding them to think out loud and try to do things as
if they were using their own computer at home (or work) 6. Have a channel of communication between tester and remote observers
(chat or text messaging) for emerging questions 7. If testing multiple projects/scenarios, introduce each one separately, one
after the other 8. Reserve time at the end to answer questions and explain your product, if
they ask 9. Thank the participants and provide incentive 10. Stop recording and sharing processes 11. Meet with team to discuss what you saw 12. Adjust test if necessary for next session
![Page 40: Lean Usability](https://reader034.vdocuments.us/reader034/viewer/2022051608/545150f6af795902348b6b63/html5/thumbnails/40.jpg)
Do it!
• Anyone can run a usability test with the right preparation and attitude
• There is lots of advice available online • Learn from mistakes and evolve your methods • Ask people who have done it before to help you
out • Team up & test each other