[webinar] with ux pioneer jeff sauro: measuring ux & usability

Download [Webinar] with UX Pioneer Jeff Sauro: Measuring UX & Usability

Post on 08-May-2015




3 download

Embed Size (px)


In this webinar we will explain the advantages of the 3 most recognized methods for measuring Usability & User Experience of a product or service: SUS (System Usability Scale), Supr – Q and UX Score. - How do you measure it? - What do you need to ask users to find out which experience they had? - Which factors influence their experience? - What methods allow you to benchmark the experience you offer compared to competitors?


  • 1.THE FINAL DEBATE ON How to Measure Usability & User Experience

2. 2 Our Speakers Lee Cooper Head of UX Research, UserZoom Limited UK Je Sauro Founding Principal, measuringusability.com Tim Bosenick Managing Director, GfK Sirvaluse Consulting Introduction by: Javier Darriba Co Founder & Co CEO, UserZoom #uzwebinar 3. 3 Quick Housekeeping Chat box is available, if you have any questions Or tweet your questions using #uzwebinar There will be time for Q&A at the end of the webinar We will be recording the webinar for future viewing All attendees will receive a copy of the slides/ recording #uzwebinar 4. 4 About UserZoom #1 Customer and User Experience Research and Analytics Platform vRemote Usability Testing vVoice of the Customer Studies vInformation architecture research tools: vCard Sorting vTree Testing vClick testing vSurvey Tool Our productsOur holistic approach Listen MeasureResearch Test & act 5. 5 Highlighted Customers #uzwebinar 6. 6 Agenda UX measurement: Performance metrics Vs Satisfaction metrics 5 mins 3 Ways of measuring the subjective part of UX 1. Scale usability system (SuS) 10 mins 2. Supr Q 10 mins 3. UX Score 10 mins Q&A (10 min) 7. 7 User Experience Measurement Performance Metrics Satisfaction Metrics ? - Success rate - Number of clicks - Time on task - - SuS - Supr-Q - UX Score - NPS #uzwebinar 8. 8 Measuring Usability & UX: SuS Lee Cooper @userzoom_uk 9. 9 Whatisit? Wherediditcomefrom? WhydoIlikeit? SUS #uzwebinar 10. 10 ItsaquesAonnaire consisAngoften statements. Eachhasave-pointscale thatrangesfromStrongly DisagreetoStronglyAgree. ThereareveposiAve statementsandve negaAvestatements,which alternate. Ityieldsasingleusability scoreonascaleof0-100. #uzwebinar What is it? 11. 11 JohnBrooke1986,Digital EquipmentCo. Toprovideuswitha measureofpeoples subjecAvepercepAonsof theusabilityofasystem Toallowustodosoin theveryshortAme availabletousduringan evaluaAonsession. Itwasprobablysomething thatcouldbeusedbyother organizaAons John Brooke. SUS: A Retrospective (2013) #uzwebinar Where did it come from? 12. 12 ItsQuick&Not-So-Dirty ItsTechnologyAgnosAc ItsFree ItWorksWithSmallSampleSizes ItsNotPerfect,but ItsOld #uzwebinar Why do I like it? 13. 13 Itsreliable Ithasbeenshowntobemorereliablethanmanyhomegrown andcommerciallyavailablequesAonnaires. Itsvalid IthasproventobeeecAveatdisAnguishingbetweenusable andunusablesystemsatleastaswellas,ando`enbeaerthan, otherquesAonnaires. ItalsocorrelateshighlywithotherquesAonnaire-based measurementsofusability. Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) #uzwebinar Its Quick & Not-So-Dirty 14. 14 Microwave Bank: NCR Knowledge Lab Symbian: UIQ/UIP Smartphones 3: 3G Services Nokia: Smartphones Its Technology Agnostic 15. 15#uzwebinar Its Free 16. 16 Itworkswithsmalland largesamplesizes. Itallowsyouto measuretheperceived usabilityofasystem withasmallsampleand becondentthatyouve gotagoodassessment. Tullis & Stetson. A Comparison of Questionnaires for Assessing Website Usability (2004) #uzwebinar It Works With Small Sample Sizes 17. 17 ItsnotdiagnosAc Itdoesnottellyouwhatmakesasystemusableornot. ItsomeAmesconfuses Itsscoresarenotpercentages,despitereturningavalue between0and100. ItsalternaAngitemsmightbeproblemaAc Itcanleadtorespondingandscoringerrors. Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) John Brooke. SUS: A Retrospective (2013) #uzwebinar Its Not Perfect, but 18. 18 There'sawealthof informaAonabout itsusealongwitha bodyofnormaAve data. Ithasstoodthe testofAme Bangor, Kortum, and Miller. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale,)(2009) Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011) #uzwebinar Its Old 19. 19 Measuring Usability & UX: Supr-Q Je Sauro @MeasuringU 20. Usability & Satisfaction SUPR-Q Jeff Sauro | Measuring Usability LLC 20 21. 21 Measuring Usability LLC #uzwebinar ISO 9241 pt 11 Denition of Usability 22. 22 Usability + Credibility & Trust + Appearance + Loyalty 75 Candidate Items Measuring Usability LLC #uzwebinar What makes an Eective Website? 23. 23 Usability This website is easy to use. It is easy to navigate within the website. Credibility (Trust, Value & Comfort) This website keeps the promises it makes. I feel confident conducting business with this website. Loyalty How likely are you to recommend this website to a friend or colleague? I will likely visit this website in the future. Appearance I found the website to be attractive. The website has a clean and simple presentation. NPS Explains 94% of SUS Standardized User Experience Percentile Rank-Questionnaire Measuring Usability LLC #uzwebinar SUPR-Q Items & Factors 24. 24 Retail Amazon eBay Zappos Apple Target JC Pennys Wal-Mart Hermes Boston Proper Container Store Crate & Barrel Crumpler Oriental Trading Sears LL-Bean Michaels Payless Shoes Pier 1 Imports Macys Netflix BestBuy OfficeMax OfficeDepot Airlines United Airlines Frontier Airlines American Airlines Southwest Jet Blue Delta Frontier 3rd Party Travel Travelocity Expedia Orbitz Kayak Government Illinois Colorado California CDC USA.gov New York State Cell-Phone Carriers AT&T Wireless Verizon Sprint T-Mobile Floral Service FTD 1800 Flowers ProFlowers 3rd Party Automotive Edmunds Kelly Blue Book Cars Autotrader Financial Services PayPal Fidelity TD Ameritrade Vanguard Bellco US Bank Chase eTrade Scottrade Wells-Fargo News & Information Wall Street Journal CNN FoxNews New York Times IMDB Yelp Craigslist Motley Fool Yahoo Social Networking Facebook Twitter LinkedIn Flickr Relationship/Dating Match.com eHarmony PlentyofFish.com Hotels Hyatt Hilton 200+ Websites Total, 10,000 responses (30 to 200 per website)Measuring Usability LLC #uzwebinar Selection of Websites in SUPR-Q Database 25. 25 A score of 50% = average score Measuring Usability LLC #uzwebinar Normalized database 26. 26 Measuring Usability LLC #uzwebinar Scoring 27. 27 30% 41% 92% 38% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Industry Average = 47% SUPR-Q Health Care Industry 28. 28 Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance Aetna 30% 22% 35% 33% 34% BCBS 41% 26% 51% 45% 33% Kaiser 92% 70% 91% 98% 79% United Healthcare 38% 25% 48% 39% 33% All Healthcare 47% 31% 55% 52% 40% Overall, scores for the industry are a bit below average, with Kaiser being a clear leader, especially in trust and loyalty Healthcare Websites Measuring Usability LLC #uzwebinar SUPR-Q subscale Scores 29. 29 85%96%66% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Industry Average = 82% Measuring Usability LLC #uzwebinar SUPR-Q Retail Banking 30. 30 Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance Bank of America 66% 67% 70% 54% 52% Chase Bank 96% 97% 93% 94% 83% Wells Fargo 85% 88% 81% 84% 66% All Banking 82% 84% 81% 77% 67% Overall, scores for the industry are above average, with Chase Bank leading the pack, especially in usability and appearance Retail Banking Websites Measuring Usability LLC #uzwebinar SUPR-Q subscale Scores 31. 31 Feb2011 Oct2011 NPS 73% -7% Credibility 99.8% 62% Usability 99.7% 73% Appearance 99.7% 80% Measuring Usability LLC #uzwebinar Netix Case Study 32. 32 MeasuringUsabilityLLCisausabilityandquanAtaAveresearchrmbasedin Denver,ColoradoUSAfocusingonthestaAsAcalanalysisofhumanbehavior andquanAfyingtheuserexperienceforFortune1000companies. www.MeasuringUsability.com Twitter @MeasuringU August 20-22nd 2014 Denver, CO Sponsored by UserZoom DenverUX.com Measuring Usability LLC #uzwebinar Measuring Usability LLC 33. 33 Measuring Usability & UX: UX Score Tim Bosenick @GfK_UX 34. 34 Development 2011-2014 GfK UX Score #uzwebinar GfK 2014 | UX Score Development 34 35. In 2011, GfK SirValUse and the GfK Verein conducted the basic research for the UX Score metric. Desk research Operationalisation & Design Draft model of User Experience Reduction to a core item set Eight different test objects 50 face-to-face interviews each Final examination of the model Four different test objects 400 face-to-face interviews Three qualities: task-oriented, self- oriented, aesthetic Standardised instrument for the measurement of UX Validation study I Validation study IIPreparation Model of UX #uzwebinar GfK 2014 | UX Score Development 35 36. Task-oriented qualities Learnability / Operability Self-oriented qualities Product Fit / Inspiration Aesthetic qualities Look & Feel Task-oriented qualities refer to the operation and learnability of the way in which the product works. User-oriented qualities refer to feelings of belonging and the stimulation of the user that the product engenders. Product-related aspects refer to the features of the product such as how it looks, feels and sounds. These are the three dimensions of UX that we found and validated with the rst and the second validation study in 2011. #uzwebinar GfK 2014 | UX Score Development 36 37. In 2013, we conducted a third validation study in Germany, US and in China. Collection of all relevant metrics to put in our questionnaire, like: SUS, NPS, AttraktDiff, Price Challenger Creation of item substitutes for the 'difficult' items. The first part was the validation study in Germany. We conducted 100 face-to-face interviews of 60 minutes in our labs in Hamburg. After the analysis of the German data confirmed the UX Score metric, User Centric and GfK China conducted the identical study in their countries. After all the lab sessions had been conducted, we started with the online validation. The questionnaire was much shorter than the in the lab (see session structure). Validation in Germany Validation in the US and China Preparation Online Validation (2014) #uzwebinar GfK 2014 | UX Score Development 37 38. Test Stimuli with a standardized setup for all three countries iPad 4 Samsung Galaxy Note 10