How to measure usability

Nikolaj Aalbæk Jensen@brugervenliglinkedin.com/in/brugervenlignikolaj.jensen@visma.com

• Why oh why?• Why should we measure usability?

• The boring stuff…• ISO 9241• Statistics 101

• So many choices…• System Usability Scale• Single Usability Metric• Standardized User Experience

Percentile Rank Questionnaire

Holy f… that is a lot of stuff• Heaven and Hell…

• The pros and cons of measuring usability

• Famous last words…• Summary• Recommended reading

Why oh why?Why should we measure usability?

http://www.skai.gr/files/1/jo/slide_225124_963376_free.jpg

• Numbers enables us to communicate UX results in a way that engages C-level management • Provides us with a way to measure the impact of changes

• Great way of introducing UX KPIs in your organisation, which makes it easier to champion your work and increases visibility

• Makes it easier to include usability requirements/benchmark in project specifications and public procurements

Why should we measure usability?

The boring stuff…The foundation

http://www.skai.gr/files/1/jo/slide_225124_963376_free.jpg

ISO 9241 – Ergonomics of human-system interaction

http://toplogos.ru/images/logo-iso.png

"The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use

ISO 9241 – Ergonomics of human-system interaction

ISO 9241 – Ergonomics of human-system interaction

task completion rateressources spent to solve a task subjective user satisfaction

EffectivenesEfficiencySatisfaction

Statistics 101

https://img0.etsystatic.com/100/0/6005823/il_fullxfull.832106216_e8ue.jpg

• Normal distributionhttps://www.youtube.com/watch?v=772_n15Ke9Q

• Standard deviationhttps://www.youtube.com/watch?v=nxKBfcwEulg

• Confidence intervalhttps://www.youtube.com/watch?v=tFWsuO9f74o

Statistics 101

http://kedoff.com.ua/uploads/posts/2014-11/1415786610_1.jpg

So many choices…A selection of methods

QUIS Questionnaire for User Interaction Satisfaction

SUMI Software Usability Measurement InventoryPSSUQ Post-study System Usability QuestionnaireSUS Software Usability ScaleASQ After-Scenario QuestionnaireSEQ Single Ease QuestionSMEQ Subjective Mental Effort QuestionER Expectation RatingsUME Usability Magnitude EstimationWAMMI

Website Analysis and Measurement Inventory

SUPR-Q

Standardized Universal Percentile Rank Questionnaire

CSUQ Computer System Usability QuestionnaireUSE Usefullness, Satisfaction, and Ease of useUMUX Usability Metric for User eXperienceHQ Hedonic QualityACSI American Customer Satisfaction IndexNPS Net Promoter ScoreCxPi Forrester Customer Experience IndexTAM Technology Acceptance ModelSUM Single Usability Metric

Standardized methodsQUIS Questionnaire for User Interaction

SatisfactionSUMI Software Usability Measurement InventoryPSSUQ Post-study System Usability QuestionnaireSUS Software Usability ScaleASQ After-Scenario QuestionnaireSEQ Single Ease QuestionSMEQ Subjective Mental Effort QuestionER Expectation RatingsUME Usability Magnitude EstimationWAMMI

Website Analysis and Measurement Inventory

SUPR-Q

Standardized Universal Percentile Rank Questionnaire

CSUQ Computer System Usability QuestionnaireUSE Usefullness, Satisfaction, and Ease of useUMUX Usability Metric for User eXperienceHQ Hedonic QualityACSI American Customer Satisfaction IndexNPS Net Promoter ScoreCxPi Forrester Customer Experience IndexTAM Technology Acceptance ModelSUM Single Usability Metric

• Created in 1986 by John Brooke

• 10 item post-session questionnaire • 5 negative biased statements• 5 positive biased statements• 5 point Likert scale

• Yields a score between 0 and 100 – average is 68

• Valid results with 12 responses

• The most reviewed and tested method available

• Free to use

System Usability Scale (SUS)

System Usability Scale (SUS)

http://uxpajournal.org/wp-content/uploads/pdf/JUS_Bangor_May2009.pdf

System Usability Scale (SUS)

http://www.measuringu.com/sus.php

Cons• You cannot use responses from individual items, only the total score• You only get an overall score• It cannot tell what or where something is wrong

Pros• Easy to use and administer• Good for tracking how changes impacts the usability• Easy to understand as a KPI• Can be used to assess when you should include users for qualitative feedback

System Usability Scale (SUS)

• SUS - A quick and dirty usability scalehttp://hell.meiert.org/core/pdf/sus.pdf

• Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale

http://uxpajournal.org/wp-content/uploads/pdf/JUS_Bangor_May2009.pdf

• SUS: A Retrospectivehttp://uxpajournal.org/wp-content/uploads/pdf/JUS_Brooke_February_2013.pdf

System Usability Scale (SUS)

One downTwo to go

https://cdn3.vox-cdn.com/thumbor/ST3W8A3Kum8HlwddNHoF07WWrhk=/0x0:1920x1080/1600x900/cdn0.vox-cdn.com/uploads/chorus_image/image/46147742/cute-success-kid-1920x1080.0.0.jpg

• Created in 2005 by Jeff Sauro and Erika Kindlund

• Based on ISO 9241 definition of usability • Task based measurement • Data driven results

• Free to use

Single Usability Metric (SUM)

Single Usability Metric (SUM)

https://www.measuringu.com/papers/HCII2005_sauro_kindlund-V9.pdf

Cons• Requires additional resources to administer• Can (almost) only be administered during usability testing sessions• Requires an unknown, and potentially relatively large, portion of data sets• Requires more knowledge about statistics than SUS

Pros• Task based data• Good at identifying specific problem areas• Closely related to ISO definition of usability• Very little subjective data

Single Usability Metric (SUM)

• A Method to Standardize Usability Metrics Into a Single Score https://www.measuringu.com/papers/p482-sauro.pdf

• Parts of the SUM: a case study of usability benchmarking using the SUM Metric https://autodeskresearch.com/publications/summetric

• 10 Things To Know About The Single Usability Metric (SUM) http://www.measuringu.com/blog/sum.php

Single Usability Metric (SUM)

Just one more

http://cdn.meme.am/images/5188389.jpg

• Created in 2011 by Jeff Sauro

• 13 item questionnaire

• Measures Usability, Trust & Credibility, Appearance, and Loyalty

• You also get both a SUS and a NPS score

• Results are comparable to a large database

• Pricing between $499 and $2999

Standardized User Experience Percentile Rank Questionnaire (SUPR-Q)

http://www.suprq.com/images/suprq-logo.png

Standardized User Experience Percentile Rank Questionnaire (SUPR-Q)

Cons• Full license (access to database) pricing starts at $1999• Is not as thoroughly tested and reviewed as other methods as it is a

relatively new method• Requires a bit more knowledge about statistics than SUM and SUS

Pros• Better insight than SUM and SUS, overall• Also yields a SUS and a NPS score• Results are comparable to large database

Standardized User Experience Percentile Rank Questionnaire (SUPR-Q)

• SUPR-Q: A Comprehensive Measure of the Quality of the Website User Experience

http://uxpajournal.org/supr-q-a-comprehensive-measure-of-the-quality-of-the-website-user-experience/

• The Essential Elements Of A Successful Websitehttp://www.measuringu.com/suprq.php

• The SUPR-Q websitehttp://www.suprq.com/

Standardized User Experience Percentile Rank Questionnaire (SUPR-Q)

http://foto-video-podarok.ru/foto/kostyum-cherta-na-novyij-god-7.jpg

Heaven and Hell…The pros and cons of measuring usability

• It’s just numbers – further insight requires other methods

• Unreliable, if used incorrectly

• Can only tell you if something is wrong, and not why

CONS

• Really good at measuring the impact of changes

• Great for communicating results - especially with C-level management

• Good way of having KPIs

• Cheap way of identifying/validating problem areas

• Great way of including UX requirements in project specifications / public procurements

PROS

Famous last words…

http://img.memecdn.com/famous-last-words_o_1527913.jpg

• Try out different methods and choose the one that works for you

• Use quantitative methods primarily to measure the impact of changes

• Champion UX KPIs in your organisation for improved visibility of your work

• Use benchmarks actively in project specifications

Summary

Recommended reading

www.measuringu.com

visma.dk/webevaluering

@brugervenlig linkedin.com/in/brugervenlig nikolaj.jensen@visma.com

visma.dk/brugervenlighed

Feel free to reach out…