soft performance - measuring
TRANSCRIPT
![Page 1: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/1.jpg)
Soft PerformanceMeasuring
![Page 2: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/2.jpg)
me
Dimiter Simov - Jimmy usability practitioner
product experience for SAP HANA Cloud Platform @ SAP
goal make you think about metrics when designing and testing
![Page 3: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/3.jpg)
Performance
a task or operation seen in terms of how successfully it is performed
pay increases are now being linked more closely to performance
the capabilities of a machine, product, or vehiclethe hardware is put through tests which assess the performance of the processor
Source: Google define
![Page 4: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/4.jpg)
recall ISTA 2013: performance has a soft sidetext and formattingFitts’ lawlayout and structureuser success and engagementpresentation of progressaesthetics 1 x 1.618 – the golden ratio
![Page 5: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/5.jpg)
1. avoid messages, especially modal onesDON’T WRITE MESSAGES, DESIGN INTERACTIONS
2. if you have to give a message, make sure it is obvious who shows itwhat happenedwhywhat users can do about it
3. be practical
recall ISTA 2014: about messages
![Page 6: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/6.jpg)
about measuring soft performance
look at how soft performance can be measured
success rates, satisfaction scores, orientation
higher scores predict better performance; not always
big data – a glimpse
![Page 7: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/7.jpg)
trinity
effectiveness
efficiency
satisfaction
![Page 8: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/8.jpg)
effectiveness | efficiency | satisfaction
![Page 9: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/9.jpg)
effectiveness
![Page 10: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/10.jpg)
effectiveness = result = success
shows whether users accomplish their tasks
the basic metric
1 or 0: users either succeed or not (could as well be a fraction)
![Page 11: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/11.jpg)
measure success rate break down user journey into tasks and subtasks
assign 1 = success, 0 = failed
count and sum for each user and task
average across tasks, users, and user groups
![Page 12: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/12.jpg)
report success rate aim for 100% but 70-80% is a great score
4 users out of 5 succeeded ≠ 80% success
confidence interval calculator[1]
[1] calculator: http://www.measuringu.com/wald.htm
![Page 13: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/13.jpg)
sample
report
![Page 14: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/14.jpg)
effectiveness | efficiency | satisfaction
![Page 15: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/15.jpg)
efficiencyorientation
time on task
effort
errors
learnability
![Page 16: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/16.jpg)
efficiency = productivity
ratio of what you get to what you invest
goal = maximum result with minimum effort and time
![Page 17: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/17.jpg)
time
on
task
![Page 18: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/18.jpg)
time on task = speed
shows how quickly usersperform tasks
measure by the clock relative: compare to another system or version
time on task
![Page 19: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/19.jpg)
subjective time matters
example 10 websites: users performed their own personal tasks no two users performed the same tasks on any site measured perceived speed and actual speed per task
![Page 20: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/20.jpg)
the faster website
about.comactual speed: 8 secondsperceived speed: slowest
amazon.comactual speed: 36 secondsperceived speed: fastest
![Page 21: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/21.jpg)
perceived speed related to success
when a site is slow, if users succeed, they feel it fast
when a site is fast, if users fail, they feel it slow
![Page 22: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/22.jpg)
estimate task times
rule of thumb
measure your timemultiply by 3 to 5
![Page 23: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/23.jpg)
effort
![Page 24: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/24.jpg)
effort
shows the consumption of resources needed to perform the tasks
involves counting: clicks,steps, calories, etc. effort
![Page 25: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/25.jpg)
clicks and performance
fewer clicks usually mean better performance:nobody likes “clicky” UIs
clicks predict about 25% of task time [1]
[1] http://www.measuringu.com/blog/click-clock.php
![Page 26: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/26.jpg)
KLM = Keystroke-Level Model
“predicts how long it will take an expert user to accomplish a routine task without errors” [1]
covers the straight case
[1] https://en.wikipedia.org/wiki/Keystroke-level_model
![Page 27: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/27.jpg)
KLM example: dropdown versus radio buttons
m thinkp point the dropdown arrowb click the arrowr dropdown expands = 0m look at the listp point the item neededb click the itemr dropdown collapses = 0m look at the value to see it is selectedp move to the Submit buttonb click
m look at the listp point the item neededb click the itemp move to the Submit buttonb click
m p b r m p b r m p b = 7.2 sec m p b p b = 3.6 sec
m thinkp pointb click in the boxm look at the list againk type (between 8 and 24 characters)m look at the value to see it is correctp move to the Submit buttonb click
m p b m kkkkkkkk m p b = 8.24 – 12.72 sec
KLM calculator: http://courses.csail.mit.edu/6.831/2009/handouts/ac18-predictive-evaluation/klm.shtml
![Page 28: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/28.jpg)
orientation
![Page 29: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/29.jpg)
orientation = ability to navigate
shows how much users deviate from an optimal path
0 to 100 (could be percent)
orientation
![Page 30: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/30.jpg)
measure coefficient of orientation define optimal path of steps for each task = 100% (O)
count the actual steps performed by users (A)
CoR = (1 – A/O)%
![Page 31: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/31.jpg)
CoR = (1 - 3/6) = 50%optimal path actual path
1
2
3
1
3
2
1’
2’ 2’’
![Page 32: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/32.jpg)
learnability
![Page 33: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/33.jpg)
learnability
shows how fast users get to use the product
fast to slow (to never)
learnability
![Page 34: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/34.jpg)
errors
![Page 35: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/35.jpg)
errors = wrong UI
show why soft performance suffers
two types of errors to count slips = I know the correct way mistakes = I am guessing the correct way
errors
![Page 36: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/36.jpg)
slip
I want to save the changesbut click Don’t Save instead of Save
my friend who has never before driven an automatic car is pressing the break pedal with his left foot
![Page 37: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/37.jpg)
mistake
I wanted to change the background of a survey to be the same color as the orange color used on the ISTA website
R:G:B 220:110:15
![Page 38: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/38.jpg)
the color I need is not in the palette
so I prepare an image with the desired color
and click the only option that looks different
![Page 39: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/39.jpg)
…I dragand dropthe image file…
![Page 40: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/40.jpg)
The photo you uploaded is too small! It must be at least 800 pixels wide and 200 pixels tall.
![Page 41: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/41.jpg)
I edited the fileto 800 by 200 pixelsuploaded it and got this header
not the same as color I uploaded
![Page 42: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/42.jpg)
effectiveness | efficiency | satisfaction
![Page 43: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/43.jpg)
task 1
task 2
task 3
overall satisfaction
![Page 44: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/44.jpg)
satisfaction = sense of accomplishment
shows how much users like the product from “not at all” to “completely”
caveat – users might be evaluating: look and feel self
![Page 45: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/45.jpg)
task
satisfaction
![Page 46: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/46.jpg)
task-level satisfaction
one question after each task
the single ease question (SEQ)
SEQ: http://www.measuringu.com/blog/single-question.php
![Page 47: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/47.jpg)
topic-level satisfaction
![Page 48: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/48.jpg)
overall
satisfaction
![Page 49: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/49.jpg)
overall satisfaction defines the overall experience variety of methods and metrics
SUS = System Usability Scale NPS = Net Promoter Score SUPR-Q = Standradized Universal Percentile Rank Questionaire
…
![Page 50: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/50.jpg)
SUS = System Usability Scale1. I think that I would like to use the system frequently2. I find the system unnecessarily complex3. I think the system is easy to use4. I think that I would need the support of a technical person to be able to use
the system5. I find the various functions in the system are well integrated6. I think there is too much inconsistency in the system7. I would imagine that most people would learn to use the system very quickly8. I find the system very cumbersome to use9. I feel very confident using the system10. I needed to learn a lot of things before I could get going with the systemSUS: https://en.wikipedia.org/wiki/System_usability_scale
![Page 51: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/51.jpg)
scoring SUS
1. do this in Excel2. scale all values from 0 to 4 (four is the most positive)
odd items: user response - 1even items: 5 - user response
3. convert to range 0 to 100 sum the scaled responses for each user and multiply the total by 2.5
![Page 52: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/52.jpg)
SUS shows three things [1]
satisfaction – the full set of items
learnability – items 4 and 10
usability – the other 8 items
[1] http://www.measuringu.com/sus.php
![Page 53: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/53.jpg)
SUS explains likelihood to recommend [1]
NPS score ≈ 0.1* SUS score
SUS score of 80 ≈ NPS of 8
[1] http://www.measuringu.com/blog/nps-sus.php
![Page 54: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/54.jpg)
average SUS
68
0 100
> 82promoters
[1] http://www.measuringu.com/products/SUSpack
overallbusiness apps
websites 75consumer apps
65cellphones
58average
41lowest
79highest
my studies in the last 2-3 years
![Page 55: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/55.jpg)
ISTA website SUS
0 100
66satisfaction
60.9usability 86.2
learnability
as of 11:00, 19 Nov 2015, 68 participants
![Page 56: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/56.jpg)
about the ISTA website, we can be: 95.0% confidentthe population SUS Scoreis between 61.78 and 70.14
97.5% confidentthe population mean SUS Scoreis above 61.78
mean = 66
margin of error = 4.18
![Page 57: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/57.jpg)
promoters: 5.88%
passives: 32.35%
detractors: 61.76%NPS = -55.88%
![Page 58: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/58.jpg)
trinity?
analytics
big data
internet of things
![Page 59: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/59.jpg)
quality out of quantity data is out there (or in there) data is neither good, nor bad what matters is how we use data
can we use data to promote wellbeing and happiness?
![Page 60: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/60.jpg)
things you might want to check quantified self PERMA happiness index – GNH = gross national happiness Sonja Lyubomirsky’s 12 steps to happiness
![Page 61: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/61.jpg)
soft performance metrics to take home
effectivenesssuccess rate
satisfactiontask-level: SEQoverall: SUS
efficiencytime on taskeffort: Keystroke-Level Modelorientation: CoRerror rate
![Page 62: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/62.jpg)
that’s
folks
@dsimovabout.me/
dsimov
all
![Page 63: Soft performance - measuring](https://reader035.vdocuments.us/reader035/viewer/2022062400/589fca8c1a28ab91398b6813/html5/thumbnails/63.jpg)
all illustrations in this presentation are by Iliya Beshkov from the book „Тошко Африкански“ (Toshko Afrikanski)