usability with project lecture 19 – 19/11/08

© Simeon Keates 2008

Usability with ProjectLecture 19 – 19/11/08Dr. Simeon Keates


Exercise

Start/continue your user trials

If you want feedback on your group presentation – then prepare slides for Friday• NOTE – this is optional, but recommended


Cost-justifying usability


The need to cost-justify

Not all companies (and managers) appreciate the benefits of usability They may cite other factors as more important Examples:

• Tight deadlines• Functionality already developed• Limited money/resources


Cost-justifying usability

However, these are often “false economies” Examples:

• Deadlines:• Releasing the “wrong” product on time is as bad (or worse) as releasing the

“right” product late• Existing functionality:• Existing functionality should have nothing to fear from usability, if it is the

“right” functionality• Limited resources:• Putting the “right” resource in at the “right” time can make the overall project

more efficient


Rosson and Carroll “Usability Engineering”

“Cost-benefit analysis of usability activities contributes to more systematic usability engineering …”

“… BUT benefits are difficult to quantify, so estimates will often be overly conservative.”

Issues: Benefits (e.g. customer satisfaction) are harder to quantify and predict

accurately Costs are very concrete and easy to identify Thus, can be difficult to justify usability…


Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 1)

Development of requirements scenarios Validation/refinements of scenarios with users and customers Development of basic-level task scenarios Refinement of design scenarios with development team and customers Development of information model Review with team members Development of paper prototypes Walk-throughs with users of paper prototypes …


Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 2)

… Analysis of transcripts/report preparation Development of interaction model Review with team members Development of running prototypes Formative evaluation Analysis of transcripts/report preparation Detailed design and prototype-driven iteration of previous three steps

All of these steps have “usability” costs


Typical usability benefits

Fewer downstream design changes Increased sales (and consequently reduced time to profitability) Reduced need for user training Enhanced customer productivity Reduced resources spent on customer support Increased loyalty in customer base (repeat and referral sales)

Many of these benefits identified in

airport case study from 2 lectures ago…


A more scientific approach

Mantei and Teorey(*) examined the cost/benefit analysis in 1988

We will look at their calculations

But first we need to examine their methodology

NOTES:• Costs are based on 1988 prices and techniques• Not all costs are required for every project – choose which ones carefully• Costs are indicative, not definitive• Assumes 32,000 lines of code to be used by 250 employees

(*) Mantei and Teorey (1988) Cost/benefit analysis for incorporating human factors in the software lifecycle.

Communications of the ACM 31(4): 428-39


Typical development stages in a prototyping lifecycle

Feasibility study Requirements definition Global design Prototype construction User evaluation System implementation Testing Update and maintenance


Typical development stages in a Human Factors software lifecycle

Market analysis Feasibility study Requirements definition Product acceptance analysis Task analysis Global design Prototype construction User testing and evaluation System implementation Product testing User testing Update and maintenance Product survey


A breakdown of tangible costs

Cost type Cost (1988 levels)

Salaried employee (SE) Fixed ($40/hr)

Hourly employee (HE) Fixed ($15/hr)

External contractor (EC) Fixed ($60/hr)

Consultant (CN) Variable

Equipment and supplies (ES) Variable

Software purchase (SP) Variable


Further assumptions

40 hours per week 4.3 weeks per month =>172 hours per working month (WM)

Examples: $6880/WM for a salaried employee (at $40/hr)

• i.e. 172 * 40

$10,320/WM for an external contractor • i.e. 172 * 60


Costs added by Human Factors stages

1 – Cost of running focus groups 2 – Cost of building product mock-ups 3 – Expense of the initial design of a prototype 4 – Expense of making a prototyping design change 5 – Expense of purchasing the prototyping software 6 – Cost of running user studies 7 – Cost of creating (or renting) a user study environment (laboratory) 8 – Cost of conducting the user survey


Cost 1 – Focus groups cost breakdown

Time cost of the individuals involved + small equipment cost Individuals involved:

• Participants • Moderator/facilitator• Video-taping/recording personnel• Any other observers/data analysts

~3 people

~10 people

1 person


Focus groups cost breakdown

Focus groups typically take 3 hours to run + 1 day to set-up and 1 day to dismantle

Minimum of 2 days to analyse the data

Moderator: Time for focus group + analysis Support staff: Time for set-up + focus group + dismantling Participants: Time for focus group [+ any preparation time]

A complete focus group analysis (3 consecutive groups) takes ~2 weeks

Used for Market Analysis and Product Acceptance Analysis


Focus groups cost breakdown

Estimated costs for conducting focus groups

Type of expense Category Amount

A Cost of operating 3 internal focus groups

30 participants, 10 per focus group (3 hrs each)

SE $3,600

Group moderator (25 hrs) CN $1,500

3 support staff (25 hrs each) HE $1,125

Videotape ES $60

Total $6,285

B Cost of contracting3 external focus groups

Fee charged by agency for complete study ( 3 focus groups and analysis for 2 weeks)

CN $10,000


Cost 2 – Estimating product mock-up costs

Building a product mock-up involves constructing a false UI scenario in software and video-taping the scenario

Script has to be written Someone has to execute the scenario Videotape needs to be professional

Videotape mock-up is used in Product Acceptance Analysis• To focus groups to initiate discussions• To target users who are then asked to complete questionnaires about the

use of the product and their response to it

Can also be used in marketing


Estimating product mock-up costs

Costs incurred in building a product mock-up of the proposed software system


Preparation of mock-up scenario (40 hrs) SE $1,600

Videotaping sessions (20 hrs) SE $800

Splicing/integration of scenarios (20 hrs) SE $800

Equipment rental for splicing, etc. ES $500

Videotape ES $60

Total $3,760


Estimating product mock-up costs

The techniques just described are somewhat out of date More usually accomplished via early development cycle prototypes

• e.g. alpha and beta versions• Flash movies, etc.

Q - What to do if no software written? A – Wizard of Oz!


Wizard of Oz

Do Action A

{Receive response to Action A}

Do Action B, etc.


Wizard of Oz (The Wizard unmasked!)

Do Action A

{Receive response to Action A}

Do Action B, etc.

Create response to Action A

Create response to Action B, etc.


Wizard of Oz explained

User interacts with a UI mock-up only There is no significant software back-end

Remote user (the Wizard) “pretends to be the system” Wizard creates the response based on expected system performance

Still needs a script and plenty of preparation• Imagine if the Wizard does not know a response!


Cost 3 – Estimating user survey costs

Used in the Product Acceptance stage to assess users’ responses to the product mock-up

Also used in Product Survey (after launch) to:• Determine the difficulties users have with the working system• Examine the tasks the system is being used for• Gather suggestions for changes to system

For a user population of 250 employees Typically half (125) would receive a user survey A typical survey is 4 pages in length… … and takes 30 minutes to complete


Estimating user survey costs

Cost breakdown for running a user survey for the software product being tested


Development of questionnaire (40 hrs) SE $1,600

Pilot testing of questionnaire (40 hrs) SE $1,600

Distributing and collecting survey (20 hrs) HE $300

Coding and entering data (20 hrs) HE $300

Analysing the results of the survey (40 hrs) SE $1,600

Cost of time “lost” in filling out survey (40 hrs) SE $1,600

Computer time ES $100

Supplies and duplicating costs ES $100

Total $7,200


Cost 4 – Estimating initial prototype building costs

Cost breakdown for building a prototype does NOT include the design time • Only the building time

Typically 4 weeks

Most prototyping systems involve 2 stages:• Stage 1 – specify the connections between the screen displays• Stage 2 – design each individual screen layout

Alternatively (and more modern)• Stage 1 – specify the states between user interactions• Stage 2 – design the individual states and the alterations that take place

because of user actions


Estimating initial prototype building costs

As the UI grows more complex, time required to build the prototype also increases

Cost of building a prototype is:

C = S × ( a + b D)

Where C = Cost S = Number of states D = Average number of new details per state a = Constant reflecting the cost of building a single state b = Constant reflecting the cost of adding a single detail


Estimating initial prototype building costs

Costs incurred in the initial design of a prototype of the proposed software system (the prior development of a global design is assumed)


Specifications of the screen transitions (80 hrs) SE $3,200

Design of the individual screen layouts (80 hrs) SE $3,200

Total $6,400


Cost 5 – Estimating costs for design changes to original prototype

Once the prototype is built, user studies will uncover problems with it• Difficulties learning and using the system

Design change suggestions will be made … … incorporated into the design … … and evaluated again, etc. … … until the number and severity of problems are “acceptable”

The initial user studies will usually find the most and most major issues Later changes should be minimal

• Especially if the prototype is close to the final product design …• … which it should be if task analyses, user surveys, etc. have been used• Also, design changes should simply be updates of parts of the prototype …• … and not a complete re-design


Estimating costs for design changes to original prototype

The initial user studies will usually find the most and most major issues Later changes should be minimal

Especially if the prototype is close to the final product design … … which it should be if task analyses, user surveys, etc. have been

used

Also, design changes should simply be updates of parts of the prototype …

… and not a complete re-design



Since changes should be restricted to parts of the UI, should only take ~1 day per change

The number of changes expected and amount of time per change depends upon complexity of the interface

Similar equation to cost of building a prototype



Costs incurred in incorporating a design change to the original prototype of the proposed software system. These costs assume that the design change does not require a complete revision of the screen transitions.


Modification of the screen transitions (4 hrs) SE $160

Re-design of the individual screen layouts (4 hrs) SE $160

Total $320


Cost 6 – Estimating the cost of purchasing the prototyping software purchase

Actual purchase cost Also time spent deciding which one … … and then learning it

1988 prices - $10,000 for the software ($2,500 to $15,000+) 2008 prices - $600 per user (AXURE)


Estimating the cost of purchasing the prototyping software purchase

Costs incurred in incorporating a design change to the original prototype of the proposed software system. These costs assume that the design change does not require a complete revision of the screen transitions.


Time spent reviewing potential packages (1 month) SE $6,080

Purchase cost of package SP $10,000

Total $16,080

+ Time spent learning package


Cost 7 – Estimating costs of performing user studies

Preparation time can be substantial• Example: preparing a manual for a completely new UI• Manual need not be complete, but it needs to be sufficient and pilot-tested• Example: preparing the study protocol• Protocol needs to be complete and pilot tested

Costs of individual user studies are often independent of complexity of UI …

… however, the number of user studies increases with complexity

3 types of user study in the Human Factors software lifecycle…


Estimating costs of performing user studies

In Task Analysis: Users are asked to perform the types of tasks the system should

support Aim is to build a model of how users approach the task

In User Testing and Evaluation and in final User Testing: More conventional user studies Conducted on prototypes or final system Final system evaluation is always needed

• May be significant changes from the prototype behaviour


Estimating costs of performing user studies

Costs incurred in conducting a single user study on 5 participants


Development of user directions [i.e. protocol] (40 hrs) SE $1,600

Pilot testing of directions (20 hrs) SE $800

Re-designing user directions (20 hrs) SE $800

Running experiment (40 hrs) SE $1,600

Analysing results of lab study (40 hrs) SE $1,600

Videotape ES $120

Cost of users in experiment (20 hrs) SE $800

Total $7,320


Cost 8 – Estimating costs of construction of a user laboratory

A user lab can be a borrowed / rented or permanent office Permanent office is recommended where significant usability activity is

expected User lab should be a mock-up of the environment of use for the

product The lab should allow space for an adjoining observation/recording

room Construction of a lab takes ~5 weeks


Estimating costs of construction of a user laboratory

Cost of establishing a permanent human factors lab. These are mid-level costs. Much more sophisticated labs can be built.


Time spent laying out lab design and selecting lab equipment (160 hrs)

SE $6,400

Cost of carpenter and electrician (20 hrs) EC $1,200

Cost of cameras, VCRs, one-way mirror ES $10,800

Total $17,600


Estimating costs – a summary

Lifecycle stage Cost item Total cost

Market analysis Focus group (set of 3) $6.285

Product acceptance analysis Focus group (set of 3) $6.285

Product mock-up $3,760

User survey $7,200

Task analysis User study $7,320

Lab construction $17,600

Prototype construction Initial design $6,400

Design change (20 @ $320) $6,400

UIMS system $16,080

User testing and evaluation User study (4 @ $7,320) $29,280

User survey $7,200

User testing User study $7,320

Product survey User survey $7,200

Total $128,330


Estimating times – a summaryLifecycle stage Cost item Total WM Dev time

Market analysis Focus group (set of 3) 1.11 6 wks

Product acceptance analysis

Focus group (set of 3) 1.11 6 wks

Product mock-up 0.47 2 wks

User survey 1.16 4 wks

Task analysis User study 1.05 4.3 wks

Lab construction 1.05 5 wks

Prototype construction Initial design 0.93 4 wks

Design change (20 @ .05 WM) 1.00 4 wks

UIMS system 1.00 4.3 wks

User testing and evaluation

User study (4 @ 1.05 WM) 4.20 17.2 wks

User survey 1.16 4 wks

User testing User study 1.05 4.3 wks

Product survey User survey 1.16 4 wks

Total 16.45 WM 69.1 wks


Common tangible benefits of Human Factors design

Direct benefits can be calculated by making several valid assumptions about the improvements to the UI, specifically:

(1) A reduction in user learning times

(2) A reduction in user errors

(3) A reduction in the cost of maintaining the system

Remember: for 32,000 line program for 250 employees…


Benefit 1 – Potential training cost savings

It is estimated that the learning time for new system will be 25% lower Learning time is typically 2 weeks of classes

• 10 working days / 80 working hours

Staff turnover is 50 hourly employees and 50 salaried employees per year

(Savings / year) = (Turnover) × (Training time save) × (Wage)

= 50 × 20 × $15

= $15,000 per year for HEs

= 50 × 20 × $40

= $40,000 per year for SEs


Benefit 2 – Potential error reduction cost savings

User “traps” exist in all Uis• i.e. a standard sequence of user responses that always result in an error

Errors may be trivial and easily corrected … … but are often annoying and time-consuming These traps are almost always the result of the UI violating the learned

behaviour of the user• Example: driving an automatic car


Potential error reduction cost savings

How many errors per year?

Assume: User has 0.025 chance of falling into a “trap” Company has 250 employees using software 3 hrs/day and 20

scenarios/hr

Errors/year = (# of employees) × p(Error) × (scenarios/hr) × (hrs/year)

= 250 employees × 0.025 × 20 × (21.5 dy/mo × 3 hrs/dy × 12 mo/yr)

= 96,750 errors per year


Potential error reduction cost savings

How much cost per year?

Assume: 96,750 errors per year 2 minutes recovery time per error

Cost/year = (errors/yr) × (hrs/error corr) × (wage/hr)

= 96,750 × (2 min/err) × (1hr ÷ 60mins) × ($15 / hr)

= $48,375 per year for hourly employees (HE)

= 96,750 × (2 ÷ 60 ) × ($40 / hr)

= $129,000 per year for salaried employees (SE)


Benefit 3 – Potential design change cost savings

It is hard to estimate maintenance savings However, it is claimed that early changes cost ¼ of later design

changes

Assume: 1 day (i.e. 8 hours) per change 25 changes

Early cost = (hr / change) × (# of changes) × (wage / hr)

= 8 × 25 × $40

= $8,000


Potential design change cost savings

Late cost = (early cost) × 4

= $8000 × 4

= $32,000

Design change savings = (late cost) – (early cost)

= $32,000 – $8,000

= $24,000


Benefit 4 – System adoption

If the development of the system is planned to meet the needs and wants of the user …

… then the probability of acceptance and use is higher

The cost benefit here is the entire cost of the systems development … … because it is not “lost”


Tangible benefits of Human Factors design – Summary

Sample estimates of 1st year savings from the introduction of HF elements in the software design process

Type of savings incurred Amount (HE) Amount (SE)

Training costs (HE or SE) $15,000 $40,000

Error reduction costs (HE or SE) $48,375 $129,000

Avoidance of late design changes (SE) $24,000 $24,000

Total $87,375 $193,000

Total savings ~$280,375

Total costs ~$128,330


Intangible costs of Human Factors software lifecycle

1 - The selection of non-critical design decisions for user studies 2 - The establishment of too high a level of usability 3 - Falling into the trap of overdesign 4 - Communication problems between Human Factors specialists and

software designers


Selection of non-critical design decisions for user studies

Not all design decisions affect the overall “quality” and “acceptability” of the final UI

How do you determine which ones are worth investigating?• Examples: Arial or Helvetica? 11 pt or 12 pt? Icon names?

Most usability experts rely on experience and intuition to decide… … but they can be wrong!

Costs: Wasted user study time Discovery of important changes may be put off until later

Solution: Make your user studies flexible enough to allow discovery of new problems


Establishment of too high a level of usability

Some UI performance targets simply cannot be met Example: “This system must be learnable within 1 week” This may appear to be a reasonable target … … but what if the task itself takes >1 week to learn?

Costs: Design and development time chasing unrealistic performance targets

Solution: Set realistic targets This can be informed by your task analysis studies Also, aim for incremental improvements in performance over design

cycles


Falling into the trap of overdesign

Prototyping software (UIMS) can make it very easy to add more features (“bells and whistles”)• Examples: borders, colours, icons, images, etc.

How to know when to stop??? How to know what are valuable additions?

Costs: Designer’s time + implementation time

Solution: Strong management control Occam’s razor


Communication problems between Human Factors specialists and software designers

There is a knowledge gap between HF specialists and the coders/developers…

Unless there is a shared common language and understanding, it is difficult to communicate effectively

Costs: Time lost in establishing a common language/understanding Time lost developing the wrong thing (or the thing wrongly) Time lost designing solutions that simply cannot be implemented

Solutions: HF specialists learns to develop own UI Better corporate culture of communication


Intangible benefits from Human Factors software lifecycle

1 - Adoption of features that save time 2 - Avoiding system sabotage problems 3 - Enhancing the ability to solve conceptual problems using the

software system


Adoption of features that save time

Feature adoption by users is reduced when the system is so complex that users give up trying to learn the advanced features• i.e. they satisfice

Users will typically adopt the least complex system that offers them the required functionality• Even if it is less efficient

The Product Acceptance Analysis should identify unnecessary features

The User Tests should reduce feature complexity


Avoiding system sabotage problems

Being asked to use a system that is inappropriate, difficult or inadequate can lead to employee frustration

This frustration can lead to employees “taking it out on the system”• Example: entering incorrect or incomplete data• Example: reporting false system failures

This is referred to as “system sabotage”

Focus groups in Market Analysis and Product Acceptance stages are designed to address this issue• They test the receptivity of the target users to the new system

Also, the Task Analysis ensures that the final product matches the needs of the users as closely as possible


Enhancing the ability to solve conceptual problems using the software system

Increasing the user’s cognitive load to use the system … … decreases the user’s cognitive capacity for “solving the problem”

So although the problem may be “solved” … … there may have been a “better” solution

User Testing is designed to remove complexities from the system … … and thus support users’ creative problem solving


Recommendations for Human Factors inclusionThe HF lifecycle stages and the type of cost reduction that they are most likely to effect

Cost reduction item Related lifecycle stage

Increased system adoption Market analysis

Product acceptance analysis

User testing and evaluation

Reduced training costs Task analysis

User testing

Reduced user errors Task analysis

User testing

Transfer of design changes to an earlier stage in the lifecycle

Prototype construction

User testing (on prototype)

Product survey (next re-design)


Cost of running User Tests vs. size of user population

Size of user population

$

Break-even point

Benefits of testing

Cost of testing

User testing not cost-effective

User testing cost-effective


Summary of cost-benefit analysis

We have examined the factors that can increase costs … … and also the tangible benefits that can be seen

For any large user group, the benefits will usually outstrip the costs … …often by a significant margin

For smaller user groups, the case is less clear-cut … … so either decrease costs …

• e.g. through “discount” usability methods

… or examine the intangible benefits more closely


Designing for mobile devices


5 products killed by the mobile phone (according to Wired.com)

1 – The PDA 2 – The camera 3 – The Ultra Mobile PC 4 – The telephone 5 – The MP3 player


7 products killed by the mobile phone (according to Wired.com readers)

1 – The pager 2 – The wristwatch 3 – The pocket calculator 4 – The alarm clock 5 – SatNav 6 – Books 7 – Handheld games consoles


Designing for Mobile devices

The “missing” lecture! Excellent sets of slides at from MobileHCI 2008 http://albrecht-schmidt.blogspot.com/2008/09/mobilehci-2008-tutorial.htm

l

Suggested reading: Scott Mackenzie’s presentation “Text input for mobile devices”…


Other slide sets from MobileHCI 2008

Mobile GUIs and Mobile Visualization• Patrick Baudisch

Understanding Mobile User Experience• Mirjana Spasojevic

Context‐Aware Communication and Interaction• Albrecht Schmidt

Haptics, audio output and sensor input in mobile HCI• Stephen Brewster

Camera‐based interaction and interaction with public displays• Michael Rohs


Exercise


Exercise

Conclude your user trials

If you want feedback on your group presentation – then prepare slides for Friday• NOTE – this is optional, but recommended

usability with project lecture 19 – 19/11/08

Documents