Assessment, Evaluation and Risk Appraisal

Using Human Factors Methods to Supportthe Procurement Process

National Hospital Procurement Conference

29 October 2015

Robyn Clay-­Williams, PhD Brette Blakely, PhD

Points for discussion

2

• What is ‘human factors’, and how can it help in procurement?

• Specifying ‘practical’ requirements for the product(s)

• Evaluating the product(s) to ensure it meets user needs

• Comparing safety and effectiveness of similar products

• How can human factors methods reduce the risk associated with procurement?

What is human factors (HF)?

3

• Applies evidence-­based methods and knowledge about people to design and improve the interaction between people, systems, and organisations

• Achieved by ensuring there is a good fit between people and their environment

• Don’t change behaviour, re-­design the system

Good vs. bad designFit for purpose?

Page 4

Good vs. bad designFit for purpose?

Page 5

Intuitive to operate?

Human factors

6

• For any task/context:• Physical needs• Cognitive needs

Human Centred Design

ergonomics

Human factors

7

• For any task/context:• Physical needs• Cognitive needs

Human Centred Design

anthropometrics

ergonomics

Human factors

8

• For any task/context:• Physical needs• Cognitive needs

Human Centred Design

Human factors

9

• For any task/context:• Physical needs• Cognitive needs

Human Centred Design

The “Human” Factor

Human factors

10

• Design examples:• Forcing functions• Colour coding

A syringe pump example

11

Specifying ‘practical’ product requirements

12

• What is the task?

• Who will perform it?

• In what context?

• How do we define success?

• Desired criteria

• Adequate criteria

Normal TasksNon –normal tasks

User

UserUser

User

Environment

Specifying ‘practical’ product requirements

13

• Three criteria for goals:

• clear, quantify performance, testable

• Bad specified requirements:• ‘[product] must be intuitive to use’

• ‘[product] must minimise frequency of user error’

• Good specified requirements (fetal heart monitor example):• ‘On average, users must be able to resolve the fetal heart rate with +/-­1 beat/minute precision’

• ‘At least 80% users must agree that the monitor cart is easy to manoeuvre’

Evaluating the product

14

To understand how good the fit is between people and their working environment, we apply a range of human factors methods, for example:• Interviews• Focus groups• Observation (and video recording) in situ and in the lab• Physical measurement of the workplace• Usability testing

These methods can tell us a lot about how people thinkthey work (and how they actually work)

Human factors methods

15

To understand how good the fit is between people and their working environment, we apply a range of human factors methods, for example:• Interviews• Focus groups• Observation (and video recording) in situ and in the lab• Physical measurement of the workplace• Usability testing

These methods can tell us a lot about how people thinkthey work (and how they actually work)

Usability – what is it?

16

Is the product:• Easy to use (user experience)?• Easy to learn?• Enjoyable to use (user satisfaction)?• Useful?

Usability in design

17

Early focus on:• Users• Task

Evaluation:• Formative testing• Summative testing

Iterative design:• Feedback• Redesign

Usability testing

18

Product evaluation by testing on users• Effectiveness – does it do the job safely? • Efficiency – is it cost effective?• User Satisfaction – does the user like using it?

Test methods:• User in the loop• Desktop

Analysis:• Formative (insights) ~5-­10• Summative (statistics, certification) ~15-­25

Usability testing

19

Product evaluation by testing on users• Effectiveness – does it do the job safely? • Efficiency – is it cost effective?• User Satisfaction – does the user like using it?

Test methods:• User in the loop• Desktop

Analysis:• Formative (insights) ~5-­10• Summative (statistics, certification) ~15-­25

User in the loop testing

20

• Choose tasks (e.g. frequent, critical, complex)• Choose participants (sample of typical users)• Choose methods (e.g. task-­based, think-­aloud,

self-­exploration, eye tracking, questionnaires)

User in the loop testing

21

Usability testing

22

Product evaluation by testing on users• Effectiveness – does it do the job safely? • Efficiency – is it cost effective?• User Satisfaction – does the user like using it?

Test methods:• User in the loop• Desktop

Analysis:• Formative (insights) ~5-­10• Summative (statistics, certification) ~15-­25

Desktop testing

23

• You don’t need to test everything

• Prioritise tasks based on risk analysis, task relevance

• Spot check manufacturer’s usability results, but apply Australian health service conditions/standards

• AIHI@MQ is developing a method

• For a simple assessment, consider heuristic evaluation

Heuristic evaluation

24

• Application of human factors principles to assess a product

• 1-­2 people, plus product

• Define tasks

• Structured walk through tasks

• Evaluate using 10 basic heuristics

Ten Usability Heuristics

25

Benchmark testing

26

• Comparing safety and effectiveness of similar products

• Limit comparisons to 3 or 4 products

• Pick a representative sample of tasks to perform on each product (tasks must be relevant to pre-­defined needs)

• Desktop testing

• User in the loop testing:

• collect performance data such as task times, subjective ratings (consider short interviews re. strengths/weaknesses)

• Same users should interact with all products

• Control for test users’ experience (especially if a user has experience with one of the products)

Minimising procurement risks

27

• How can human factors methods contribute to reducing the risk associated with procurement?

• Safety: identify use-­related hazards;; confirm risk mitigation strategies for known hazards are adequate

• Function: will the product do the job (WAI vs WAD)?

• Training: is the product easy to learn?

• Adoption: is the product likely to garner wide clinician acceptance?

Relevant human factors standards

28

Therapeutic Goods Administration (TGA). 2011. Australian regulatory guidelines for medical devices (ARGMD), Version 1.1.https://www.tga.gov.au/publication/australian-­regulatory-­guidelines-­medical-­devices-­argmd

US Food and Drug Administration (FDA). 2011. Applying human factors and usability engineering to optimize medical device design (draft guidance). http://www.fda.gov/RegulatoryInformation/Guidances/ucm259748.htm

International Organization for Standardization IEC 62366:2007. 2007. Medical devices – application of usability engineering to medical devices.

International Organization for Standardization ISO 14971:2007. 2007. Medical devices – application of risk management to medical devices.

International Organization for Standardization IEC 60601-­1-­8. Medical electrical equipment – General requirements for basic safety and essential performance.

International Organization for Standardization ISO 9241-­11. Ergonomic requirements for office work and visual display terminals (VDTs) – Part 11: guidance on usability.

Association for the Advancement of Medical Instrumentation ANSI/AAMI HE75:2009. 2010. Human factors engineering – design of medical devices.

NISTIR 7741. 2010. NIST guide to the processes approach for improving the usability of electronic health records. http://www.nist.gov/itl/hit/upload/Guide_Final_Publication_Version.pdf

NISTIR 7804. 2011. Technical evaluation, testing and validation of the usability of electronic health records. http://www.nist.gov/customcf/get_pdf.cfm?pub_id=909701

Thank youEND