7 components to medical device usability testing success

37
Confidential Copyright © 2014 August 21, 2014 1 7 Components of Successful Usability Testing in Medical Device Development JARED SCHNEIDER KALEIDOSCOPE, onsite at ETHICON INC., SOMERVILLE, NJ

Upload: margee-moore

Post on 18-Dec-2014

39 views

Category:

Design


1 download

DESCRIPTION

Despite the publication of various relevant guidance on medical device usability testing and standards for human factors testing confusion regarding best practices still exists. This presentation provides clear language and seven components for planning successful usability testing for medical device development.

TRANSCRIPT

Page 1: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 1

7 Components of Successful Usability Testing in Medical Device Development

JARED SCHNEIDER

KALEIDOSCOPE, onsite at ETHICON INC., SOMERVILLE, NJ

Page 2: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 2

Contents1. Formative and validation tests

2. Planning for tests

3. User groups and participants

4. Moderating usability tests

5. Use scenarios, prototypes, and training

6. The test environment

7. Test results

Page 3: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 3

1. Formative and validation tests

Page 4: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 4

Formative testing

Formative usability testing is an exploration. Its purpose is to:

• Identify user needs and potential hazards

• Design out use errors while the design can still be modified

Page 5: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 5

Usability validation

Usability validation is the proof. Its purpose is to:

• Demonstrate intended users can safely and effectively perform critical and essential tasks

• Test the final design with intended users in real or simulated environment

Page 6: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 6

Market research

Market research is useful for measuring opinions, attitudes, and speculation, but should not be confused with usability testing

Usability testing is primarily about risk mitigation. User preference or acceptance is secondary. Usability testing

be limited to subjective rating scale surveys (e.g. Likert).

Page 7: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 7

Dual purpose

Often research efforts involve both usability testing and market research to serve two purposes:

Risk mitigationFind interface problems leading to use error

Product enhancementImprove ease of use, look and feel, desirability

Page 8: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 8

Is testing required?

Formative testing is not explicitly required by law. However, failure to perform

to the FDA, and bad business practice. It is appropriate to scale the exercise based on risk and complexity.

Usability validation is required per 21 CFR Part 820

Page 9: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 9

2. Planning for tests

Page 10: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 10

Scheduling testing

Build usability tests into the development schedule early, especially since recruitment of participants takes time.

Testing can actually shortenthe overall schedule by:• Finding issues early when

they are easier to fix• Helping teams make

decisions and move forward

Page 11: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 11

Pilot testing

Especially for validation testing, it is wise to perform pilot testing to fine tune the test protocol before collecting data that counts. Pilot testing provides an opportunity to:• Adjust scenarios and

questions• Check equipment• Rehearse roles• Review test schedule

Photo Courtesy of: navypilotoverseas.worldpress.com

Page 12: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 12

and dirty

If a project schedule or budget limits formative testing, a

test can answer some basic design questions.

Whether the effort can be called a formative test depends on methods and recruitment. At minimum, describe the test and results in

Page 13: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 13

Unalterable designs

Even if a design is already

still worthwhile. Critical issues that arise may cause reconsideration. Even if design changes are not pursued,adjustments could be made to learning aids, training, or marketing strategy.

Photo Courtesy of: pouted.com

Page 14: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 14

3. User groups and participants

Page 15: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 15

User groups

User groups are distinct populations that interact with the device differently.

They perform different tasks or understand treatment or device operation differently.

Some high level categories are:• Professional caregivers•

• Pediatric users

Page 16: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 16

User groups

When recruiting participants:

• Be inclusive of variability in user abilities relevant to device use

• Incorporate worst case users if they represent a consistent portion of the user population

• If a disease being treated affects device use, only test people with the disease

Page 17: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 17

How many participants?

For each distinct user group, use a minimum of 5-8 participants for formative testing or 15 for validation.

The goal is to find problems! 5-8 participants reveal 80-90% of problems. Additional participants offer diminishing returns.

Recruit extra participants to -

Pe

rce

ntag

e o

f P

rob

lem

s U

nco

vere

d

Number of Test Participants

Page 18: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 18

Internal participants

often suggested to save time and money, but rarely makes sense. They are not members of the user group, and know too much about the project to be unbiased.

A formative test may present an exception if it solely examines physical ability and the internal participants fit the physical user profile.

Page 19: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 19

Group tests

Group usability tests are often proposed for time/cost savings, but should be avoided unless a team approach reflects actual use.

It is important to measure individual performance, and shield participants from the influence of others.

Individual tests may be run concurrently to save time if it is feasible.

Page 20: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 20

4. Moderating usability tests

Page 21: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 21

Internal moderators

There are advantages and disadvantages to having team members conduct tests of their own designs.

ProIn-depth product knowledgeCost savings

ConDifficult to be objective and avoid unintentional bias

Page 22: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 22

Moderatingtests

If moderating a usability test, try to be clear and unbiased in your interaction with participants.

Common mistakes include:

• Talking too much• Leading or biasing• Assisting• Rushing• Losing track of time

Photo Courtesy of: mckatalyn.deviantart.com

Page 23: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 23

Providing assistance

If participants have difficulty completing tasks, promote perseverance and let them struggle.

Assistance should only be provided at a true impasse. Time limits may help to make that decision. Assistance may ramp up as needed from a hint

Page 24: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 24

5. Use scenarios, prototypes, and training

Page 25: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 25

Task selection

Usability testing must involve representative use, with tasks selected and prioritized based on risk:• Critical tasks with a

potential for harm• Essential tasks required for

medical benefit

State the selection rationale using preliminary analyses and formative testing.

Page 26: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 26

Use scenarios

Usability testing involves observing participants in use scenarios, groups of tasks with

task 1, now do task 2

Use scenarios should include critical and essential tasks. They may be crafted to repeat tasks for closer evaluation, and force difficult or atypical situations as needed.

Page 27: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 27

Prototype fidelity

Formative testing usually involves iterative prototypes to examine use errors mitigated or induced by design features.

Usability validation must be performed with equivalent units.

intuitool.unemed.com

Page 28: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 28

Training

Thought should be given to training users during usability testing. Formative tests can explore how much (if any) training to provide, while validation must be consistent with the training actual users would receive.

Provide any learning tools that would be available (e.g. instructions for use), and build in a decay period after training.

Page 29: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 29

Training

If actual users receive training, withholding it to show intuitiveness can be counterproductive:• Testing may fail to expose

issues with training• The project team may see

the added challenge as sabotage

Also consider that optimizing a device for intuitiveness can hurt long term ease of use.

Page 30: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 30

6. The test environment

Page 31: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 31

Environment fidelity

Test environments should be representative of actual use environments to the extent that is necessary and practical.Increased fidelity (e.g. noise, distraction) should be guided by preliminary analyses.

Page 32: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 32

7. Test results

Page 33: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 33

Objective data

Objective data must be gathered during usability validation. Count and explain performance failures for critical and essential tasks. These include:

• Actions (or inactions) that would cause harm or make treatment ineffective

• Confusion, difficulty, and close calls that could lead to harm

Page 34: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 34

Subjective assessment

Subjective data must also be gathered during usability validation through interview-based retrospective analysis. Inquire about:

• Each failure, close call, or difficulty

• Any difficulties not noted by the observer

• Thoughts on all critical and essential tasks

Page 35: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 35

Key takeaways

• Usability testing is about optimizing the user interface to ensure safety.

User preference or acceptance is secondary.

• Explore risks with formative (early/often before design freeze),

and prove safety with validation

• Adjust test fidelity as needed per preliminary analyses

• Avoid internal participants, group tests, and subjective rating scales

Validation is not just a list of failures, but an analysis of why

Page 36: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 36

Bibliography and references

Dumas, J. & Loring, B. Moderating Usability Tests. Elsevier, 2008.

FDA. Applying Human Factors and Usability Engineering to Medical Devices to Optimize Safety and Effectiveness in Design. 2011.

Kaye, R., North B., Edwards, E & Israelski, E. Human Factors for Medical Devices. Lecture conducted from AAMI, Arlington, VA. 2013.

Wiklund, M., Kendler, J. & Strochlic, A. Usability Testing of Medical Devices. CRC, 2005.

Page 37: 7 Components to Medical Device Usability Testing Success

Confidential Copyright © 2014 August 21, 2014 37

We enjoy a good challenge..

Chris CollinsChallis HodgeCHIEF STRATEGY OFFICER

[email protected]

+1.513.417.0006

DIRECTOR OF DESIGN

[email protected]

+1.513.214.3726