medical device usability: polly shelton presents at uk upa (usability professional’s association)...

Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk1

Click icon to add picture

| Human Factors Consultant| Polly Shelton

Medical Device Usability UKUPA and LEG, January 2012



Industrial Design

FMCG

Consumer

Services

Medical

v

pdd



Industrial Design& Prototyping

1980

Computer AidedDesign & Manufacture

Engineering &Analysis

1990

History



2000 2010

Human factors

Innovation planning

Design Insight



IV Pump

Inhaler dose counter

Heart-rate monitor

Medical oxygen

Wireless stethoscope

Auto injector device

Needle safe catheter

Radiation does meter

Urine meter

Pen injector

Medical devices group


What is a medical device?

Device classes (FDA)• Class I – general controls• Class II – general controls w/ special

controls• Class III – general controls & premarket

approval• & Combination products.

These products should all undergo a usability engineering approach

Depending on the class, more or less HFE required

Class depends on complexity, use characteristics, risk to patient & amount of regulatory control needed

A product used to diagnose, treat or monitor a medical condition



\

Ensuring that products are developed for safe and effective use and that potential use-errors have been mitigated.

Human Factors and Usability

International/EU

ISO/IEC 62366

FDA (USA)

ANSI/AAMI HE:75



HistoryInternational

ISO/IEC 62366

Usability engineering process

International (EU & FDA)

6 pages (+80page annex)

Helps “design in” usability

and “design out” use error

Used with ISO 14971

FDA (USA)

ANSI/AAMI HE:75

Medical device manufacturers must demonstrate that all potential use-related

hazards in their devices have been identified, tested, and mitigatedThe device must be shown

to be usable by the target population in the intended environment.



History

International

ISO/IEC 62366

FDA (USA)

ANSI/AAMI HE:75

Provides design principles and practical

guidelines

25 chapters, 465 pages

Officially recognised by FDA in October 2010

FDA draft guidance document June 2011

A virtual encyclopaedia that provides human factors engineering design guidance, case studies, and checklists



History

2009

AAMI HE:

AAMI HE:48

ANSI/AAMI HE:74

ANSI/AAMI HE:74

1988

1993

2001

Human factors engineering guidelines and preferred practices for the design of medical devices

Human factors engineering guidelines and preferred practices for the design of medical devices

Human factors design process for medical devices

Human factors design process for medical devices (reaffirmed)

ISO/IEC 62366 Medical devices – Application of usability engineering to medical devices2007

ANSI/AAMI HE:75

Human factors engineering – Design of medical devices2009

March 2010, EU

FDA

FDA

November2010, FDA

Institute of Medicine Report : to Err is Human

1999



Process flowchart: IEC/ISO 62366



How do they interact?

ISO 62366 has a complex flowchartshowing how.

Usability & Risk management



Insight Iteration Validation




Iteration Validation

Expert interviews

Desk research

Contextual enquiry

Insight

Ethnographic research

Task analysis



Insight Validation

Iteration

Concept refinement

Formative usability testing

Interaction design & prototyping

Anthropometry & biomechanics

Use error risk analysis



Insight Iteration

Validation

Summative usability testing

Residual risk Usability goals and acceptance criteria

Regulatory submission



Insight Iteration Validation

Contextual enquiry Interaction design

& prototyping

Usability goals and acceptance criteria

Use error risk analysis

Regulatory submission

Summative usability testing

Residual risk


Concept refinement

Formative usability testing

Anthropometry & biomechanics

Desk research

Task analysis

Ethnographic research

Expert interviews



Ethnographic research Observation, shadowing, journaling, contextual enquiry, journey mapping

• Gain a holistic view of your users, the task and the context of use

• Identify opportunities and pain points

• Deep-dive into users needs and experiences

• Create personas, scenarios, journey/experience maps



Desk researchUser and task characteristics, human factors design guidelines

• Understand your stakeholders and the problem

• Review similar products• Gather anthropometric and

strength data• Source best practice design

guidelines



Development & refinementPrototyping, cognitive walkthrough, heuristic review, formative usability tests

• Involve users early and often• Test concepts• Identify design problems early • Understand mental models and user

interactions • Involve designers and clients where possible• Inform the design



Task analysis and user FMEA Cognitive walkthrough, use scenarios, journey and task mapping

• Understand the task • Identify physical, sensory

and cognitive challenges• Identify and understand

potential risks and foreseeable use error

• Control and mitigate use error risk through design



Validation & residual riskFormal summative testing against quantitative usability goals

• Identify critical task scenarios• Validate risk mitigation

effectiveness through testing with users

• Error analysis • Root cause analysis• Residual risk analysis and

mitigation • Follow-up testing• Regulatory submission


Medical device user testing• Why do usability testing?• Formative vs. summative• Recruiting participants• Legal and ethical considerations• Task selection• Setting usability goals• Test environment• Set-up and pilot• Moderating sessions• Providing assistance• Providing training• Testing instructions• Data collection and error logging• Error analysis and classification



Why do usability testing?

Improve ease-of-use, ease of

learning and user experience

Meet device regulators’

expectations

Increase device sales, customer loyalty, reduce

customer support

Improve safety and reduce

medical device use errors



For medical devices, poor usability is literally a matter of life or death



Some stats

301fataliti

es2,38

3serious injuries

and 403 Manufacture

r’s Field Safety

Corrective Actions

undertaken

Deaths and serious injuries

involving medical

devices up

29% last year

10,280 adverse

incident reports in UK 2010 (MHRA) An

estimated15-20% of all

errors due to mechanical failure; 60-80% human

error*

“One third of medical device reports each year involve use-error and human factors are inherent in virtually all device-related incidents.” FDA

100medical device alerts issued

(MHRA,2010)

*Institute of Medicine Report 2000, & Perrow 1984)



What’s the difference?

Formative• Part of the iterative design process• Prototypes of any fidelity • More explorative, informing the

design • Identifies an evolving device’s

strengths and weaknesses• Extensive discussion of likes and

dislikes • Preliminary usability objectives• Small samples (5-8 users per

group) • User research and marketing

questions can easily be incorporated

• Modest size report including design recommendations

• 30-90 minute sessions

Formative vs. summative tests

Summative• Determines if a finished device is vulnerable

to potential harmful use errors• Production equivalent devices• Tasks must include primary operating

functions and all safety-related tasks• Actual or simulated use environments • Focus is on use-safety • Effectiveness, efficiency, user satisfaction &

ease of user learning also evaluated• Usability goals and acceptance criteria • Record use errors, close calls and root

causes • Can be large sample sizes (25+ per user

group) • Training and final IFU may be included• 90 minute + sessions


Recruiting participants• Reflect the actual user population• Define inclusion and exclusion criteria within each user

group• Include users with physical, cognitive and sensory

impairments• Consider co-morbidities• Use a medical specialist agency with BHBIA/EPhMRA• Allow at least 4-6 weeks for recruitment • Over-recruit by 10% • Be prepared for difficult recruits and arranging in-home

testing where necessary• Beware of “professional” participants

‐ ask for ID / registration numbers where possible ‐ patients to bring current medication/ a recent prescription

• Pilot your recruitment materials • Test in different markets


Legal and ethical considerations• Ethics committee / IRB approval• BHBIA and EPhMRA codes of conduct • Data protection act, 1998• ABPI code of conduct • Adverse events and pharmacovigilance• Country-specific requirements• Video recording, streaming and transfer• Incentives (BMA guidance):

‐ minimum level ‐ proportionate to time involved‐ appropriate to respondent type and task

nature‐ air market value of the services provided‐ vouchers and not cash for children

• Testing with children


Task selection• If possible, test every user task!

• If not, prioritize by: ‐ Safety Critical tasks, and‐ Frequent tasks

• Address worst case scenarios where possible

• Link tasks to risk management


Usability goals • Derived from:

‐ Risk analysis ‐ Primary operating functions ‐ User interface requirements ‐ Worst case scenarios

• Define acceptance criteria by criticality and risk level

But…. • Don’t base the success of your test on the

achievement of usability goals against quantitative acceptance criteria!

• Always conduct:‐ root cause analysis‐ follow-up risk analysis


• Simulate the real context of use where possible– Lighting levels– Thermal environment – Noise– Furniture – Other equipment – Potential hazards– Working space– Multi-tasking and distractions

• Consider testing devices in-field – In the home – In clinical environments (e.g. ICU)

• For high risk products (e.g. infusion pumps) consider validation with clinical evaluation.

Test environment


Set-up and pilot• Test logistics, protocol and timings • Equipment checklist • Video recording

‐ Define work area‐ If possible, use two cameras to capture

detailed interactions and scene-view‐ Make sure the audio is ON! ‐ Remote control

• Live streaming • Seating arrangements:

‐ Sit behind / next to participant ‐ Observer out of field of view

• Consider recruiting “real” pilot participant• Pilot data collection materials and analysis


Moderating sessions• Protect participants’ rights • Be professional and build trust • Create rapport; but stay in charge • Ask open questions• Don’t be a robot! – remember the script is a

guide • Be unbiased • Be consistent• Let the participant speak• Take care with “think-aloud” • Give participants time • Let participants struggle… but keep them safe• Watch non-verbal cues • Probe where appropriate to understand root

causes

• Good seminar from UIE and Beth Loring http://www.uie.com/events/virtual_seminars/good_moderating/


Providing assistance• An assisted task is a task failure • Observing participants struggle, commit use errors or

near misses provides insight • Only assist as a last resort

- safety risk - repeated pattern of unproductive behaviour- task abandonment- after pre-established time-limit- in order to continue

• Provide appropriate levels of assistance• Pre-establish prompts for consistency• All interventions should be reported


Training

• Simulate the real-life training scenario• Consider including a retention interval • Simulate the worst case scenario

where appropriate• Consider including some untrained users

Do not… • Deliver better than normal training!• Ask trainees to study for their test!


Testing instructions

• Simulate the anticipated situation-of-use• Ask users to behave naturally

- refer to IFU/ quick guide as they would at home • Follow-up to address IFU in detail

- test all risk mitigations and warnings• Evaluate and rate specific attributes e.g.

‐ Clarity of text‐ Terminology ‐ Pictograms ‐ Structure & flow ‐ Text size and legibility ‐ Ease of use‐ Ease of finding information ‐ Completeness‐ Density of information


Data collection and error logging• Collect data concurrently • If possible, have 2 people taking notes • Debrief and align data post-session • Review video where required Task log and Excel spread sheet

Data: • Success rate • Use errors, close calls,

operational difficulties and their root causes

• Assistance provided• Verbatim comments • Other observations and

behaviours• Responses to questions and

ratings


Error analysis & classification• FDA are interested in understanding the root causes of

errors• Probe to understand observed errors• ISO 62366 includes a taxonomy (adapted from James

Reason)• Close calls and operational difficulties should also be

recorded


Case Study: IV infusion set• Pre-summative test• 22 participants• 3 hour sessions • 13 primary usability goals• 40 secondary usability goals

10 major use

errors observe

d

15 additional risks identifie

d to FMEA

Contextual

insights gathere

d

Changes recomme

nd to design & instructio

ns

11 primary &14

secondary usability

goals

unmet

Training needs

identified


Case Study: easypod device

ROI: drug sales rose

7.5% in the first year to €58m

User requireme

nts specificati

on

Improved ease-of-use,

ease-of-dosing, comfort,

confidence & safety

Full User Centred Design

approach

Ethnographic

fieldwork identified

need



Further reading Some good resources to get started


Get in contactand keep updated via …

medical device usability: polly shelton presents at uk upa (usability professional’s association)...

Design

confidence pdd group

bs registered

usability engineering

design principles

medical devices2009ansiaami

usability risk managementhow

page annextheir devices

euhuman factors