Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk1
Click icon to add picture
| Human Factors Consultant| Polly Shelton
Medical Device Usability UKUPA and LEG, January 2012
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk2
Click icon to add picture
Industrial Design
FMCG
Consumer
Services
Medical
v
pdd
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk3
Click icon to add picture
Industrial Design& Prototyping
1980
Computer AidedDesign & Manufacture
Engineering &Analysis
1990
History
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk4
Click icon to add picture
2000 2010
Human factors
Innovation planning
Design Insight
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk5
Click icon to add picture
IV Pump
Inhaler dose counter
Heart-rate monitor
Medical oxygen
Wireless stethoscope
Auto injector device
Needle safe catheter
Radiation does meter
Urine meter
Pen injector
Medical devices group
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk6
What is a medical device?
Device classes (FDA)• Class I – general controls• Class II – general controls w/ special
controls• Class III – general controls & premarket
approval• & Combination products.
These products should all undergo a usability engineering approach
Depending on the class, more or less HFE required
Class depends on complexity, use characteristics, risk to patient & amount of regulatory control needed
A product used to diagnose, treat or monitor a medical condition
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk7
Click icon to add picture
\
Ensuring that products are developed for safe and effective use and that potential use-errors have been mitigated.
Human Factors and Usability
International/EU
ISO/IEC 62366
FDA (USA)
ANSI/AAMI HE:75
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk8
Click icon to add picture
HistoryInternational
ISO/IEC 62366
Usability engineering process
International (EU & FDA)
6 pages (+80page annex)
Helps “design in” usability
and “design out” use error
Used with ISO 14971
FDA (USA)
ANSI/AAMI HE:75
Medical device manufacturers must demonstrate that all potential use-related
hazards in their devices have been identified, tested, and mitigatedThe device must be shown
to be usable by the target population in the intended environment.
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk9
Click icon to add picture
History
International
ISO/IEC 62366
FDA (USA)
ANSI/AAMI HE:75
Provides design principles and practical
guidelines
25 chapters, 465 pages
Officially recognised by FDA in October 2010
FDA draft guidance document June 2011
A virtual encyclopaedia that provides human factors engineering design guidance, case studies, and checklists
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk10
Click icon to add picture
History
2009
AAMI HE:
AAMI HE:48
ANSI/AAMI HE:74
ANSI/AAMI HE:74
1988
1993
2001
Human factors engineering guidelines and preferred practices for the design of medical devices
Human factors engineering guidelines and preferred practices for the design of medical devices
Human factors design process for medical devices
Human factors design process for medical devices (reaffirmed)
ISO/IEC 62366 Medical devices – Application of usability engineering to medical devices2007
ANSI/AAMI HE:75
Human factors engineering – Design of medical devices2009
March 2010, EU
FDA
FDA
November2010, FDA
Institute of Medicine Report : to Err is Human
1999
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk11
Click icon to add picture
Process flowchart: IEC/ISO 62366
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk12
Click icon to add picture
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk13
Click icon to add picture
How do they interact?
ISO 62366 has a complex flowchartshowing how.
Usability & Risk management
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk14
Click icon to add picture
Insight Iteration Validation
Usability engineering process
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk15
Click icon to add picture
Iteration Validation
Expert interviews
Desk research
Contextual enquiry
Insight
Ethnographic research
Task analysis
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk16
Click icon to add picture
Insight Validation
Iteration
Concept refinement
Formative usability testing
Interaction design & prototyping
Anthropometry & biomechanics
Use error risk analysis
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk17
Click icon to add picture
Insight Iteration
Validation
Summative usability testing
Residual risk Usability goals and acceptance criteria
Regulatory submission
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk18
Click icon to add picture
Insight Iteration Validation
Contextual enquiry Interaction design
& prototyping
Usability goals and acceptance criteria
Use error risk analysis
Regulatory submission
Summative usability testing
Residual risk
Usability engineering process
Concept refinement
Formative usability testing
Anthropometry & biomechanics
Desk research
Task analysis
Ethnographic research
Expert interviews
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk19
Click icon to add picture
Ethnographic research Observation, shadowing, journaling, contextual enquiry, journey mapping
• Gain a holistic view of your users, the task and the context of use
• Identify opportunities and pain points
• Deep-dive into users needs and experiences
• Create personas, scenarios, journey/experience maps
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk20
Click icon to add picture
Desk researchUser and task characteristics, human factors design guidelines
• Understand your stakeholders and the problem
• Review similar products• Gather anthropometric and
strength data• Source best practice design
guidelines
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk21
Click icon to add picture
Development & refinementPrototyping, cognitive walkthrough, heuristic review, formative usability tests
• Involve users early and often• Test concepts• Identify design problems early • Understand mental models and user
interactions • Involve designers and clients where possible• Inform the design
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk22
Click icon to add picture
Task analysis and user FMEA Cognitive walkthrough, use scenarios, journey and task mapping
• Understand the task • Identify physical, sensory
and cognitive challenges• Identify and understand
potential risks and foreseeable use error
• Control and mitigate use error risk through design
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk23
Click icon to add picture
Validation & residual riskFormal summative testing against quantitative usability goals
• Identify critical task scenarios• Validate risk mitigation
effectiveness through testing with users
• Error analysis • Root cause analysis• Residual risk analysis and
mitigation • Follow-up testing• Regulatory submission
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk24
Medical device user testing• Why do usability testing?• Formative vs. summative• Recruiting participants• Legal and ethical considerations• Task selection• Setting usability goals• Test environment• Set-up and pilot• Moderating sessions• Providing assistance• Providing training• Testing instructions• Data collection and error logging• Error analysis and classification
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk25
Click icon to add picture
Why do usability testing?
Improve ease-of-use, ease of
learning and user experience
Meet device regulators’
expectations
Increase device sales, customer loyalty, reduce
customer support
Improve safety and reduce
medical device use errors
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk26
Click icon to add picture
For medical devices, poor usability is literally a matter of life or death
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk27
Click icon to add picture
Some stats
301fataliti
es2,38
3serious injuries
and 403 Manufacture
r’s Field Safety
Corrective Actions
undertaken
Deaths and serious injuries
involving medical
devices up
29% last year
10,280 adverse
incident reports in UK 2010 (MHRA) An
estimated15-20% of all
errors due to mechanical failure; 60-80% human
error*
“One third of medical device reports each year involve use-error and human factors are inherent in virtually all device-related incidents.” FDA
100medical device alerts issued
(MHRA,2010)
*Institute of Medicine Report 2000, & Perrow 1984)
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk28
Click icon to add picture
What’s the difference?
Formative• Part of the iterative design process• Prototypes of any fidelity • More explorative, informing the
design • Identifies an evolving device’s
strengths and weaknesses• Extensive discussion of likes and
dislikes • Preliminary usability objectives• Small samples (5-8 users per
group) • User research and marketing
questions can easily be incorporated
• Modest size report including design recommendations
• 30-90 minute sessions
Formative vs. summative tests
Summative• Determines if a finished device is vulnerable
to potential harmful use errors• Production equivalent devices• Tasks must include primary operating
functions and all safety-related tasks• Actual or simulated use environments • Focus is on use-safety • Effectiveness, efficiency, user satisfaction &
ease of user learning also evaluated• Usability goals and acceptance criteria • Record use errors, close calls and root
causes • Can be large sample sizes (25+ per user
group) • Training and final IFU may be included• 90 minute + sessions
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk29
Recruiting participants• Reflect the actual user population• Define inclusion and exclusion criteria within each user
group• Include users with physical, cognitive and sensory
impairments• Consider co-morbidities• Use a medical specialist agency with BHBIA/EPhMRA• Allow at least 4-6 weeks for recruitment • Over-recruit by 10% • Be prepared for difficult recruits and arranging in-home
testing where necessary• Beware of “professional” participants
‐ ask for ID / registration numbers where possible ‐ patients to bring current medication/ a recent prescription
• Pilot your recruitment materials • Test in different markets
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk30
Legal and ethical considerations• Ethics committee / IRB approval• BHBIA and EPhMRA codes of conduct • Data protection act, 1998• ABPI code of conduct • Adverse events and pharmacovigilance• Country-specific requirements• Video recording, streaming and transfer• Incentives (BMA guidance):
‐ minimum level ‐ proportionate to time involved‐ appropriate to respondent type and task
nature‐ air market value of the services provided‐ vouchers and not cash for children
• Testing with children
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk31
Task selection• If possible, test every user task!
• If not, prioritize by: ‐ Safety Critical tasks, and‐ Frequent tasks
• Address worst case scenarios where possible
• Link tasks to risk management
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk32
Usability goals • Derived from:
‐ Risk analysis ‐ Primary operating functions ‐ User interface requirements ‐ Worst case scenarios
• Define acceptance criteria by criticality and risk level
But…. • Don’t base the success of your test on the
achievement of usability goals against quantitative acceptance criteria!
• Always conduct:‐ root cause analysis‐ follow-up risk analysis
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk33
• Simulate the real context of use where possible– Lighting levels– Thermal environment – Noise– Furniture – Other equipment – Potential hazards– Working space– Multi-tasking and distractions
• Consider testing devices in-field – In the home – In clinical environments (e.g. ICU)
• For high risk products (e.g. infusion pumps) consider validation with clinical evaluation.
Test environment
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk34
Set-up and pilot• Test logistics, protocol and timings • Equipment checklist • Video recording
‐ Define work area‐ If possible, use two cameras to capture
detailed interactions and scene-view‐ Make sure the audio is ON! ‐ Remote control
• Live streaming • Seating arrangements:
‐ Sit behind / next to participant ‐ Observer out of field of view
• Consider recruiting “real” pilot participant• Pilot data collection materials and analysis
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk35
Moderating sessions• Protect participants’ rights • Be professional and build trust • Create rapport; but stay in charge • Ask open questions• Don’t be a robot! – remember the script is a
guide • Be unbiased • Be consistent• Let the participant speak• Take care with “think-aloud” • Give participants time • Let participants struggle… but keep them safe• Watch non-verbal cues • Probe where appropriate to understand root
causes
• Good seminar from UIE and Beth Loring http://www.uie.com/events/virtual_seminars/good_moderating/
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk36
Providing assistance• An assisted task is a task failure • Observing participants struggle, commit use errors or
near misses provides insight • Only assist as a last resort
- safety risk - repeated pattern of unproductive behaviour- task abandonment- after pre-established time-limit- in order to continue
• Provide appropriate levels of assistance• Pre-establish prompts for consistency• All interventions should be reported
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk37
Training
• Simulate the real-life training scenario• Consider including a retention interval • Simulate the worst case scenario
where appropriate• Consider including some untrained users
Do not… • Deliver better than normal training!• Ask trainees to study for their test!
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk38
Testing instructions
• Simulate the anticipated situation-of-use• Ask users to behave naturally
- refer to IFU/ quick guide as they would at home • Follow-up to address IFU in detail
- test all risk mitigations and warnings• Evaluate and rate specific attributes e.g.
‐ Clarity of text‐ Terminology ‐ Pictograms ‐ Structure & flow ‐ Text size and legibility ‐ Ease of use‐ Ease of finding information ‐ Completeness‐ Density of information
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk39
Data collection and error logging• Collect data concurrently • If possible, have 2 people taking notes • Debrief and align data post-session • Review video where required Task log and Excel spread sheet
Data: • Success rate • Use errors, close calls,
operational difficulties and their root causes
• Assistance provided• Verbatim comments • Other observations and
behaviours• Responses to questions and
ratings
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk40
Error analysis & classification• FDA are interested in understanding the root causes of
errors• Probe to understand observed errors• ISO 62366 includes a taxonomy (adapted from James
Reason)• Close calls and operational difficulties should also be
recorded
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk41
Case Study: IV infusion set• Pre-summative test• 22 participants• 3 hour sessions • 13 primary usability goals• 40 secondary usability goals
10 major use
errors observe
d
15 additional risks identifie
d to FMEA
Contextual
insights gathere
d
Changes recomme
nd to design & instructio
ns
11 primary &14
secondary usability
goals
unmet
Training needs
identified
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk42
Case Study: easypod device
ROI: drug sales rose
7.5% in the first year to €58m
User requireme
nts specificati
on
Improved ease-of-use,
ease-of-dosing, comfort,
confidence & safety
Full User Centred Design
approach
Ethnographic
fieldwork identified
need
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk43
Click icon to add picture
Further reading Some good resources to get started
Page COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk44
Get in contactand keep updated via …