Amanda ReckerJamie Pina, MSPH, PhD

Barbara L. Massoudi, MPH, PhD

RTI International

2013 International Symposium on Human Factors and Ergonomics in Health Care:

Advancing the CauseMarch 11, 2013

Long-term Usability Testing for Public Health Information Technology:

BioSense 2.0

BioSense 1.0: Web-based syndromic surveillance• Mandated in the Public Health Security and

Bioterrorism (BT) Preparedness and Response Act of 2002

• Nationwide integrated system for early detection and assessment of potential BT-related illness

• 2003 Funding provided by Congress to CDC• Development of BioSense infrastructure started, initial

focus on:– VA and DoD– Direct reporting to CDC of detailed clinical data by

civilian hospitals

• Began soliciting more limited data from health departments (HDs) that had already established automated systems for ED-based syndromic surveillance– By 2007, 8 state/local HDs connected

Recommendations from Prior Evaluations• Strengthen state and local public health engagement

– Enhance state/local HD syndromic surveillance capacity– Increase participation of state/local HD syndromic

surveillance systems (improve coverage)– Share data with HDs from hospitals reporting directly to

CDC– Share governance with public health community

• Leverage investments in electronic health records (EHRs)

• GAO, 2008: Adopt an “open, distributed computing model”

• Improve utility of the data and data sources• Preparedness role: Greater “all hazards” emphasis• Expand uses for broader spectrum of public health

concerns

BioSense 2.0: Timeline• June 2010: Redesign begins• November 2011: Opened for business• November 2011 – June 2013: Onboarding new

jurisdictions – 35 jurisdictions signed the Data Use Agreement (DUA)– 17 fully onboarded

• April 2012: Retired BioSense 1.0

BioSense 2.0: Approach• Shift from a need-to-know to a need-to-share and co-

create approach• User-centered design

– Stakeholders engaged in every step of the redesign– HDs fully control “their data” at the level of granularity

they choose– More options for data sharing with other jurisdictions

and CDC

• Alignment with ONC and Meaningful Use– Agreed-upon core syndromic surveillance data elements– Collaborations with public health professional

associations– Funding to states: Meaningful Use syndromic

surveillance adoption, build capacity, join BioSense 2.0

• Cloud technology: distributed, easy to adopt, cost effective, secure

Application Home Page

Encephalitis, Meningitis, WNV = CNS Inflammatory Disease

Why Long-term Usability Testing?

• Long user-centered design lifecycle• Expectation management• Stakeholder ownership• BioSense 2.0 continuously changing and growing• Longitudinal usability testing

– User satisfaction– Efficiency

• Functions not changing: building a query, viewing results, analyzing the data, sending and saving information

• Two approaches to testing

1. User-Centered Design• Qualitative data collection methods• The user knows best• Test the right participants• Broad range of public health professionals

– Public health generalist – less sophisticated users– Syndromic surveillance epidemiologist – more

sophisticated users

2. Activity-Centered Design

• Quantitative data collection methods• Behavior vs. opinions• Based on empirical data

– Time-on-task analysis– Mouse-click analysis– Pathway analysis

1. Establish requirements2. Design alternatives3. Develop prototype4. Conduct evaluation

Basic Activities in Interaction Design

(Rogers, Sharp, & Preece, 2011)(Rogers, Sharp, & Preece, 2011)

How to Choose Users

• Interact directly with the system– Epidemiologists, state and local public health

professionals

• Manage direct users – decision-makers– Public health directors

• Use similar syndromic applications

• User-centered design – Open question response– Focus groups– Expectation testing– SUS

• Expert evaluation– Usability heuristics (Nielsen)

• Activity-centered design– Morae software– Scenarios and tasks– Closed/open

question response– Time on tasks– Mouse-click analysis– Pathway analysis– Critical incidents

Generating Design Alternatives

Scenarios and Tasks

Scenario: “Over dinner at [a public health conference] an argument has erupted, but luckily as a BioSense 2.0 user you can settle this dispute. Health authorities in Virginia suspect that the flu season was more severe than it was in Michigan.”

Task: “Please determine which state, Virginia or Michigan, had more cases of influenza-like illness (ILI) starting in October 1, 2010 through March 1, 2011.”

Pathway Efficiency Analysis• Time on task• Mouse-click analysis• Pathway analysis

Design Alternatives

How to Choose Among These Alternatives?• If one person says something is a problem, do you

change the design?• Resolved conflicting alternatives• Conducted feasibility analysis• In the end…relied on face validity

What Happens After Design Changes?

• User training through webinars and videos• Expert user testing• Focus group sessions• Continually comparing SUS scores

Future Activities

• Incorporate eye tracking into testing protocols• Conduct on-site testing and evaluation • Task analyses of routine versus event surveillance• Information models for routine and event surveillance