wh2014 workshop: mhealth evidence
DESCRIPTION
Wireless Health 2014 Conference Workshop. Speakers include David Mohr, PhD, Northwestern University; Inbal Nahum-Shani, PhD, University of Michigan; and Bethany Raiff, PhD, Rowan University.TRANSCRIPT
WLSACONVERGENCE SUMMIT
MHEALTH EVIDENCE: EVALUATING MOBILE AND WIRELESS HEALTH
AUDIE ATIENZA, NATIONAL CANCER INSTITUTE, NIH
BETHANY RAIFF , ROWAN UNIVERSITYINBAL NAHUM-SHANI , U. MICHIGANDAVID MOHR, NORTHWESTERN UNIVERSITY
Title of your module
Your name, title and affiliation
http://obssr.od.nih.gov
mHealth Evidence WorkshopWireless Health 2014October 29, 2014NIH Campus, Bethesda, MD
3
In this training, you will work in groups to explore different methods for evaluating mobile and wireless technologies
There are participants from across academia, industry, and government participating
The objective is to have a better understanding of when to use different study designs and how the choice effects what information you get from your research
Today is divided into three modules:
– Single case study design
– SMART and Factorial Designs
– Design for Evolving Behavioral Intervention Technologies
Each module is scheduled for 1 hour and will include:
– A briefing to provide a common background and understanding for discussion
– Discussion among the participants
In each module discussion, participants will discuss these questions:
– How does what was just presented effect your project?
– What questions can you answer using the method?
4
Agenda 9:00-9:15 Introduction to the workshop
» Audie Atienza, PhDNational Cancer Institute, NIH
9:15-9:45 Single case study design
» Bethany Raiff , PhDRowan University
9:45-10:15 Small Group breakouts
10:15-10:30 Report back
10:30-11:00 SMART and Factorial Designs
» Inbal Nahum-Shani , PhDUniversity of Michigan
11:00-11:30 Small Group breakouts
11:30-12:00 Design for Evolving Behavioral Intervention Technologies
» David Mohr, PhDNorthwestern University
12:00-12:15 Closing and questions
5
Today’s taskYour goal for this look at different study designs to test the intervention in the following scenario:– Your hospital has identified smoking cessation as a critical issue for its
quality improvement program. They would like to include an intervention to increase smoking cessation in hospitalized patients. Their current psychoeducational program and smoking cessation medications, the average rate for smoking cessation at two months is 15%. The hospital has asked you to help them create an intervention that can provide sustained support for their affected adult patient population (ages 40-60, approximately 1000/year). Based on recent research, your team decides that the components of successful cessation programs include: education on smoking cessation, follow up with patients after they leave the hospital, electronic information and follow-up, stress reduction tools and pharmacy refills for smoking cessation medications delivered through a wireless system that is geared to release the meds to match people’s high craving times.
6
Questions for Breakouts
As you apply the method to your problem, answer the following questions:–How does what was just presented effect
your project?–What questions can you answer using the
method?
Single case experimental designs
Bethany Raiff, PhD, BCBA-D
Rowan University
Outline
Introduce single-case designs (SCDs) to evaluate behavioral data
Introduce each SCD and discuss advantages and disadvantagesProvide examples
Analysis of SCDs
Single Case Design (SCD)
Focus on one individual at a time aka n-of-1 designs (even though n in a study is > 1) Not the same as a “Case study”
Common characteristics: Repeated assessment Replication of effects Each case serves as his/her/its own control
When to use SCDs?
Able to repeatedly assess behavior/symptoms over time
Desire to change behavior/symptoms (not merely assess) and establish preliminary efficacy
When it is unethical to withhold treatment from some participants.
Can be applied across a range of funding levels
SCDs
Types of designs Reversal-replication (e.g., ABA,
ABAB) Alternating treatments Multiple-baseline Changing criterion Combination
Talia
Day
0
2
4
6
8Baseline
Intervention
Baseline
Edward
0 2 4 6 8 10 12 14 16
Da
ily fre
qu
en
cy o
f b
loo
d g
luco
se te
stin
g
0
2
4
6
8
Bonita
Andrea
Day
0 2 4 6 8 10 12 14 16
Baseline
Intervention
Baseline
Ran out of strips
Reversal Design: Increasing blood glucose testing with Internet-based incentives (Raiff & Dallery, 2010)
$35
$45
$48
Reversal Design: Increasing peer interactions (Allen et al, 1964)
Alternating treatments design (Kratchowill et al., 2012)
Alternating Treatments: Promoting moderate-to-vigorous activity (Larson et al., 2014)
Reversal and Alternating Treatments Designs
Advantages Within-subject replication Clear demonstration of the effect of the independent variable Flexible (ABA, ABAB, ABACACB, BAB)
Disadvantages Must remove a treatment to demonstrate experimental control
(not always ethical) Risk of carryover and order effects Will not work with irreversible treatments
Multiple-Baseline: Adherence to weight management with personal electronic device (Cushing et al., 2010)
Multiple Baseline: Different communities (Hawkins et al, 2007)
Changing Criterion Design: Decreasing daily smoking (Hartmann & Hall, 1976)
Changing Criterion: Increasing number of steps taken per day in sedentary adults (Kurti & Dallery, 2013)
Multiple Baseline & Changing Criterion Designs
Advantages: Do not need to remove an effective
treatment Flexible (behavior, settings, participants)
Disadvantages: Requires quantitative criteria that can be
targeted in step-wise fashion (CCD) May need more subjects to convincingly
show experimental control
Data analysis
Is there a reliable effect of the intervention?
What is the magnitude of the effect?
Are the results clinically meaningful and socially valid?
Emphasis on individual subject effects.
Statistical analyses can be used along with visual analysis.
Variability, overlapping data, trends
Mean Shift
Level Shift
Emphasis on individual subjects (Dallery et al., 2013)
Emphasis on individual subjects(Dallery et al., 2013)
Standards for evaluating SCDs The following guidelines have been
proposed: Systematically manipulate an independent
variable Collect inter-observer agreement Phase must include at least THREE data
points (preferably FIVE or more) At least THREE attempts to demonstrate an
intervention effect at different time points (e.g., participants, conditions, criteria, etc)
Kratchowill et al. (2012); Logan et al. (2008)
General Conclusion about SCDs
Rigorously and efficiently establish feasibility and preliminary efficacy
Overcome ethical barriers to withholding or discontinuing effective treatments
Obviate logistical issues (e.g., access to a large number of participants)
For more information about SCDs
Books Barlow & Nock (2008). Single-case experimental designs: strategies for
studying behavior change. Pearson Publishers Kazdin (2010). Single-case research designs: methods for clinical and
applied settings. Oxford PublishersJournal Articles
Dallery, J. & Raiff, B.R. (2014). Optimizing behavioral health interventions with single-case designs: from development to dissemination. Translational Behavioral Medicine: Practice, Policy and Research (online first).
Dallery, J., Cassidy, R., Raiff, B.R. (2013). Single-case experimental designs to evaluate technology-based health interventions. Journal of Medical Internet Research, 15, online. doi: 10.2196/jmir.2227
Thank you!
Questions?
NIH mHealth Institute Event
Inbal Nahum-Shani
Experimental Designs for Optimizing Interventions
Key DefinitionKey Definition
Multi-Component Interventions• Component:
► The content of the intervention (e.g., topics in prevention program)► The intervention modality (e.g., phone calls/emails) ► Features to promote compliance or adherence (e.g., reminder emails)
Example: • Optimizing a technology supported lifestyle intervention for weight loss:
Bonnie Spring, PI. DK097364 ► Telephone Caching ► Report to Primary Care Provider ► Text Messages ► Meal Replacements► Buddy Training
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 33
How do We Typically Develop How do We Typically Develop Interventions?Interventions?
1. Theoretical Model
1. Theoretical Model
2. Intervention Components
2. Intervention Components
4. Confirm Effectiveness
4. Confirm Effectiveness
3. Intervention Package
3. Intervention Package
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 34
How do We Typically Develop How do We Typically Develop Interventions?Interventions?
1. Theoretical Model
1. Theoretical Model
2. Intervention Components
2. Intervention Components
4. Confirm Effectiveness
4. Confirm Effectiveness
3. Intervention Package
3. Intervention Package
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 35
Open Questions Open Questions Efficacy of Individual components► Which components are effective?► Which level is more appropriate?► Which components work well together?
Sequencing of components► Which component to offer first?► Which to offer subsequently?► How should I tailor components over time?
Briefly describe the overarching guiding principles for your work
► Sub-bullets of explanation List and define Key Terms necessary to work with experts in your field
Thursday, April 13, 2023© Inbal Nahum-Shani 2014 36
Open Questions Open Questions Efficacy of Individual components► Which components are effective?► Which level is more appropriate?► Which components work well together?
Sequencing of components► Which component to offer first?► Which to offer subsequently?► How should I tailor components over time?
Briefly describe the overarching guiding principles for your work
► Sub-bullets of explanation List and define Key Terms necessary to work with experts in your field
Factorial DesignsFactorial Designs
SMARTSMART
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 37
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
► Should I include Text Messages?• Factor 1: Text (On/Off)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 38
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
► Should I include Text Messages?• Factor 1: Text (On/Off)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Experimental conditions 2X2 factorial N=400
Experiment Condition
Factor
Text Meal
1 (N=100) On On
2 (N=100) On Off
3 (N=100) Off On
4 (N=100) Off Off
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 39
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
► Should I include Text Messages?• Factor 1: Text (On/Off)• Main effect: On (N=200) vs. Off (N=200)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)
Experimental conditions 2X2 factorial N=400
Experiment Condition
Factor
Text Meal
1 (N=100) On On
2 (N=100) On Off
3 (N=100) Off On
4 (N=100) Off Off
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 40
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
► Should I include Text Messages?• Factor 1: Text (On/Off)• Main effect: On (N=200) vs. Off (N=200)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)• Main effect: On (N=200) vs. Off (N=200)
Experimental conditions 2X2 factorial N=400
Experiment Condition
Factor
Text Meal
1 (N=100) On On
2 (N=100) On Off
3 (N=100) Off On
4 (N=100) Off Off
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 41
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
► Should I include Text Messages?• Factor 1: Text (On/Off)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)
► Should I include Buddy Training?• Factor 3: Buddy (On/Off)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 42
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
Experimental conditions 2X2X2 factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
► Should I include Text Messages?• Factor 1: Text (On/Off)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)
► Should I include Buddy Training?• Factor 3: Buddy (On/Off)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 43
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
Experimental conditions 2X2X2 factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
► Should I include Text Messages?• Factor 1: Text (On/Off)• Main effect: On (N=200) vs. Off (N=200)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)
► Should I include Buddy Training?• Factor 3: Buddy (On/Off)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 44
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
Experimental conditions 2X2X2 factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
► Should I include Text Messages?• Factor 1: Text (On/Off)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)• Main effect: On (N=200) vs. Off (N=200)
► Should I include Buddy Training?• Factor 3: Buddy (On/Off)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 45
Factorial DesignsFactorial Designs
Factorials: More than 1 factor; levels of each factor crossed with levels of other factors.
Experimental conditions 2X2X2 factorial N=400
Condition Factor
Text Meal Buddy
1 (N=50) On On On
2 (N=50) On On Off
3 (N=50) On Off On
4 (N=50) On Off Off
5 (N=50) Off On On
6 (N=50) Off On Off
7 (N=50) Off Off On
8 (N=50) Off Off Off
► Should I include Text Messages?• Factor 1: Text (On/Off)
► Should I include Meal Replacement?• Factor 2: Meal (On/Off)
► Should I include Buddy Training?• Factor 3: Buddy (On/Off)• Main effect: On (N=200) vs. Off (N=200)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 46
SMART Designs SMART Designs
Adaptive Intervention:
► Intervention in which components are sequenced and adapted over time in a way that addresses the specific and changing needs of participants
Motivation in the context of technology-supported interventions:
► Cost: Try less expensive components first and more expensive components later - for those who really need them
► Boredom: Introduce new components over time to increase interest ► Cognitive overload: Offering all components simultaneously -
difficulty to allocate attention
SMART:
► Randomized Trials ► Multiple stages of randomization► Each stage corresponds to a critical question concerning the sequencing and
adaptation of intervention options over time
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 47
SMART DesignsSMART Designs
Hypothetical Example► Aim: Develop an adaptive technology-supported weight loss intervention
• I consider 3 components: Phone, Text and Buddy • Phone is most expensive; Text least expensive
► What is the best way to sequence and adapt?• Which component to offer first: Phone or Text • Which component to add for non-responders: Phone or Buddy
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 48
Questions We Can Address with Questions We Can Address with SMARTSMART First-stage intervention component:
► Is it better to start with Phone Coaching or Text Messages?► (SG1+SG2+SG3) vs. (SG4+SG5+SG6) ► Phone Coaching vs. Text Messages
• Controlling for subsequent intervention component
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-down (SG4)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 49
Questions We Can Address with Questions We Can Address with SMARTSMART Second-stage intervention component:
► Is it better to add Phone Coaching or Buddy Training?► (SG2+SG5) vs. (SG3+SG6) ► Phone Coaching vs. Buddy Training
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 50
Questions We Can Address with Questions We Can Address with SMART SMART Embedded adaptive interventions
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Stage 1 = {Phone}, Then
IF response = {NO} THEN stage 2 = {Add Phone} ELSE IF response = {YES} THEN stage 2= {Step-Down}
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 51
Questions We Can Address with Questions We Can Address with SMART SMART Embedded adaptive interventions
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Stage 1 = {Phone}Then, IF response = {NO} THEN stage 2 = {Add Buddy}ELSE IF response = {YES} THEN stage 2= {Step-Down}
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 52
Questions We Can Address with Questions We Can Address with SMART SMART Embedded adaptive interventions
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Stage 1 = {Text}, ThenIF response = {NO} THEN stage 2 = {Add Buddy} ELSE IF response = {YES} THEN stage 2 = {Step-Down}
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 53
Questions We Can Address with Questions We Can Address with SMART SMART Embedded adaptive interventions
Stage 1 = {Text}, ThenIF response = {NO} THEN stage 2 = {Add Phone} ELSE IF response = {YES} THEN stage 2 = {Step-Down}
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 54
Questions We Can Address with Questions We Can Address with SMART SMART Embedded adaptive interventions
Stage 1 = {Text}, ThenIF response = {NO} THEN stage 2 = {Add Buddy} ELSE IF response = {YES} THEN stage 2 = {Step-Down}
VS.VS.
Stage 1 = {Phone}, Then
IF response = {NO} THEN stage 2 = {Add Phone} ELSE IF response = {YES} THEN stage 2= {Step-Down}
R
Phone
Text
Response
Non-Response RBuddy (SG3)
Phone (SG2)
Response
Non-Response RBuddy (SG6)
Phone (SG5)
Step-Down (SG1)
Step-Down (SG4)
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 55
Summary Summary
Factorial Designs:
Efficacy of Individual components
► Which components are effective?► Which level is more appropriate?► Which components work well together?
SMART Designs:
Sequencing and adaptation of components
► Which component to offer first?► Which to offer subsequently?► How should I tailor components over time?
Thursday, April 13, 2023 © Inbal Nahum-Shani 2014 56
Evaluation of Behavioral
Intervention TechnologiesDavid C. Mohr, Ph.D.Ken Cheung, Ph.D.
Stephen M. Schueller, Ph.D.
Northwestern University&
Columbia University
WHAT’S A BIT?BITs are applications that use technologies such as mobile phones, computers, tablets, and sensors, to support behaviors that improve health, mental health, and wellness.
bit = behavioral intervention technology
COMPUTERS
SENSORS
MOBILE PHONES
TABLETS
RCTs IN BEHAVIORAL AND PSYCHOLOGICAL HEALTH
• RCT Methods • A fixed intervention (pharmacological
agent, device).
• Development of methods of evaluation for psychological interventions (Eysenck, 1952)• Intervention locked down, with
manualization, therapist training and supervision.
Timeline for Technology Development
2000 2012201020042002 2014
Android release
2006
End of Symbian
2016
Obtain Grant
Develop & Pilot
Recruit
Follow-up Eval
Data Analysis & Pub
1-2 yrs
1-2 yrs 1-2 yrs
1-3 yrs
1 yr
Academic Timeline
2008
Symbian release
Problems
• Mismatch between research and technology innovation timeline.
• BITs change in the real world• Patient expectations change• Patient Horizon
• Meds – 10s of millions over decades• Behavioral – 10s of millions over decades• BITs – potentially millions, over small
period of time
Should we be testing BITs in traditional RCTs ?
WHAT’S A PRINCIPLE?
Evaluate [Behavioral Strategy], delivered using [Detailed description of essential BIT functionality], to affect [Clinical outcome].
Principle = An underlying rule or model
BIT MODEL:Why, What, How,
WhenHow:
Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
Behavioral
Instantiation
cbits.northwestern.edu
Mohr et al. J Med Internet Res 2014;16(6):e146
BIT MODEL:Why, What, How,
WhenHow:
Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
Why:Clinical Aims
•Lose Weight• Increase Exercise• Decrease Calories
•Decrease Depression• Increase Positive
Activities• Decrease
Avoidance•Usage Aims
Why:Clinical Aims
•Lose Weight• Increase Exercise• Decrease Calories
•Decrease Depression• Increase Positive
Activities• Decrease
Avoidance•Usage Aims
cbits.northwestern.edu
Mohr et al. J Med Internet Res 2014;16(6):e146
BIT MODEL:Why, What, How,
WhenHow:
Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
How:Behavioral Strategies
•Education•Goal Setting•Monitoring•Feedback•Motivation Enhancement
How:Behavioral Strategies
•Education•Goal Setting•Monitoring•Feedback•Motivation Enhancement
cbits.northwestern.edu
Mohr et al. J Med Internet Res 2014;16(6):e146
BIT MODEL:Why, What, How,
WhenHow:
Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
What:Elements
•Information Delivery•Notifications•Logs•Passive Data Collection•Reports
• Visualizations
What:Elements
•Information Delivery•Notifications•Logs•Passive Data Collection•Reports
• Visualizations
cbits.northwestern.edu
Mohr et al. J Med Internet Res 2014;16(6):e146
BIT MODEL:Why, What, How,
WhenHow:
Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
How:Characteristics
•Medium (Text, Audio, Video)•Complexity•Aesthetics•Tailored to user in other ways
How:Characteristics
•Medium (Text, Audio, Video)•Complexity•Aesthetics•Tailored to user in other ways
cbits.northwestern.edu
Mohr et al. J Med Internet Res 2014;16(6):e146
BIT MODEL:Why, What, How,
WhenHow:
Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
When:Workflow
•User defined•Conditions
• Time-based rules• Task completion
rules• Event-based rules
•Tunneling•Frequency•Length of treatment
When:Workflow
•User defined•Conditions
• Time-based rules• Task completion
rules• Event-based rules
•Tunneling•Frequency•Length of treatment
cbits.northwestern.edu
Mohr et al. J Med Internet Res 2014;16(6):e146
PRINCIPLE STATEMENT
How:Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
cbits.northwestern.edu
Evaluate [Behavioral Strategies], delivered using [Elements, Characteristics, Workflow], to affect [Clinical Aim] .
PRINCIPLE STATEMENT EXAMPLE
How:Behavioral Strategies
How:Behavioral Strategies
Why: Clinical Aims
Why: Clinical Aims
What: BIT Elements
What: BIT Elements
How:Characteristics
How:Characteristics
When: Workflow
When: Workflow
cbits.northwestern.edu
MyFitnessPal aims to support users in goal setting and self-monitoring, using logging features and feedback to support weight loss and physical activity. •Changes over the last years
• Aesthetics• Added of barcode scanning• Added social networking
TRIGGERS FOR CHANGES
• Use data – Expected vs. observed• User feedback – qualitative
information about the problem• Clinical outcomes
DECISION CRITERIA
Does the change interfere with a primary principle being tested?•Clinical Aim: Usually off limits.•Behavioral Strategies: Often these are clearly defined, and therefore would not be changeable. •Instantiation Components (Elements, Characteristics, Workflow): These are often the target of engineering studies, but less commonly the target of clinical research.
• But may affect behavioral strategies.
How:Behavioral Strategies
How:Behavioral Strategies
Why: Clinical
Aims
Why: Clinical
Aims
What: BIT
Elements
What: BIT
Elements
How:Characteristi
cs
How:Characteristi
cs
When: Workflow
When: Workflow
DECISION CRITERIA
If the introduction of the change is successful, would it create an alternative explanation for the success of the trial?
• Unintended consequences
DOCUMENTATION AND REPORTING
Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and Highly Recommended comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was “frozen” during the trial.
EVALUATION OF CLINICAL OUTCOMES• Use standard evaluation methods
• Assuming changes improve primary targets and do not open alternative interpretations, this would provide a conservative estimate.
• May wish to examine if substantive changes are associated with improved use or outcomes.• Include a time varying ordinal component.
Decisions with Uncertainty
• Uncertainty about 2 or more solutions (optimization problem).• Adaptive Randomization using reinforcement
learning techniques (Q-learning).
Trial within a trial
PRINCIPLED TRIAL
• If viewed as an optimization problem (incorporate learning into trial), accuracy-centric framework and type I error rate not appropriate. • P-values closer to .50
may provide reasonable confidence for optimization but would not permit generalizability.
End of Tx
RR
ControlControl
BITBIT
Clinical AimsClinical Aims
Beh Strategies Beh Strategies
BIT ElementsBIT Elements
CharacteristicsCharacteristics
WorkflowWorkflow
CC
CC
CC
EE
CC
Cheung K, Duan N.. Am. J. Public Health. 2014;104:e23-e30.
SUMMARY• Principled Trials
• Define principles (Behavioral and technical)• Decision making methods• Documentation & Reporting• Evaluation
• Limitations• Does not validate a specific app• Risk of unintended consequences
• Advantages• Consistent with what BITs are• Learning during trial• Cost effective knowledge generation• Produces more generalizable data
WLSACONVERGENCE SUMMIT
www.wirelesshealth2014.org