paul bolton applied mental health research group johns hopkins bloomberg school of public health
DESCRIPTION
Public health systems: holding governments accountable. Establishing standards, measuring implementation. Paul Bolton Applied Mental Health Research Group Johns Hopkins Bloomberg School of Public Health Baltimore, USA . Outcome Measures. - PowerPoint PPT PresentationTRANSCRIPT
Public health systems: holding governments accountable. Establishing standards,
measuring implementation
Paul BoltonApplied Mental Health Research Group
Johns Hopkins Bloomberg School of Public HealthBaltimore, USA.
Site population problems interventionSouthwest Uganda adults depression IPT-G
Northern Uganda Displaced adolescents
Maladaptive behavior, depression
IPT-GCreative play
Indonesia (Aceh) Adults affected by conflict
Depression, distress Non-specific counseling
Kurdistan (Iraq) Genocide, repression
Depression, anxiety, PTSD, Traumatic Grief
CPTBAnon-specific
Southern Iraq Conflict, repression.
As above CPT, CETA
Thai Burma border Displacement, repression,
Depression, anxiety, PTSD
CETA
Colombia Repression Depression, anxiety, trauma
CETA
Democratic Republic of Congo
Female sexual violence
Depression, anxiety, trauma, stigma
CPT VSLA
Zambia Child sexual abuse Depression, withdrawal trauma, stigma
TF-CBT
Outcome Measures
• Survivors participate in the rehabilitation process (GC3)
• Programmes…take into account a victim’s culture, personality, history and background (GC3)
• So how can we have standard instruments/measures when client needs and situations vary?
Some mental health problems are both predictable and similar:
– Depression (Hopkins Symptom Checklist or HSCL)– Anxiety (HSCL)– Trauma (Harvard Trauma Questionnaire or HTQ)
But others vary
A Qualitative Approach to outcome measures
1. Start with free listing asking:‘what are the problems of people who have been tortured?’
2. List all commonly mentioned problems, then choose priority problems based on frequency and severity.
3. Key informant interviews to explore those problems.
• Feeling handicapped • Social injustice. We are not treated equally• Divorce• Poverty• Drinking alcohol • We are not respected as we should be. We feel inferior • We are insulted; especially women and girls are called
names, that you are raped. • We regret helping this government. • Social relationships have become weak.• Disappointment. Nothing has been done for us. • Rage • No-one is honest. So we are obliged to do bad things
Qualitative Problem descriptions used to:
• Other problems are included as separate questions
• Add to depression/anxiety/trauma instruments to increase local validity– Thinking too much– Cannot accept person is gone– hating the world
Qualitative Problem descriptions used to:
• Used for translation
• Decide on appropriate interventions
Program Monitoring
• Monitoring – constantly tracking indicator.• Purpose of monitoring is to identify and address
problems as they occur (iterative)• Evaluation – determining if there is a change in
the indicator between program beginning and end.
• However: Everything that is evaluated is also monitored.
program monitoring and evaluation for “effective Implementation”*†
FidelityAvailability/Access*UptakeSurvivor Compliance/CooperationAppropriateness/Acceptability*FeasibilityCostEffectiveness*
†Dissemination and Implementation Research
Fidelity Monitoring
Gender M Main Syxs Nightmares, anxious, tense, sleeping problems
Behavior Act Need Score (/9)
3
Trauma Type
Prison, Torture, Military attack, taken from family
Total Score 59 Functioning Items (general/out of 84)
55
Relax Need Score (/12)
12 Live Exposure Score (/3) 0
Age 40 Clinical Observations: rubbing hands and looking around room Flow ___ Standard _X__ + Relaxation
___ + Behavioral Activation ___ + Live Exposure
Accessibility/Reach
Can be defined in various ways:distancecosttimeopportunity cost.
Uptake
• How many of those who have access to services and know about them, try them?
• Can be defined as:
Uptake
Uptake
Compliance/cooperation
• How many survivors who begin treatment complete it?
Compliance
Appropriateness/Acceptability
• Combination of uptake and compliance:
• If high uptake and compliance, program is considered acceptable.
• If either is low, acceptability is considered low.
Feasibility
• Requires a vision of who is going to pay for the services for as long as they are needed.
• Can this payer(s) afford to pay for the duration?• Is this payer(s) willing to pay for the duration?
Effectiveness
• Does NOT refer to whether survivors improve.
Refers to what would happen to the survivor in the absence of services:
survivor would be worse off: effectivesurvivor would be the same: not effectivesurvivor would be better off: not effective
and harmful.
Reduction of depression symptoms by group
25
30
35
40
45
50
Baseline Follow-Up
Dep
ress
ion
Sym
ptom
Sc
ores
Reduction of depression symptoms by group
25
30
35
40
45
50
Baseline Follow-Up
ControlIPT-GCP
Dep
ress
ion
Sym
ptom
Sc
ores
Effectiveness: what outcomes?
• “Rehabilitation…refers to the restoration of function or the acquisition of new skills required as a result of the changed circumstances of a victim in the aftermath of torture or ill-treatment.”
• Ultimate goal is restoration of dignity.
• Interpretation: Main aim is the restoration of survivor’s roles in terms of self, family and society.
How to operationalize this?
Main impact outcomes are the functions that make up locally defined roles.
These vary, so there is no single instrument.
Replace single instrument with process that has survivors define them locally.
Function Assessment
• Qualitative methods:– Free listing
• What are the activities that men/women normally do (to take care of themselves/family/community?
• What do children normally do?• How do you know when a man/woman/child is doing
well.• What are the activities that survivors cannot do that
they need to do?
Iraq (women)• Housework• Cooking• Other manual labor• Caring for family
members• Giving advice• Exchanging ideas• Having harmonious
relationship with family• Raising children
correctly• Contributing to the
community• Sympathize with others
• Visiting and socializing• Asking for help • Getting help• Making decisions• Taking part in family
activities • Taking part in community
activities• Learning something new• Concentrating• Dealing with strangers • Attending mosque or
religious gathering• Assisting others
Holding states accountable
• Treat the state as a partner whose priorities must also be met.
• Try to help the state meet them.• Typical state priorities:
– Increase access to effective health services– Reduce mortality for the whole population– Reduce morbidity for the whole population
Meeting State and Survivor Needs
•Address priority problems OF torture survivors, not just focus on problems DUE TO torture.•Most problems of torture survivors are shared by many others.•Where possible, support services that deal with these problems for everyone. Survivors should access these same services as others.
New Structure in Iraq• Most survivors receive mental health and counseling services integrated
into physical health system which addresses these problems for everyone (ie, also non-torture survivors)
• Providers are primary clinic staff with little mental health background who are trained to provide effective psychotherapy
• Supervisors are mental health professionals based in psychiatric centers and in torture treatment centers.
• Referrals (non-torture) to psychiatric centers• Referrals (torture) to TTC.• Advantages:
– Government more supportive (supports wider need)– Clients like it better (more anonymity, less singling them out)– Reach and access enhanced +++++ through integration– Priorities better match client priorities and more accessible– Torture survivors who need it still get specialist care– TTC can focus on those who really need them.
Apprenticeship Model of Training and Supervision
• Key to expanding quality treatment access in low resource countries
• Based on research on training:– One-off trainings are ineffective for behavior change.
– “Train and hope” approach to implementation does not work (e.g., Kelly et al., 2000)
– Ongoing supervision with on-the-job training is critical
SUPERVISION MODEL Monitoring quality Continuous Training
Continuous intervention development
Trainers
Supervisors
Counselors
Clients
Purpose of apprenticeship training
• Provides real skills development of provider and supervisor by learning while doing
• Allows non professionals to really learn to provide treatment while assuring survivor gets quality treatment.
• Iterative correction/improvement of survivor and provider and supervisor problems
• Cares for providers – monitors and prevents/treats burnout.
Impact Assessment
Standard evaluation vs Experimental evaluation
• Standard evaluation– Post intervention measure only, OR– Pre and post intervention measure– Assesses whether problem improved– Does not assess why.
• Research– Pre and post evaluation(s)– Compare with controls– Assesses whether problem improved and whether
this was because of the intervention.
Uganda
• Intervention study:
• Pre-intervention - 110/117 (94%) depressed• Post-intervention - 64/117 (54.7%)
depressed
(p<0.001)
Uganda
• Control arm:• Pre-intervention - 110/117 (94%) depressed• Post-intervention - 64/117 (54.7%) depressed
• Intervention arm:• Pre-intervention - 92/107 (86%) depressed• Post-intervention - 7/107 (6.5%) depressed
(p<0.001)
Pre Post0
1
2
3
4
5
6
7
controlintervention
Pre Post0
1
2
3
4
5
6
7
controlintervention
Pre Post0
1
2
3
4
5
6
7
controlintervention
Pre Post0
1
2
3
4
5
6
7
controlintervention
Reduction of depression symptoms by group
25
30
35
40
45
50
Baseline Follow-Up
Dep
ress
ion
Sym
ptom
Sc
ores
Reduction of depression symptoms by group
25
30
35
40
45
50
Baseline Follow-Up
ControlIPT-GCP
Dep
ress
ion
Sym
ptom
Sc
ores
Which one?
• Use Standard Evaluation when other factors stable.
• Use experimental evaluation when likely effect of intervention is not known and other factors not stable.
– Control group is essential to answering the question: ‘was change due to the intervention?’
RCT as program evaluation
RCTs perceived as:• Unnecessary• Unethical – making people wait.• Too complex/difficult• Too expensive - $,time,effort.• Waste/diversion of resources from ‘real’ aid• Not appropriate for low resource/difficult
environments
500
250
250
250
250
Assess
Randomize
Reassess and compare
Service capacity=500
1,000
500
500
500
500
Assess
Randomize
Reassess and compare
Service capacity=500
What makes RCTs costly/complex?
• Design/training/implementation of intervention• Monitoring quality of intervention• Supervision of workers• Monitoring acceptability/feasibility/compliance• Assessing who has priority for services• Reassessing persons after treatment.
True costs beyond normal M&E
• Randomization (cheap)• Analysis (cheap)• Assessing additional cases (relatively cheap –
economies of scale).
• Small compared with savings from stopping ineffective programs– Saves money/time/effort of funder and population– More ethical
When is an RCT Appropriate?
• Existing evidence base poor
• Programs vulnerable to cross-cultural variation
• Situation unstable
• Therefore, uncertainty about local impact of interventions.
• EFFECT SIZE is a measure of the difference between intervention and control in a way that is comparable across studies
50
Effect Size Percent (%) of Controls with Scores below the Average of Intervention Clients
0.2 58%0.5 69%0.8 79%1.4 92%2.0 98%2.5 99%
Interpretation of Effect Size
• Interpretation:0.0 = no effect0.2 = small effect0.5 = moderate effect0.8 = large effect
51
Comparison of Effect Sizes (Iraq)
52
Barriers to Treatment Access
• Most trauma affected persons in low resource countries have multiple problems.
• Many treatments but each focused on 1-2 problems
• Therefore, ?need to provide multiple treatments.