evaluating health policy through natural experiments andrew b. bindman, md professor medicine,...
TRANSCRIPT
Evaluating Health Policy Evaluating Health Policy through Natural Experimentsthrough Natural Experiments
Andrew B. Bindman, MDAndrew B. Bindman, MD
Professor Medicine, Professor Medicine, Health Policy, Health Policy, Epidemiology & Epidemiology & BiostatisticsBiostatistics
UCSFUCSF
Insurance Reforms and Insurance Reforms and Other Imminent Policy Other Imminent Policy ChangesChanges Many of you identified specific Many of you identified specific policies that are or are about to policies that are or are about to be implemented that are relevant be implemented that are relevant to your research intereststo your research interests
Opportunity to study this change Opportunity to study this change as a way to inform the policy as a way to inform the policy processprocess
Learning About PoliciesLearning About Policies
Read the newspaperRead the newspaper– NY Times, Washington Post, PoliticoNY Times, Washington Post, Politico
Health newswire servicesHealth newswire services– California Healthline (California Healthline (www.chcf.org))– Kaiser Health NewsKaiser Health News
(www.kaiserhealthnews.org)(www.kaiserhealthnews.org) Academic faculty Academic faculty Professional Societies/Scientific Professional Societies/Scientific OrganizationsOrganizations
Community-based organizationsCommunity-based organizations Directly from policy decision-makersDirectly from policy decision-makers
My Research InterestMy Research Interest
Health consequences of Health consequences of public policiespublic policies
Access to and quality of Access to and quality of care for low-income, care for low-income, diverse, and patient diverse, and patient populations vulnerable to populations vulnerable to poor health because of their poor health because of their social circumstancessocial circumstances
Medi-Cal: Medi-Cal: CaliforniaCalifornia’’s Medicaid s Medicaid ProgramProgram ~8 million beneficiaries~8 million beneficiaries $41 billion last year$41 billion last year 22ndnd largest use of general fund (17%) largest use of general fund (17%) Pays for 1 in every 2 births in the Pays for 1 in every 2 births in the statestate
Approximately half of beneficiaries Approximately half of beneficiaries are Latinoare Latino
Provides 2/3rds of safety net Provides 2/3rds of safety net fundingfunding
Medicaid Population: Medicaid Population: Clinical QuestionClinical Question
Many Medicaid beneficiaries have Many Medicaid beneficiaries have interruptions in coverage interruptions in coverage (churning)(churning)
Many uninsured gain Medicaid Many uninsured gain Medicaid coverage when hospitalizedcoverage when hospitalized
Does Medicaid coverage provided at Does Medicaid coverage provided at the time of a hospitalization the time of a hospitalization adequate or do interruptions in adequate or do interruptions in Medicaid enrollment have a negative Medicaid enrollment have a negative impact on the health of impact on the health of beneficiaries?beneficiaries?
Designing a Research StudyDesigning a Research Study
Randomized trialRandomized trial– Feasibility?Feasibility?
Designing a Research StudyDesigning a Research Study
Randomized trialRandomized trial– Unethical and impracticalUnethical and impractical
Observational studyObservational study– Compare the experiences of Compare the experiences of beneficiaries who have interruptions beneficiaries who have interruptions in coverage with those who have in coverage with those who have continuous coveragecontinuous coverage
Reverse CausalityReverse Causality
Interruption in coverage might Interruption in coverage might not predict worse health not predict worse health outcome so much as worse health outcome so much as worse health might predict whether or not might predict whether or not have interrupted coveragehave interrupted coverage
Bias of higher admissions among Bias of higher admissions among those with continuous coveragethose with continuous coverage
Designing a Research StudyDesigning a Research Study
Randomized trial- is it Randomized trial- is it feasible?feasible?
Observed variation - is it Observed variation - is it biased?biased?
Natural experiment - does a Natural experiment - does a good one exist?good one exist?
Natural ExperimentsNatural Experiments
A A naturalnatural or q or quasi-experimentuasi-experiment is a is a naturally occurring instance of naturally occurring instance of observable phenomena which observable phenomena which approximate or duplicate the approximate or duplicate the properties of a controlled properties of a controlled experiment. In contrast to . In contrast to laboratory experiments, these events , these events aren't created by scientists, but aren't created by scientists, but yield data which nonetheless can be yield data which nonetheless can be used to make causal inferences. used to make causal inferences.
What Are the Elements of a What Are the Elements of a Good Natural Experiment in Good Natural Experiment in Health PolicyHealth Policy
Policy implementation not Policy implementation not biased by patient biased by patient characteristics such as health characteristics such as health statusstatus
Policy can be effectively tied Policy can be effectively tied to a to a ““treatmenttreatment”” exposed group exposed group
Access to before/after dataAccess to before/after data
Medicaid Population: Medicaid Population: Policy QuestionPolicy Question
Federal law requires re-determination of Federal law requires re-determination of eligibility for beneficiaries at a eligibility for beneficiaries at a minimum of every 12 months but states minimum of every 12 months but states have option to do more frequentlyhave option to do more frequently
Beneficiaries who do not Beneficiaries who do not ““re-sign upre-sign up”” are are dropped from programdropped from program
Does frequency of a stateDoes frequency of a state’’s re-enrollment s re-enrollment process increase the number of process increase the number of beneficiaries with interruptions in beneficiaries with interruptions in coverage and if so is this in turn coverage and if so is this in turn associated with patientsassociated with patients’’ health? health?
Natural Experiment of Natural Experiment of Interrupted Medicaid Interrupted Medicaid CoverageCoverage
California extended Medicaid re-California extended Medicaid re-enrollment period for enrollment period for allall children in children in California from every 3 to every 12 California from every 3 to every 12 months on January 1, 2001months on January 1, 2001
Extension of eligibility re-Extension of eligibility re-determination period should be determination period should be associated with an increase in associated with an increase in continuity of Medicaid coverage, but continuity of Medicaid coverage, but should not except through its influence should not except through its influence on continuity of coverage be associated on continuity of coverage be associated with the health status of children. with the health status of children.
Challenging Issues in Challenging Issues in Studying Natural Studying Natural ExperimentsExperiments Learning about a policy Learning about a policy change as it is about to change as it is about to happen or after the fact happen or after the fact makes it harder to collect makes it harder to collect baseline databaseline data
Primary Data CollectionPrimary Data Collection
Can be challenging to organize in Can be challenging to organize in time to assess pre-policy time to assess pre-policy condition condition
Lots of work but lots of control Lots of work but lots of control over data collection (eg surveys, over data collection (eg surveys, physiological measures, etc)physiological measures, etc)– Time consumingTime consuming– ExpensiveExpensive
Difficult to maintain over timeDifficult to maintain over time
Secondary DatabasesSecondary Databases
Pre-existing data that are often Pre-existing data that are often collected for an alternative collected for an alternative purposepurpose
Individual level or sometimes Individual level or sometimes aggregate dataaggregate data
Examples:Examples:– National surveys National surveys – RegistriesRegistries– Study cohortsStudy cohorts– Administrative dataAdministrative data
Secondary Data:Secondary Data:Advantages/ChallengesAdvantages/Challenges
Efficient - cheap, fast and often very Efficient - cheap, fast and often very largelarge
Little control on what was collectedLittle control on what was collected Collection is often Collection is often longitudinal/repeated cross-sectional longitudinal/repeated cross-sectional
Potential to analyze temporal changesPotential to analyze temporal changes If source is a payer or a provider may If source is a payer or a provider may have incomplete capturehave incomplete capture
Can be scooped by others with access Can be scooped by others with access to same datato same data
Finding Secondary DataFinding Secondary Data
www.ctsi.ucsf.edu/research/celdac
www.phpartners.org/health_stats.html
Medicaid Data for Studying Medicaid Data for Studying Interruptions in CoverageInterruptions in Coverage
Comprehensive and detailed Comprehensive and detailed regarding eligibilityregarding eligibility
Fee for service claims complete Fee for service claims complete Missing claims information for Missing claims information for beneficiaries in managed carebeneficiaries in managed care
WonWon’’t reflect experience of t reflect experience of beneficiaries when they arenbeneficiaries when they aren ’’t t coveredcovered
Statewide Hospital Patient Statewide Hospital Patient Discharge AbstractsDischarge Abstracts
Comprehensive capture of all Comprehensive capture of all hospitalizations in state hospitalizations in state regardless of payerregardless of payer
Includes information on hospital Includes information on hospital admission diagnosesadmission diagnoses
Provides payer source at time of Provides payer source at time of hospitalizationhospitalization
Ambulatory Care Sensitive Conditions:Ambulatory Care Sensitive Conditions:AHRQ Prevention Quality IndicatorsAHRQ Prevention Quality Indicators
1.1. Condition with acute course and window for Condition with acute course and window for interventionintervention
2.2. Condition with chronic course amenable to self-Condition with chronic course amenable to self-managementmanagement
ACS Conditions
Acute Conditions:Acute Conditions:– DehydrationDehydration– Ruptured Ruptured Appendicitis Appendicitis
– CellulitisCellulitis– Bacterial Bacterial PneumoniaPneumonia
– Urinary Tract Urinary Tract InfectionInfection
Chronic Chronic Conditions:Conditions:– AsthmaAsthma– Hypertension Hypertension – COPDCOPD– Diabetes Diabetes MellitusMellitus
– Heart FailureHeart Failure– AnginaAngina
Statewide Hospital Patient Statewide Hospital Patient Discharge AbstractsDischarge Abstracts
Provides payer source at time of Provides payer source at time of hospitalization but not over timehospitalization but not over time
Critical question for hospitalizations Critical question for hospitalizations for ambulatory care sensitive for ambulatory care sensitive admissions is what the insurance admissions is what the insurance status was prior to the admission status was prior to the admission since many uninsured gain coverage in since many uninsured gain coverage in association with the hospitalizationassociation with the hospitalization
Linked CA Hospital Linked CA Hospital Discharge and Medicaid Discharge and Medicaid Eligibility FilesEligibility Files
OSHPD: Hospital Discharge Data
1998 2003
DHS: Medi-Cal Enrollment Database
1998 2003• Demographics• Monthly enrollment history• Aid Category (e.g. TANF or SSI)• FFS, managed care • Other insurance
• Diagnosis (ICD-9 Code)• Month/Year of admission
Linkage
Pre/Post Study of Re-Pre/Post Study of Re-Enrollment Policy Change for Enrollment Policy Change for ChildrenChildren
Children 1-17 years with a Children 1-17 years with a minimum of 1 month of Medicaid minimum of 1 month of Medicaid coverage in California coverage in California
Outcome = time to a hospital Outcome = time to a hospital admission for an ambulatory care admission for an ambulatory care sensitive condition sensitive condition
Main predictor = time periodMain predictor = time period– Pre policy change = Jan Pre policy change = Jan ‘‘99-December 99-December
‘‘0000– Post policy change = Jan Post policy change = Jan ‘‘01-December 01-December
‘‘0202
Children 1-17 Years in California Medicaid Children 1-17 Years in California Medicaid Before and After Policy to Change EnrollmentBefore and After Policy to Change Enrollment
1999-2000 1999-2000 2001-20022001-2002
NN 3,288,171 3,288,171
3,230,120 3,230,120
Mean Age (yrs)Mean Age (yrs) 99 99
% Female% Female 5050 5151
Ethnicity (%)Ethnicity (%)
HispanicHispanic 5454 5656
Black Black 1313 1212
AsianAsian 88 88
OtherOther 2525 2424
Aid Group (%)Aid Group (%)
TANFTANF 4747 5050
SSISSI 33 33
OtherOther 5050 4747
Managed Care (%)Managed Care (%) 4747 4141
Children with Continuous Children with Continuous Medicaid Enrollment by Time Medicaid Enrollment by Time PeriodPeriod
0%
10%
20%
30%
40%
50%
60%
70%
Pre: 1999-2000 Post: 2001-2002
49
62
Years of Enrollment
Percenta
ge
Probability of a Probability of a Hospitalization for an ACS Hospitalization for an ACS Condition Over TimeCondition Over Time
0.00
0.05
0.10
0.15
0.20
0.25
0.300.35
0.40
0 3 6 9 12 15 18 21 24
Before 2001Enrollment Extension After 2001 EnrollmentExtension
Months
Children: Adjusted Risk of ACS Children: Adjusted Risk of ACS HospitalizationHospitalization
Relative HazardRelative Hazard P-ValueP-Value
Post policy Post policy 0.740.74
<.0001<.0001
AgeAge 0.880.88
<.0001<.0001
FemaleFemale 0.970.97
0.01750.0175
EthnicityEthnicity
HispanicHispanic
3.263.26
<.0001<.0001
Black Black
4.704.70
<.0001<.0001
AsianAsian
1.101.10
0.09260.0926
OtherOther
2.972.97
<.0001<.0001
Aid GroupAid Group
TANFTANF
1.471.47
<.0001<.0001
SSISSI
24.9024.90
<.0001<.0001
Managed Managed CareCare
0.820.82
<.0001<.0001
Quasi- (natural) Quasi- (natural) ExperimentsExperiments
"Estimating the internal validity of a "Estimating the internal validity of a relationship is a deductive process in relationship is a deductive process in which the investigator has to which the investigator has to systematically think through how each of systematically think through how each of the internal validity threats may have the internal validity threats may have influenced the data. Then the investigator influenced the data. Then the investigator has to examine the data to test which has to examine the data to test which relevant threats can be ruled out. . . . relevant threats can be ruled out. . . . When all of the threats can plausibly be When all of the threats can plausibly be eliminated it is possible to make confident eliminated it is possible to make confident conclusions about whether a relationship is conclusions about whether a relationship is probably causal." probably causal."
Cook and Cook and CampbellCampbell
LimitationsLimitations
Could secular changes other Could secular changes other than the policy change than the policy change explain the observed explain the observed differences? differences?
Strengthening the Design Strengthening the Design of a Natural Experimentof a Natural Experiment
Pre/post changes ideally with a Pre/post changes ideally with a comparison group not exposed to policy comparison group not exposed to policy
““Difference in differencesDifference in differences””
Need to establish conceptual basis for Need to establish conceptual basis for selection of specific comparison group selection of specific comparison group
Potential Comparison Potential Comparison Group: Adults in Medi-CalGroup: Adults in Medi-Cal
Medicaid eligibility re-Medicaid eligibility re-determination period did not determination period did not change during study period for change during study period for adultsadults
Therefore, would not expect a Therefore, would not expect a decrease over time in decrease over time in hospitalizations for ambulatory hospitalizations for ambulatory care sensitive conditions among care sensitive conditions among adultsadults
Comparison Group: Comparison Group: Adults in MedicaidAdults in Medicaid
Adults with Medicaid coverageAdults with Medicaid coverage– 1999-2000 = 62%1999-2000 = 62%– 2001-2002 = 60%2001-2002 = 60%
Adjusted relative hazard of a Adjusted relative hazard of a hospitalization for an ACS hospitalization for an ACS condition for adults in post vs condition for adults in post vs pre period= 1.11pre period= 1.11
Second Comparison Group: Second Comparison Group: Children with Continuous Children with Continuous CoverageCoverage
Comparison of children with Comparison of children with continuous coverage in each time continuous coverage in each time period revealed no significant period revealed no significant difference in hospitalizations for difference in hospitalizations for ambulatory care sensitive ambulatory care sensitive conditionsconditions
Suggests no difference in treatment Suggests no difference in treatment approach to ambulatory care approach to ambulatory care sensitive conditions over time sensitive conditions over time period of studyperiod of study
How Do We Know This is How Do We Know This is About Access to Ambulatory About Access to Ambulatory Care?Care?
Hospitalization rates among Hospitalization rates among children for non ambulatory children for non ambulatory care sensitive conditions care sensitive conditions (appendicitis and (appendicitis and gastrointestinal gastrointestinal obstruction) did not change obstruction) did not change over timeover time
Most with Interruption in Most with Interruption in Medicaid Coverage Do Not Medicaid Coverage Do Not Have Alternative for Have Alternative for Ambulatory CareAmbulatory Care At the time of At the time of hospitalizationhospitalization–59% regain Medi-Cal with 59% regain Medi-Cal with admissionadmission
–7% remain uninsured 7% remain uninsured –33% had another form of 33% had another form of insuranceinsurance
Policy ImplicationsPolicy Implications
States need to become more aware States need to become more aware of the hidden costs in their of the hidden costs in their Medicaid policiesMedicaid policies
Continuity of Medicaid coverage Continuity of Medicaid coverage can support better health and can support better health and decrease wasteful spending on decrease wasteful spending on hospitalizations that could have hospitalizations that could have been avoided with less costly been avoided with less costly outpatient careoutpatient care
Translating Research into Translating Research into PolicyPolicy
Results of study used Results of study used – in testimony to California in testimony to California legislature to prevent more frequent legislature to prevent more frequent eligibility re-determination as part eligibility re-determination as part of budget cut processof budget cut process
– in Congress to support Maintenance of in Congress to support Maintenance of Effort requirements as a part of CHIP Effort requirements as a part of CHIP reauthorizationreauthorization
Published in scientific journals for Published in scientific journals for other states to consider in their other states to consider in their policy makingpolicy making
Are There Opportunities Are There Opportunities for Randomized Evaluations for Randomized Evaluations of Health Policies?of Health Policies?
Randomized designs are least Randomized designs are least susceptible to biassusceptible to bias
Political considerations often Political considerations often make this approach impractical make this approach impractical in health policy interventionsin health policy interventions
May be opportunities to use a May be opportunities to use a lottery in implementing lottery in implementing policies that have more demand policies that have more demand than supply (a wait list)than supply (a wait list)
Oregon Health Study: Oregon Health Study: Randomized ImplementationRandomized Implementation
Opportunity for those 19-64 yrs Opportunity for those 19-64 yrs <100% FPL otherwise ineligible <100% FPL otherwise ineligible to obtain Medicaidto obtain Medicaid
85,000 applied but only 85,000 applied but only available for 30,000available for 30,000
Lottery used to randomly select Lottery used to randomly select who gained Medicaid coveragewho gained Medicaid coverage
Oregon Health Study: Oregon Health Study: Analytic PlanAnalytic Plan
Comparisons made in utilization and Comparisons made in utilization and outcomes between those offered outcomes between those offered Medicaid coverage through lottery and Medicaid coverage through lottery and those who were notthose who were not
Intention to treat analysis- some Intention to treat analysis- some offered did not accept offered did not accept – ~10,000 of 30,000 selected enrolled~10,000 of 30,000 selected enrolled
Some not offered may have gained Some not offered may have gained coverage through other meanscoverage through other means
Oregon Health Study: Oregon Health Study: Results Results
Those offered Medicaid were Those offered Medicaid were – 70% more likely to have a regular 70% more likely to have a regular source of caresource of care
– 60% more likely to have a mammogram60% more likely to have a mammogram– 20% more likely to have cholesterol 20% more likely to have cholesterol screeningscreening
Also improvements in self reported Also improvements in self reported health statushealth status
HomeworkHomework
Identify or design a Identify or design a plausible natural experiment plausible natural experiment to evaluate a policy relevant to evaluate a policy relevant to your area of researchto your area of research
Describe data you could use Describe data you could use to study it and possible to study it and possible comparison group(s)comparison group(s)
Not All Policy Changes Not All Policy Changes Make Good Natural Make Good Natural Experiments Experiments
Voluntary Medicare managed careVoluntary Medicare managed care–voluntary implementation can voluntary implementation can have health selection biashave health selection bias
Expansion of public insurance Expansion of public insurance coveragecoverage–uptake by uninsured and uptake by uninsured and ““crowd crowd outout”” of privately insured can of privately insured can make it hard to isolate who make it hard to isolate who got got ““treatmenttreatment”” of insurance of insurance