an assessment of the impact of two distinct survey design modifications on health insurance coverage...

An Assessment of the Impact of Two Distinct An Assessment of the Impact of Two Distinct Survey Design Modifications on Health Survey Design Modifications on Health

Insurance Coverage Estimates in a National Insurance Coverage Estimates in a National

Health Care SurveyHealth Care Survey

Steven B. Cohen, Trena Ezzati-Rice, and Marc ZodetSteven B. Cohen, Trena Ezzati-Rice, and Marc Zodet

International Total Survey Error Workshop 2010 Stowe, VT

June 13-16, 2010

Background and motivation for Background and motivation for research research

The Medical Expenditure Panel Survey The Medical Expenditure Panel Survey (MEPS) is a national resource to inform health (MEPS) is a national resource to inform health care policy care policy

MEPS is a key survey resource to monitor:MEPS is a key survey resource to monitor:– Trends in estimates of the uninsuredTrends in estimates of the uninsured– Population characteristics of the uninsured Population characteristics of the uninsured – Duration of spells of uninsurance and long term Duration of spells of uninsurance and long term

uninsureduninsured– Financial consequences of being uninsuredFinancial consequences of being uninsured– Relationship between uninsurance and health Relationship between uninsurance and health

statusstatus

Thus need for timely, high quality MEPS dataThus need for timely, high quality MEPS data

MEPS survey backgroundMEPS survey background

Annual survey since 1996; nationally representative sample of Annual survey since 1996; nationally representative sample of householdshouseholds

5 rounds of data collection covering 2 calendar years5 rounds of data collection covering 2 calendar years

Used to estimate medical care utilization, access to care, and Used to estimate medical care utilization, access to care, and health care expenses for the U.S. civilian noninstitutionalized health care expenses for the U.S. civilian noninstitutionalized populationpopulation

Integrated survey designIntegrated survey design

– Each annual sample is a subsample of responding households (from Each annual sample is a subsample of responding households (from prior year) from another large ongoing U.S. health survey, the National prior year) from another large ongoing U.S. health survey, the National Health Interview Survey (NHIS)Health Interview Survey (NHIS)

Overlapping panel designOverlapping panel design

– Data from 1st year of new panel combined with data from 2nd year of Data from 1st year of new panel combined with data from 2nd year of previous panelprevious panel

Illustration of how panels/rounds Illustration of how panels/rounds comprise MEPS calendar year datacomprise MEPS calendar year data

MEPS survey design modifications in MEPS survey design modifications in 2007: Panel 122007: Panel 12

1.1. Re-engineered CAPI survey instrument: Windows-based platform Re-engineered CAPI survey instrument: Windows-based platform replaced the DOS-based system (questionnaire remained virtually replaced the DOS-based system (questionnaire remained virtually unchanged)unchanged)

2.2. New sample design resulting from the sample redesign of the NHIS New sample design resulting from the sample redesign of the NHIS in 2006 in 2006 – MEPS Panel 12 fielded in January 2007 was the 1st Panel to be MEPS Panel 12 fielded in January 2007 was the 1st Panel to be

selected based on the new NHIS sampleselected based on the new NHIS sample– New NHIS sample design conceptually very similar to the 1995-New NHIS sample design conceptually very similar to the 1995-

2005 design: complex area probability sample 2005 design: complex area probability sample

– Changes in 2006 NHIS design that affected 2007 MEPSChanges in 2006 NHIS design that affected 2007 MEPS Sampling PSUs and SSUs independent of those sampled under the Sampling PSUs and SSUs independent of those sampled under the

previous designprevious design

Previously only HHs with Black and Hispanic persons were Previously only HHs with Black and Hispanic persons were oversampled - oversampled - New New – oversampling of Asian persons – oversampling of Asian persons

Goals of this researchGoals of this research

Evaluate if any significant impact Evaluate if any significant impact on MEPS on MEPS estimates of insurance coverage as a result of estimates of insurance coverage as a result of the dual survey changes implemented in Panel the dual survey changes implemented in Panel 12 which began January 200712 which began January 2007

Evaluate the effectiveness of the MEPS Evaluate the effectiveness of the MEPS nonresponse adjustment strategies in reducing nonresponse adjustment strategies in reducing potential biaspotential bias

Two evaluation approachesTwo evaluation approaches

1. Internal survey comparisons1. Internal survey comparisons– takes advantage of MEPS overlapping panel designtakes advantage of MEPS overlapping panel design

compare the two individual panel specific estimates of health compare the two individual panel specific estimates of health insurance coverage for 2007 and other yearsinsurance coverage for 2007 and other years

– conduct multivariate analysis to determine if panel is a conduct multivariate analysis to determine if panel is a significant predictor of insurance status after controlling significant predictor of insurance status after controlling for other independent variablesfor other independent variables

2. External comparisons2. External comparisons– takes advantage of MEPS linkage to NHIS surveytakes advantage of MEPS linkage to NHIS survey– after MEPS household nonresponse and survey after MEPS household nonresponse and survey

attrition, can insurance estimates for the sampling frame attrition, can insurance estimates for the sampling frame source (NHIS) be replicated using just the MEPS survey source (NHIS) be replicated using just the MEPS survey respondents and the MEPS nonresponse adjusted respondents and the MEPS nonresponse adjusted weights?weights?

Sources of dataSources of data

2007 MEPS compared to 2004-20062007 MEPS compared to 2004-2006– Panel specific estimates within each calendar year Panel specific estimates within each calendar year

(using panel specific weights)(using panel specific weights)

2003-2006 NHIS data2003-2006 NHIS data

Evaluation Approach 1:Evaluation Approach 1:Internal MEPS ComparisonInternal MEPS Comparison

Step 1: Panel specific estimates within yearsStep 1: Panel specific estimates within years

Step 2: Multivariate analysisStep 2: Multivariate analysis

Step 1: Step 1: Evaluation of concordance of Evaluation of concordance of panel specific health insurance coverage panel specific health insurance coverage estimatesestimates

Focus is on 2007 coverage estimatesFocus is on 2007 coverage estimates Panel 12 – dual survey changesPanel 12 – dual survey changes Panel 11 – original sample design and Panel 11 – original sample design and

original DOS-based survey instrumentoriginal DOS-based survey instrument

MEPS annual estimation weightsMEPS annual estimation weights

Each panel is weighted separately and then Each panel is weighted separately and then combinedcombined

Panel specific weight is a function of:Panel specific weight is a function of:– NHIS base weightNHIS base weight

– Poststratification to NHIS full samplePoststratification to NHIS full sample

– Nonresponse adjustment for dwelling unit Nonresponse adjustment for dwelling unit nonresponse and survey attrition within year 1 nonresponse and survey attrition within year 1 and year 2and year 2

– Final raking adjustment to CPS population Final raking adjustment to CPS population control totalscontrol totals

Evaluating the joint effect of the Evaluating the joint effect of the dual survey design changes in 2007dual survey design changes in 2007

Comparison of panel specific national health Comparison of panel specific national health insurance coverage estimates for the insurance coverage estimates for the population under age 65population under age 65

Measures evaluated: Measures evaluated: – 1) some private coverage during the year 1) some private coverage during the year

– 2) public-only coverage during the year 2) public-only coverage during the year

– 3) full year uninsured 3) full year uninsured

Estimates of health insurance coverage for the Estimates of health insurance coverage for the civilian non-institutionalized populationcivilian non-institutionalized population<65 years of age by panel within year<65 years of age by panel within year

CalendarCalendar CoverageCoverage Year 2 in PanelYear 2 in Panel Year 1 in PanelYear 1 in Panel

YearYear MeasureMeasure %% (SE)(SE) %% (SE)(SE)

20072007 Any privateAny private 69.269.2 (0.83)(0.83) 68.768.7 (1.04)(1.04)

Public onlyPublic only 15.815.8 (0.61)(0.61) 15.815.8 (0.73)(0.73)

UninsuredUninsured 15.015.0 (0.53)(0.53) 15.515.5 (0.68)(0.68)










Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07

Step 2: Multivariate analysisStep 2: Multivariate analysis

Logistic regression analysis to test for a Logistic regression analysis to test for a panel effect controlling for other panel effect controlling for other explanatory variables related to health explanatory variables related to health insurance coverageinsurance coverage

Separate models by year: 2004-2007Separate models by year: 2004-2007

Covariates included in the modelsCovariates included in the models

Panel classificationPanel classification SexSex Race/ethnicityRace/ethnicity Self-reported health statusSelf-reported health status Marital statusMarital status EducationEducation IncomeIncome MSA, Census regionMSA, Census region Total healthcare expensesTotal healthcare expenses

20072007 20062006 20052005 20042004

Degrees of Degrees of FreedomFreedom

Wald FWald F P-valueP-value Wald FWald F P-valueP-value Wald FWald F P-valueP-value Wald FWald F P-valueP-value

Overall ModelOverall Model 2626 94.8494.84 <0.001<0.001 120.92120.92 <0.001<0.001 101.55101.55 <0.001<0.001 84.2184.21 <0.001<0.001

Model minus interceptModel minus intercept 2525 42.6642.66 <0.001<0.001 66.9266.92 <0.001<0.001 50.9150.91 <0.001<0.001 50.2850.28 <0.001<0.001

Panel classificationPanel classification 11 1.791.79 0.18140.1814 0.060.06 0.81350.8135 0.340.34 0.56270.5627 0.030.03 0.86440.8644

SexSex 11 71.5771.57 <0.001<0.001 66.0966.09 <0.001<0.001 75.2475.24 <0.001<0.001 69.6869.68 <0.001<0.001

Race/ethnicityRace/ethnicity 33 53.8353.83 <0.001<0.001 42.4242.42 <0.001<0.001 56.0356.03 <0.001<0.001 47.5747.57 <0.001<0.001

Health statusHealth status 44 2.152.15 0.07370.0737 2.722.72 0.03020.0302 1.181.18 0.31940.3194 4.904.90 0.00080.0008

Marital statusMarital status 44 14.8414.84 <0.001<0.001 13.0213.02 <0.001<0.001 18.2118.21 <0.001<0.001 26.2626.26 <0.001<0.001

Highest year of educationHighest year of education 44 12.3112.31 <0.001<0.001 32.7132.71 <0.001<0.001 15.3715.37 <0.001<0.001 12.6912.69 <0.001<0.001

Poverty statusPoverty status 44 69.9669.96 <0.001<0.001 103.81103.81 <0.001<0.001 71.1471.14 <0.001<0.001 79.5379.53 <0.001<0.001

RegionRegion 33 21.0021.00 <0.001<0.001 9.019.01 <0.001<0.001 11.9611.96 <0.001<0.001 17.5817.58 <0.001<0.001

Total health care Total health care expendituresexpenditures

11 34.8834.88 <0.001<0.001 68.4468.44 <0.001<0.001 32.4732.47 <0.001<0.001 35.6835.68 <0.001<0.001

-2 x Normalized Log--2 x Normalized Log-Likelihood Full Model:Likelihood Full Model:

13,422.913,422.9 14,319.714,319.7 14,223.514,223.5 14,052.514,052.5

Pseudo Model R-Square:Pseudo Model R-Square: 0.15500.1550 0.15880.1588 0.15160.1516 0.15690.1569


Note: In logistic regression analysis, Y=1 specifies full year uninsured, Y=0 specifies otherNote: In logistic regression analysis, Y=1 specifies full year uninsured, Y=0 specifies other

Final logistic regression model of the Final logistic regression model of the uninsureduninsured, , ages 18-64 years; testing for panel effectages 18-64 years; testing for panel effect

20072007 20062006 20052005 20042004

Degrees of Degrees of FreedomFreedom

Wald FWald F P-valueP-value Wald FWald F P-valueP-value Wald FWald F P-valueP-value Wald FWald F P-valueP-value

Overall ModelOverall Model 2626 105.10105.10 <0.001<0.001 140.17140.17 <0.001<0.001 108.79108.79 <0.001<0.001 100.36100.36 <0.001<0.001

Model minus interceptModel minus intercept 2525 89.8389.83 <0.001<0.001 130.70130.70 <0.001<0.001 105.91105.91 <0.001<0.001 92.4892.48 <0.001<0.001

Panel classificationPanel classification 11 2.642.64 0.10480.1048 2.392.39 0.12340.1234 0.310.31 0.57630.5763 0.510.51 0.47490.4749

SexSex 11 17.3717.37 <0.001<0.001 10.7310.73 0.00120.0012 13.4213.42 0.00030.0003 28.8128.81 <0.001<0.001

Race/ethnicityRace/ethnicity 33 41.3441.34 <0.001<0.001 34.1434.14 <0.001<0.001 49.2249.22 <0.001<0.001 45.7345.73 <0.001<0.001

Health statusHealth status 44 20.3620.36 <0.001<0.001 31.8431.84 <0.001<0.001 32.9632.96 <0.001<0.001 21.9121.91 <0.001<0.001

Marital statusMarital status 44 34.0834.08 <0.001<0.001 28.2628.26 <0.001<0.001 33.9533.95 <0.001<0.001 37.6537.65 <0.001<0.001

Highest year of educationHighest year of education 44 41.5641.56 <0.001<0.001 76.0676.06 <0.001<0.001 64.0764.07 <0.001<0.001 58.3258.32 <0.001<0.001

Poverty statusPoverty status 44 245.13245.13 <0.001<0.001 308.26308.26 <0.001<0.001 261.46261.46 <0.001<0.001 282.70282.70 <0.001<0.001

RegionRegion 33 3.703.70 0.01190.0119 3.993.99 0.00850.0085 0.490.49 0.69060.6906 2.092.09 0.10270.1027

Total health care Total health care expendituresexpenditures

11 19.9419.94 <0.001<0.001 57.9457.94 <0.001<0.001 30.6330.63 <0.001<0.001 34.5134.51 <0.001<0.001

-2 x Normalized Log--2 x Normalized Log-Likelihood Full Model:Likelihood Full Model:

14555.514555.5 15,608.215,608.2 15,268.015,268.0 15,354.915,354.9

Pseudo Model R-Square:Pseudo Model R-Square: 0.28370.2837 0.28540.2854 0.29180.2918 0.28830.2883


Note: In logistic regression analysis, Y=1 specifies some private insurance in year, Y=0 specifies otherNote: In logistic regression analysis, Y=1 specifies some private insurance in year, Y=0 specifies other

Final logistic regression model of the Final logistic regression model of the privatelyprivately insuredinsured, ages 18-64 years; testing for panel effect, ages 18-64 years; testing for panel effect

Evaluation Approach 2: External Evaluation Approach 2: External EvaluationEvaluation

Takes advantage of the linkage of the MEPS to the NHISTakes advantage of the linkage of the MEPS to the NHIS

National estimates of health insurance are derived from National estimates of health insurance are derived from NHIS (NHIS insurance variables) for two analytical samples:NHIS (NHIS insurance variables) for two analytical samples:– Total NHIS sample Total NHIS sample →→ NHIS weights NHIS weights– Reduced NHIS sample obtained from matching with Reduced NHIS sample obtained from matching with

MEPS full year respondents MEPS full year respondents →→ MEPS weights as MEPS weights as adjusted for MEPS nonresponseadjusted for MEPS nonresponse

Since this analysis examines only NHIS insurance data, the Since this analysis examines only NHIS insurance data, the estimates are not impacted by the MEPS CAPI redesignestimates are not impacted by the MEPS CAPI redesign– Thus, assessment of joint impact of the MEPS sample Thus, assessment of joint impact of the MEPS sample

redesign and effectiveness of MEPS nonresponse redesign and effectiveness of MEPS nonresponse adjustments.adjustments.

MEPS-NHIS linked analysisMEPS-NHIS linked analysis

The following 4 NHIS measures of health insurance The following 4 NHIS measures of health insurance coverage were examined:coverage were examined:

any coverageany coverage at the time of the interview (covered; not at the time of the interview (covered; not covered; refused/not ascertained/DK)covered; refused/not ascertained/DK)

private coverageprivate coverage at the time of the interview at the time of the interview (mentioned; not mentioned; refused/not (mentioned; not mentioned; refused/not ascertained/DK)ascertained/DK)

Medicaid coverageMedicaid coverage at the time of the interview at the time of the interview (mentioned; not mentioned; refused/not (mentioned; not mentioned; refused/not ascertained/DK)ascertained/DK)

No health insurance coverageNo health insurance coverage for more than one year for more than one year (yes, no; refused/not ascertained/DK)(yes, no; refused/not ascertained/DK)

NHIS VariableNHIS Variable Using the MEPS Full Year panel Using the MEPS Full Year panel specific weight for calendar year specific weight for calendar year

respondentsrespondents

Using the full sample NHIS and Using the full sample NHIS and NHIS weight (prior year)NHIS weight (prior year)

Calendar YearCalendar Year NOTCOV (Health NOTCOV (Health insurance insurance coverage status)coverage status) %% SESE %% SESE

20072007 Not coveredNot covered 15.315.3** (0.69)(0.69) 16.816.8 (0.29)(0.29)

CoveredCovered 83.783.7** (0.71)(0.71) 82.082.0 (0.29)(0.29)

Refused/NA/DKRefused/NA/DK 1.11.1 (0.18)(0.18) 1.21.2 (0.08)(0.08)

20062006 Not coveredNot covered 15.915.9 (0.53)(0.53) 16.316.3 (0.24)(0.24)

CoveredCovered 83.383.3 (0.56)(0.56) 82.882.8 (0.24)(0.24)



CoveredCovered 82.782.7 (0.56)(0.56) 82.882.8 (0.24)(0.24)



CoveredCovered 83.483.4 (0.61)(0.61) 82.682.6 (0.27)(0.27)


Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07Source: Medical Expenditure Panel Survey Household Component (MEPS-HC), 2004-07; National Health Interview Survey (NHIS), 2004-07

*significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the .05 level*significant difference in MEPS derived NHIS estimate relative to the NHIS derived estimate at the .05 level

NHIS NHIS any insurance coverageany insurance coverage at time of interview at time of interview derived from MEPS sample and estimation weights derived from MEPS sample and estimation weights compared to NHIS sample and weights, age <65compared to NHIS sample and weights, age <65

NHIS NHIS private insuranceprivate insurance derived from MEPS derived from MEPS sample and estimation weights compared to sample and estimation weights compared to NHIS sample and weights, age <65NHIS sample and weights, age <65



Using the full sample NHIS andUsing the full sample NHIS and

NHIS weight (prior year)NHIS weight (prior year)

Calendar YearCalendar Year HIKINDA (Private HIKINDA (Private insurance)insurance) %% SESE %% SESE

20072007 MentionedMentioned 68.968.9** (1.00)(1.00) 66.366.3 (0.44)(0.44)

Not MentionedNot Mentioned 30.130.1** (1.01)(1.01) 32.632.6 (0.44)(0.44)


20062006 MentionedMentioned 69.269.2 (0.81)(0.81) 68.368.3 (0.39)(0.39)

Not MentionedNot Mentioned 30.030.0 (0.80)(0.80) 30.730.7 (0.39)(0.39)










NHIS NHIS Medicaid coverageMedicaid coverage derived from MEPS derived from MEPS sample and estimation weights compared to NHIS sample and estimation weights compared to NHIS sample and weights, age <65sample and weights, age <65



Using the full sample NHIS and Using the full sample NHIS and

NHIS weight (prior year)NHIS weight (prior year)

Calendar YearCalendar Year HIKINDD HIKINDD (Medicaid)(Medicaid)

%% SESE %% SESE















NHIS NHIS insurance coverage for >1 yearinsurance coverage for >1 year derived from derived from MEPS sample and estimation weights compared MEPS sample and estimation weights compared to NHIS sample and weights, age <65to NHIS sample and weights, age <65



Using the full sample NHIS and Using the full sample NHIS and NHIS weight (prior year)NHIS weight (prior year)

Calendar YearCalendar Year HILAST (No HILAST (No health insurance health insurance for more than for more than one year)one year) %% SESE %% SESE















What did we learn?What did we learn?

MEPS overlapping panel design facilitates the MEPS overlapping panel design facilitates the evaluation of periodic survey design changes. evaluation of periodic survey design changes.

The linkage of MEPS and NHIS also facilitates data The linkage of MEPS and NHIS also facilitates data quality assessments.quality assessments.

The internal comparison of MEPS insurance The internal comparison of MEPS insurance estimates (<65) by panel for 2007 and for years estimates (<65) by panel for 2007 and for years prior to the redesign did not show any significant prior to the redesign did not show any significant differences.differences.

The logistic regression analysis likewise did not The logistic regression analysis likewise did not reveal any significant effect for Panel.reveal any significant effect for Panel.

What did we learn (cont.)?What did we learn (cont.)?

Phase 2 of this study evaluated the joint impact of the new Phase 2 of this study evaluated the joint impact of the new 2007 sample design and the survey’s post-survey 2007 sample design and the survey’s post-survey adjustments.adjustments.

The MEPS-NHIS linked analysis only showed modest The MEPS-NHIS linked analysis only showed modest differences in 2 of 4 NHIS insurance variables examined.differences in 2 of 4 NHIS insurance variables examined.

No evidence of nonresponse bias attributable to year 1 No evidence of nonresponse bias attributable to year 1 survey attrition.survey attrition.

Limitation of this study: We could not tease out the Limitation of this study: We could not tease out the independent effects of the two survey modifications.independent effects of the two survey modifications.

Nevertheless, the dual survey changes in 2007 did not Nevertheless, the dual survey changes in 2007 did not appear to have a major impact on insurance coverage appear to have a major impact on insurance coverage estimates.estimates.

Future researchFuture research

Assess the 2008 MEPS panel specific estimates Assess the 2008 MEPS panel specific estimates CAPI platform and sample design will be the same in CAPI platform and sample design will be the same in

both individual overlapping panels. both individual overlapping panels.

Continued evaluations of variables used to adjust for Continued evaluations of variables used to adjust for nonresponse.nonresponse.

Evaluate transitions in insurance coverage from NHIS to Evaluate transitions in insurance coverage from NHIS to MEPS, pre-and post-new design.MEPS, pre-and post-new design.

Evaluate trends in transitions in coverage in MEPS from Evaluate trends in transitions in coverage in MEPS from year 1 to year 2, pre- and post new design.year 1 to year 2, pre- and post new design.

Review CAPI programming and edit specifications. Review CAPI programming and edit specifications.

DiscussionDiscussion

Comments on the internal and external Comments on the internal and external evaluation strategies used in this study? evaluation strategies used in this study?

What other strategies have been (or can be) What other strategies have been (or can be) used to evaluate survey data quality used to evaluate survey data quality following implementation of design or following implementation of design or methodological modifications?methodological modifications?

an assessment of the impact of two distinct survey design modifications on health insurance coverage...

Documents

health survey

meps panel

new sample design

nhis design

key survey resource

health care policy meps

dual survey changes

previous design