operational and methodological lessons learned from the 2003 joint canada/u.s. survey of health
DESCRIPTION
Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health. Catherine Simile, Ph.D. National Center for Health Statistics. Overview. Genesis and Objectives Lessons learned Questionnaire Design Sampling and Data Collections Data Processing and Release. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/1.jpg)
Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health
Catherine Simile, Ph.D. National Center for Health Statistics
![Page 2: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/2.jpg)
Overview
• Genesis and Objectives
• Lessons learned– Questionnaire Design– Sampling and Data Collections– Data Processing and Release
![Page 3: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/3.jpg)
How It Started• Both NCHS and STC were involved in
international efforts to improve cross national comparisons of health data
• NCHS/STC yearly interchanges to discuss common interests
• Idea for a joint survey launched at the October 2000 Interchange
![Page 4: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/4.jpg)
Objectives
• Produce highly comparable data on the Canadian and American populations unaffected by difference in data collection methodology on the following core indicators:– Health care– Functional status– Health status– Risk factors
![Page 5: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/5.jpg)
Objectives (cont.)
• Influence content of the each country’s ongoing, national health surveys to enhance comparability and data quality
• Develop a model for successful collaboration towards standardizing concepts
![Page 6: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/6.jpg)
Organization• Project Team –Responsible for day to day
operations
• Steering Committee – project oversight
• All interviews conducted from Statistics Canada’s Regional Offices using RDD and CATI
![Page 7: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/7.jpg)
Survey tasks• Questionnaire design - questions taken from ongoing
surveys in both countries -the Canadian Community Health Survey and the National Health Interview Survey
• Average duration of questionnaire – 25 minutes
• Editing specifications specific to questions
• Interviews conducted in English, Spanish (US), and French (Canada)
![Page 8: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/8.jpg)
Population Covered
• Residents of Canada and the US aged 18 and older living in private dwellings with telephones
• Canadian samples stratified by province• US samples stratified by 4 regions• Sample designed to produce reliable national
estimates for 3 age groups (18-44, 45-64, 65 and over) by gender
![Page 9: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/9.jpg)
Sample Size
• Canada: 3,505
• US: 5,183
![Page 10: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/10.jpg)
Timeline
• October 2000 – Idea hatched
• June 2002 – Final content decided
• Nov. 4, 2002 – Data collection starts
![Page 11: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/11.jpg)
Timeline (cont.)
• July 14, 2003 - collection officially ended
• September, 2003 – official response rates calculated
• June, 2004 data and analytic report jointly released on websites
![Page 12: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/12.jpg)
Cognitive Testing: Window of Opportunity
Normal practice:• U.S.: individual one-on-one English interviews in
Washington D. C. agency office• Canada: focus groups in English in Ottawa and in
French in Montreal
JCUSH:• Combined approach: one-on-one interviews and focus
groups for both U.S. and Canada in agency labs and off-site
![Page 13: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/13.jpg)
Cognitive Testing, (cont’d)
– Forced us to think about comparability differently: More difference between subsamples in country than between countries
– Led to a new way of doing cognitive testing in U.S.
![Page 14: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/14.jpg)
Translation: Implementation of New Guidelines
Languages used in this survey– English (both countries)– French (Canada only, required by law)– Spanish (U.S. only, customary)
![Page 15: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/15.jpg)
Steps from the Guidelines:
1. Pre-translation preparation--NO2. Selection of contractor--YES3. Completion of translation from final text--NO4. Review of translation—YES
• Bilingual review used survey and topic experts, U. S. Census interviewers, translators, and French speaker from Canada
5. Adjudication--YES6. Development of survey instrument --YES7. Pretest of survey, including translators--NO8. Selection of interviewers YES9. Training of Interviewers--NO10. Incorporation of feedback from the field--YES
![Page 16: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/16.jpg)
Data Collection Expectations
– “Clean sample” and resolving cases
– Cooperativeness of Canadian and U.S. population different
– Differences in implementing legal requirements impacted discretion and authority of data collection staff
![Page 17: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/17.jpg)
DATA COLLECTION RESULTS: Resolution, Cooperation and Final Response Rates, JCUSH
80
50
69.3 69.362.7
100
0
10
20
30
40
50
60
70
80
90
100
Resolution Cooperation Response
United States Canada
![Page 18: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/18.jpg)
Resolving Cases
• Sampling methodologist for each country determined number of telephone lines necessary to reach intended sample sizes
# of lines selected Targeted Sample size
United States 32,009 5,000Canada 10,334 3,500
![Page 19: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/19.jpg)
Resolving Cases (Cont’d)
• Definition of clean sample different– GENYSIS removed 1/3 of original U.S. sample– The remaining 2/3 sent to Statistics Canada
assumed to be “clean”
• Working residential numbers cannot be verified in the U.S. like they can in Canada
![Page 20: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/20.jpg)
Cooperation: Refusals
Percentage of all eligible cases:
U.S. Canada• Refusals 21% 14%• Breakoffs 11% <.01%
TOTAL 33% 14%
![Page 21: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/21.jpg)
Collection Monitoring• Not necessary to monitor unresolved cases where 100% of
the cases can be relatively easily resolved
• The monitoring system used works well for a Canadian sample, but not for a U.S. sample where much of resolution work is done by field staff
• Throughout the data collection period, the staff was always uncertain as to what was happening, and so had difficulty ascertaining how to allocate resources
• Did not help that U.S. staff could not easily travel to Canada
![Page 22: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/22.jpg)
Discretion and Authority of Data Collection Staff
• Restrictions and delays to convert non response (all communications approved by NCHS Institutional Review Board)
![Page 23: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/23.jpg)
Data Release
Followed Statistics Canada’s usual practice. Preparation for release included both:
• editing and review of microdata for public release• collaborative analytical report released at the same
time
Collaborative analysis hampered by legal restrictions on data access
![Page 24: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/24.jpg)
Summary• Windows of Opportunity Provided
– New ways of doing cognitive testing– First opportunity to implement new translation guidelines
• Communication and Clarifying Assumptions Crucial– “Clean sample” and resolving cases– Differences in cooperativeness of Canadian and US population require
different monitoring and collection strategies
• Difference in implementing legal requirements not insignificant in their impacts– impacted the discretion and authority of interviewers– difficult to collaborate in analysis
![Page 25: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/25.jpg)
Access Data and Reports
NCHS website at www.cdc.gov/nchs/nhis.htm
Statistics Canada website at www.statcan.ca
![Page 26: Operational and Methodological Lessons Learned from the 2003 Joint Canada/U.S. Survey of Health](https://reader035.vdocuments.us/reader035/viewer/2022070420/56815e51550346895dcccbbf/html5/thumbnails/26.jpg)
Contact Information
Catherine M. Simile, Ph.D.National Center for Health Statistics
Division of Health Interview Statistics3311 Toledo Road, Room 2115
Hyattsville, MD 20782Phone: (301) 458-4499Email: [email protected]