data collection methods
TRANSCRIPT
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Jonathan HallettEvaluation of Health Promotion 382DATA COLLECTION METHODS
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Lecture overview
Recap – validity and study designs
Quantitative methods
Qualitative methods
Mixed methods
Indicators and measures
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Draw the following study designs
1. Pre- and post-test with single group
2. Time series
3. Pre- and post-test with comparison group
4. Randomised control trial
5. Post-intervention measure only
6. Staggered design
7. Cross-over design
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Increasing HIV education in government senior high schools Yr 8 Health Education teachers from Perth metro high
schools were allocated to a comparison group and an intervention group.
The study consisted of a 1 year intervention trial aimed at evaluating HIV/AIDS education strategies
Baseline measures were recorded and measures on completion of the program.
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Answer:
O1____X1____O2O1__________O2
Pre-post test with non-equivalent comparison group (Design 4)
Advantages – strong study Disadvantages – expensive, attrition
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
School Leaver Cohort Study
The study focused on Year 10 and Year 12 students from high schools in Western Australia.
The study aimed to identify annual exercise participation rates and psychosocial factors experienced by participants as they moved 3 years beyond their secondary schooling.
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Answer:
Yr 12 – O1 ______ O2 ______ O3
Yr 10 – O1 ______ O2 ______ O3
Cohort – Observation
Time series (Design 5)
Disadvantages: Attrition, Maturation
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Alcohol Price Variation Study
A cross sectional telephone survey of all liquor licensed premises formed the basic study design.
The study aimed to map the costs of buying alcohol in different locations.
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Answer:
O1
Single observation
Formative research
Just one observation
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
National Tobacco Campaign
The National Tobacco Campaign in 1999 featured the mass media‘every cigarette is doing you damage’ campaign.
The government claimed over 190 000 Australians gave up smoking as a result of the campaign during 1999 by looking at data in the 1996 and 2001 census.
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Answers
O1______X1______O2
Cross sectional data (1996 and 2001 census)
Post test with intervention (single group)
Consider population changes?
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Internal and external validity
Measurement validity
Study validity
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Internal and external validity
Study validity:
Refers to the degree to which an inference or generalisation can be made within the sample (internal validity) and beyond the study sample (external validity).
Consider:- Study methods
- Representativeness of the study sample
- Population from which the sample is drawn
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Internal validity
Did the intervention in fact make a difference in this specific instance?
The extent to which the results of the study can be generalised to the participants.
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
External validity
To what populations, settings, treatment variables, and measurement variables can this effect be generalised?
The extent to which the results be generalised beyond the study participants to the target population as a whole.
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Threats to internal validity
Selection bias
Attrition
Testing effects
History
Instrumentation
Maturation
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Threats to external validity
Interaction effects of selection bias and the intervention
Reactive or interaction effects of pre-testing
Reactive effects of intervention procedures
Multiple treatment interference
Refers to Generalisability
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Activity – Justify an Evaluation Design
Identify an appropriate evaluation design for your project. Illustrate the design using X (intervention) and O
(observation/measurement). Justify why you have selected this design. Consider advantages and problems with your evaluation
design. Present to the class your project and evaluation design.
Evidence + Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Group activity
Revisit your group health issue. Identify an appropriate evaluation design for your project. Illustrate the design using X (intervention) + O
(observation/measurement). Justify why you have selected this design. Consider advantages and problems with your evaluation
design. Present to the class your project and evaluation design.
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
How would you describe these approaches?
1. Qualitative
2. Quantitative
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Methods
1. Qualitative:How and why a phenomena takes place
2. Quantitative:How many, how much and how often
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Characteristic Qualitative Quantitative
Objective To gain understanding of underlying reasons and motivations
To quantify the data and includes possibility of generalising results from a sample to a population of interest
Sample Small number of non-representative cases
Usually a larger number of representative cases
Data Collection Unstructured/semi-structured Structured
Data Analysis Non-statistical Statistical
Outcome Develop an initial or deeper understanding – adds meaning
Recommend a course of action
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
• frequency• distribution• scale of changes• representativeness
• range of experience• interrelationships• explanations
Qualitative Quantitative
Qualitative and quantitative approaches
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Brainstorm methods
Qualitative
Evaluation of Health Promotion 382
Quantitative
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
• Generally:• Surveys• Audits
• Post-only• Pre-and post• Time series• RCT
• In-depth interviews• Focus groups• Observation• Document review• Visual images
Qualitative Quantitative
Qualitative and quantitative approaches
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Why do you think qualitative approaches often get a bad rap?
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Mixed methods
Narrative can be used to add meaning to numbers
Numbers can be used to add precision to narrative
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Mixed methods
Triangulation
Complementary
Initiation
Development
Expansion
Evaluation of Health Promotion 382
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Group activity
Discuss whether qualitative or quantitative approaches are more useful for your project evaluation.
Are there some aspects of the evaluation that require one over the other?
Consider the challenges that might be relevant. Report back.
Evidence and Effectiveness in Health Promotion
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Next week
Surveys + Questionnaires
How do we write a questionnaire?
When are questionnaires appropriate?
Should we go online or use paper?
Evaluation of Health Promotion 382