1 making sense of evaluation data mary michaud, mpp university of wisconsin— cooperative extension...
TRANSCRIPT
1
Making Sense of Evaluation Data
Mary Michaud, MPP
University of Wisconsin—Cooperative Extension
Fall 2002
2
Why are these skills important?
• Fundamental skills for public health service
• Make sense of your own data
• Interpret other data
• Make your case
3
Let’s get started
• Introductions and agenda
• Making sense of evaluation data:– Top ten takeaway lessons from the
workshop
• What about this report do you find informative?
• What would you change?
4
Myths
• One report is enough.
• People read written reports.
• Complex analysis and big words impress people.
• Oral reports have the same effect as written reports.
• Describing limitations weakens report.
• Everything should be reported.
• The audience knows why they are getting the report.
5
Building an evaluation plan
1. Identify the purpose of evaluation
2. Clarify who will use the results
3. Clearly describe what is being evaluated (use a logic model)
4. Specify questions to ask
5. Identify sources of information
6. Select methods to collect information
7. Analyze and interpret information
8. Report and use results
6
Making sense of the data
• Start with a plan before you collect data– Purpose– Who will use the information– Resources– Sources of information and data collection
methods
• Collect data• Clean data• Code data• Tabulate your data• Describe and interpret data
7
8
9
10
Why collect quantitative data?
• To make comparisons…– Between groups
• Smokers vs. Non-smokers• Opinions of people who heard radio ad vs.
people who didn’t• Men vs. Women
– Over time• Change in public support for smoke-free
ordinance
11
Why collect qualitative data?
• Explore meaning, motivation, emotion
• Understand experiences
• Understand language people use to describe their experiences
• Examples: interviews, focus groups, journals, document review
12
13
How many? How often?
• Frequency, or count
• Useful when real numbers adequately tell the story
Ten worksites in Williams County have more than 300 employees.
Between 2001 and 2003, six of those worksites implemented policies to ban smoking. As a result, 2,400 workers in Williams County now work in smoke-free environments.
This report documents the role the Williams County Tobacco Free Coalition played in promoting this change.
14
*Surveys sampled worksites with more than five employees.
**Sources: University of Wisconsin Monitoring and Evaluation Program. Results of 2001 Wisconsin Worksite Smoking Policy Survey. March 2002. Williams County Tobacco Free Coalition. Results of 2001 Worksite Smoking Policy Survey. October 2001.
What proportion?Percentage of Williams County and Wisconsin worksites
that ban smoking indoors, 2001*
60%
74%
0
10
20
30
40
5060
70
80
90
100
Williams County Worksites Wisconsin Worksites
Per
cen
tag
e o
f w
ork
site
s**
15
What’s the norm? (or central tendency)
16
How much do the data vary?
• Range
• Standard deviation (SD)– The larger the SD, the greater the
variability in data– With a smaller sample size, outliers receive
more “weight”– In a normal distribution, 65% of data lie
within one SD and 95% lie within two SD
• Key to interpreting other studies
17
Mean
1 SD
2 SD
Number of 6th graders
Height
y
x
n = 32
n = 320
18
Sampling
• What is a random sample? • Why sampling works• Claims you cannot make
• Save time, save money
19
Is my survey “valid?”
Validity of results depends on:• Sampling
– Quality of sampling frame– Sampling method
• Questionnaire design• Questionnaire administration
– Telephone– Mail
• …And other things!
20
Validity and reliability
• Validity: Are you measuring what you think you are measuring?– There are multiple types of validity
• Reliability: If something was measured again using the same instrument, would it produce the same (or damn near the same) results?– There are multiple places reliability can break
down
• Why are these important?
21
Interviewer 1. I would like to ask you a few questions about smoking.
Interviewer 2. I would like to ask you a few questions about smoking. [I’m a smoker, so you don’t have to worry about telling me if you smoke. It will really help us if you are honest about this.]
22
Do you favor or oppose a city ordinance that would make all Williamsburg restaurants smoke-free?
Yes 71.3%
No 28.7%
23
Supports restaurant ordinance
Opposes restaurant ordinance
Undecided/ declined to
comment
Current smokers(n=55)
8
(15% of smokers)
33
(60% of smokers)
14
(25% of smokers)
Non-smokers(n=200)
170
(86% of non-smokers)
16
(8% of non-smokers)
12
(6% of non-smokers)
Total (N=255)
178
(70% of all respondents)
49
(19% of all respondents)
26
(11% of all respondents)
24
“Farming it out”
Pros• Expertise• Time• Scope
Cons• Expertise?• Expense• Supervision required
Find out:• Exactly what services they provide • Types of past accounts • Willingness to share the work with you • Willingness to do “pro bono” or reduced-fee work • If they will provide you technical assistance
25
“Farming it out”
Remember:
• Do not show your gold!
• Request a proposal
• Get your raw data after it is collected
26
Analysis tips
• Analyzing “by hand”
• Excel
• Other programs:– Epi info (CDC data management and
analysis program: www.cdc.gov/epiinfo)– SPSS (statistical software)– Microsoft Access (database)
27
28
Analyzing qualitative data
“Content analysis” steps:
1. Transcribe data (if audio taped)
2. Read transcripts
3. Highlight quotes and note why important
4. Code quotes according to margin notes
5. Sort quotes into coded groups (themes)
6. Interpret patterns in quotes
7. Describe these patterns
29
Qualitative data analysis
• Words• Context• Internal consistency• Frequency of comments• Extensiveness of comments• Intensity of comments• Specificity of responses• What was not said
30
31Example data set
32
Discussing limitations
Written reports: • Be explicit about your limitationsOral reports: • Be prepared to discuss limitations• Be honest about limitations• Know the claims you cannot make
– Do not claim causation without a true experimental design
– Do not generalize to the population without random sample and quality administration (e.g., <60% response rate on a survey)
33
Reporting results
• Format depends on purpose and audience
• Written, oral• Summative, formative
• What is the audience used to hearing or seeing?
• Common graphics– Photographs
34
Using graphics
• Title • Clear units of measure • Date(s) data collected• Simple, straightforward design without
“clutter”• Font size 10 point or larger• Explicit data source(s)• Sample size, if applicable for the
audience
35
Percentage of Williams County and Wisconsin worksites
that ban smoking indoors, 2001*
60%
74%
0
10
20
30
40
5060
70
80
90
100
Williams County Worksites Wisconsin Worksites
Per
cen
tag
e o
f w
ork
site
s**
*Surveys sampled worksites with more than five employees.
**Sources: University of Wisconsin Monitoring and Evaluation Program. Results of 2001 Wisconsin Worksite Smoking Policy Survey. March 2002. Williams County Tobacco Free Coalition. Results of 2001 Worksite Smoking Policy Survey. October 2001.
36
Reporting results to the media
All media:
• Avoid using too many statistics. Focus on the key points.
• For quotes, speak more globally about the issue.
• Always give the source and timeliness of your stats. It’s the “news peg.”
Steve Busalacchi
Director, News & Information
Wisconsin Medical Society
37
Reporting results to the media
Radio and TV:• Do not offer exact statistics—ear cannot
track. “73.6% of respondents”
vs.“Nearly three quarters of those surveyed”
• Don’t go into great detail. Have backup info ready.
Steve Busalacchi
Director, News & Information
Wisconsin Medical Society
38
Analyze!
• Worksite data set• What is the average number of employees?• Do worksites with smoking policies tend to be
larger or smaller?• Is there a need for health insurance coverage
for cessation at these worksites? • How many worksites have had cessation
programs on site?• What is the most important reason worksites
have instituted smoking policies?
39
Myths
• One report is enough.
• People read written reports.
• Complex analysis and big words impress people.
• Oral reports have the same effect as written reports.
• Describing limitations weakens report.
• Everything should be reported.
• The audience knows why they are getting the report.
40
Online resources
Evaluation assistance
• www.uwex.edu/ces/tobaccoeval
State and local data
• www.census.gov
• www.medsch.wisc.edu/mep/
Economic impact of smoking policies
• www.no-smoke.org
41
42
43
44
45
46
Making sense of your data
• What challenges have you faced?
• Top ten lessons: Review