1 making sense of evaluation data mary michaud, mpp university of wisconsin— cooperative extension...

1

Making Sense of Evaluation Data

Mary Michaud, MPP

University of Wisconsin—Cooperative Extension

Fall 2002

2

Why are these skills important?

• Fundamental skills for public health service

• Make sense of your own data

• Interpret other data

• Make your case

3

Let’s get started

• Introductions and agenda

• Making sense of evaluation data:– Top ten takeaway lessons from the

workshop

• What about this report do you find informative?

• What would you change?

4

Myths

• One report is enough.

• People read written reports.

• Complex analysis and big words impress people.

• Oral reports have the same effect as written reports.

• Describing limitations weakens report.

• Everything should be reported.

• The audience knows why they are getting the report.

5

Building an evaluation plan

1. Identify the purpose of evaluation

2. Clarify who will use the results

3. Clearly describe what is being evaluated (use a logic model)

4. Specify questions to ask

5. Identify sources of information

6. Select methods to collect information

7. Analyze and interpret information

8. Report and use results

6

Making sense of the data

• Start with a plan before you collect data– Purpose– Who will use the information– Resources– Sources of information and data collection

methods

• Collect data• Clean data• Code data• Tabulate your data• Describe and interpret data

7

8

9

10

Why collect quantitative data?

• To make comparisons…– Between groups

• Smokers vs. Non-smokers• Opinions of people who heard radio ad vs.

people who didn’t• Men vs. Women

– Over time• Change in public support for smoke-free

ordinance

11

Why collect qualitative data?

• Explore meaning, motivation, emotion

• Understand experiences

• Understand language people use to describe their experiences

• Examples: interviews, focus groups, journals, document review

12

13

How many? How often?

• Frequency, or count

• Useful when real numbers adequately tell the story

Ten worksites in Williams County have more than 300 employees.

Between 2001 and 2003, six of those worksites implemented policies to ban smoking. As a result, 2,400 workers in Williams County now work in smoke-free environments.

This report documents the role the Williams County Tobacco Free Coalition played in promoting this change.

14

*Surveys sampled worksites with more than five employees.

**Sources: University of Wisconsin Monitoring and Evaluation Program. Results of 2001 Wisconsin Worksite Smoking Policy Survey. March 2002. Williams County Tobacco Free Coalition. Results of 2001 Worksite Smoking Policy Survey. October 2001.

What proportion?Percentage of Williams County and Wisconsin worksites

that ban smoking indoors, 2001*

60%

74%

0

10

20

30

40

5060

70

80

90

100

Williams County Worksites Wisconsin Worksites

Per

cen

tag

e o

f w

ork

site

s**

15

What’s the norm? (or central tendency)

16

How much do the data vary?

• Range

• Standard deviation (SD)– The larger the SD, the greater the

variability in data– With a smaller sample size, outliers receive

more “weight”– In a normal distribution, 65% of data lie

within one SD and 95% lie within two SD

• Key to interpreting other studies

17

Mean

1 SD

2 SD

Number of 6th graders

Height

y

x

n = 32

n = 320

18

Sampling

• What is a random sample? • Why sampling works• Claims you cannot make

• Save time, save money

19

Is my survey “valid?”

Validity of results depends on:• Sampling

– Quality of sampling frame– Sampling method

• Questionnaire design• Questionnaire administration

– Telephone– Mail

• …And other things!

20

Validity and reliability

• Validity: Are you measuring what you think you are measuring?– There are multiple types of validity

• Reliability: If something was measured again using the same instrument, would it produce the same (or damn near the same) results?– There are multiple places reliability can break

down

• Why are these important?

21

Interviewer 1. I would like to ask you a few questions about smoking.

Interviewer 2. I would like to ask you a few questions about smoking. [I’m a smoker, so you don’t have to worry about telling me if you smoke. It will really help us if you are honest about this.]

22

Do you favor or oppose a city ordinance that would make all Williamsburg restaurants smoke-free?

Yes 71.3%

No 28.7%

23

  Supports restaurant ordinance

Opposes restaurant ordinance

Undecided/ declined to

comment

 Current smokers(n=55)

 8

(15% of smokers)

 33

(60% of smokers)

 14

(25% of smokers)

 Non-smokers(n=200)

 170

(86% of non-smokers)

 16

(8% of non-smokers)

 12

(6% of non-smokers)

 Total (N=255)

 178

(70% of all respondents)

 49

(19% of all respondents)

 26

(11% of all respondents)

24

“Farming it out”

Pros• Expertise• Time• Scope

Cons• Expertise?• Expense• Supervision required

Find out:• Exactly what services they provide • Types of past accounts • Willingness to share the work with you • Willingness to do “pro bono” or reduced-fee work • If they will provide you technical assistance

25

“Farming it out”

Remember:

• Do not show your gold!

• Request a proposal

• Get your raw data after it is collected

26

Analysis tips

• Analyzing “by hand”

• Excel

• Other programs:– Epi info (CDC data management and

analysis program: www.cdc.gov/epiinfo)– SPSS (statistical software)– Microsoft Access (database)

27

28

Analyzing qualitative data

“Content analysis” steps:

1. Transcribe data (if audio taped)

2. Read transcripts

3. Highlight quotes and note why important

4. Code quotes according to margin notes

5. Sort quotes into coded groups (themes)

6. Interpret patterns in quotes

7. Describe these patterns

29

Qualitative data analysis

• Words• Context• Internal consistency• Frequency of comments• Extensiveness of comments• Intensity of comments• Specificity of responses• What was not said

30

31Example data set

32

Discussing limitations

Written reports: • Be explicit about your limitationsOral reports: • Be prepared to discuss limitations• Be honest about limitations• Know the claims you cannot make

– Do not claim causation without a true experimental design

– Do not generalize to the population without random sample and quality administration (e.g., <60% response rate on a survey)

33

Reporting results

• Format depends on purpose and audience

• Written, oral• Summative, formative

• What is the audience used to hearing or seeing?

• Common graphics– Photographs

34

Using graphics

• Title • Clear units of measure • Date(s) data collected• Simple, straightforward design without

“clutter”• Font size 10 point or larger• Explicit data source(s)• Sample size, if applicable for the

audience

35

Percentage of Williams County and Wisconsin worksites

that ban smoking indoors, 2001*

60%

74%

0

10

20

30

40

5060

70

80

90

100

Williams County Worksites Wisconsin Worksites

Per

cen

tag

e o

f w

ork

site

s**

*Surveys sampled worksites with more than five employees.

**Sources: University of Wisconsin Monitoring and Evaluation Program. Results of 2001 Wisconsin Worksite Smoking Policy Survey. March 2002. Williams County Tobacco Free Coalition. Results of 2001 Worksite Smoking Policy Survey. October 2001.

36

Reporting results to the media

All media:

• Avoid using too many statistics. Focus on the key points.

• For quotes, speak more globally about the issue.

• Always give the source and timeliness of your stats. It’s the “news peg.”

Steve Busalacchi

Director, News & Information

Wisconsin Medical Society

37

Reporting results to the media

Radio and TV:• Do not offer exact statistics—ear cannot

track. “73.6% of respondents”

vs.“Nearly three quarters of those surveyed”

• Don’t go into great detail. Have backup info ready.

Steve Busalacchi

Director, News & Information

Wisconsin Medical Society

38

Analyze!

• Worksite data set• What is the average number of employees?• Do worksites with smoking policies tend to be

larger or smaller?• Is there a need for health insurance coverage

for cessation at these worksites? • How many worksites have had cessation

programs on site?• What is the most important reason worksites

have instituted smoking policies?

39

Myths

• One report is enough.

• People read written reports.

• Complex analysis and big words impress people.

• Oral reports have the same effect as written reports.

• Describing limitations weakens report.

• Everything should be reported.

• The audience knows why they are getting the report.

40

Online resources

Evaluation assistance

• www.uwex.edu/ces/tobaccoeval

State and local data

• www.census.gov

• www.medsch.wisc.edu/mep/

Economic impact of smoking policies

• www.no-smoke.org

41

42

43

44

45

46

Making sense of your data

• What challenges have you faced?

• Top ten lessons: Review