WRITING GREAT TOBACCO CONTROL FINAL EVALUATION REPORTS

WEBINAR TRAININGAPRIL 4 & 5, 2013

Welcome!

AGENDA

Thursday, April 4th

10:00-10:10 Welcome!

10:10-10:45 The Project Description Section::

Overview, scoring the report, discussion

10:45-11:30 The Methods Section

Overview, scoring the report, discussion

Friday, April 5th

10:00-10:45 Writing Up Results & Conclusions, Overview

10:45-11:30 Scoring and Discussion

2

WHY FINAL REPORTING?

OrganizationalHistory

Sharing with colleagues

idi.org

3

STATEWIDE & NATIONAL RELEVANCE

TCEC publishes summary reports and articles

4

Reporting Guidelines5

FER RATING FORM (maximum 32 points)

Report Section Maximum Points

Title Page 2

Abstract 6

Project Description 6

Evaluation Methods 8

Evaluation Results 4

Conclusion & Recommendations 6

6

PROVIDING SOME CONTEXT: PROJECT DESCRIPTION SECTION

PROJECT DESCRIPTION SCORING(maximum 6 points)

Project Description

Maximum Points

Background 2Objective 2Intervention 2

8

PROJECT DESCRIPTION ELEMENTS

Provide context for the objective Explain rationale for choosing the objective

State the objective and indicator Describe what you did and why Identify activity timeframes Explain any changes to plan

9

IT’S YOUR TURN TO SCORE

10-15 minutes • Read and score the project description

of this report• Write on the report any questions you

have

5-10 minutes• Discuss scores as a group.

10

EVALUATION METHODS

DESIGN TYPEPROCESS AND OUTCOME

Experimental (control group, random assignment)

Quasi-experimental (at least three waves, comparison group)

Non-experimental (no comparison, only two waves)

Process and/or Outcome

12

DESIGN

LIMITATIONS

13

SAMPLES

14

DATA COLLECTION INSTRUMENTS AND PROCEDURES

15

DATA ANALYSIS

16

EVALUATION METHODS SCORING(MAXIMUM 8 POINTS)

Evaluation Methods

Evaluation design 2

Samples from which data were collected

2

Data collection instruments and procedures

2

Data analysis 2

17

IT’S YOUR TURN TO SCORE

10-15 minutes• Read and score the Evaluation

Methods section in pairs• Write on the report any questions you

have

5-10 minutes• Discuss scores as a group.

18

SCORING METHODS SECTION

19

REMEMBER TO JOIN US TOMORROW!PART 2 begins at 10:00 a.m. Same login info:

1. Call in for audio at 1.866.740.1260 and enter access code 2974659, AND

2. Log in for the visuals at www.readytalk.com and enter the access code 2974659

20

Our team — Diana Cassady, Jeanette Treiber, Robin Kipke, Catherine Dizon, Diana Dmitrevsky

Our website — http://tobaccoeval.ucdavis.edu

Our email — tobaccoeval@ucdavis.edu

Our main line — 530.752.9951

We are the statewide technical assistance center on evaluation for all Prop. 99-funded projects in California

21

WRITING GREAT TOBACCO CONTROL FINAL EVALUATION REPORTS

WEBINAR TRAININGAPRIL 4 & 5, 2013

Part 2

Welcome!

AGENDA

Thursday, April 4th

10:00-10:10 Welcome!

10:10-10:45 The Project Description Section

Overview, scoring the report, discussion

10:45-11:30 The Methods Section

Overview, scoring the report, discussion

Friday, April 5th

10:00-10:45 Writing Up Results & Conclusions, Overview

10:45-11:30 Scoring and Discussion

23

WRITING UP RESULTS

BRAINSTORM

What is the purpose of the results section of a report?

What function(s) should it serve?

25

PURPOSE OF THE RESULTS SECTION

Present the findings of data analysisExplain what the data revealPoint out what is importantDescribe how the data informed your

projectJustify report conclusions

26

FER SCORING CRITERIA

Evaluation Results Maximum Points

Findings are clearly & logically presented. Tables and figures are used when needed and are clearly labeled.

2

Findings are objective (include both positive and negative when relevant)

2

Section total 4

27

WHAT TO INCLUDE

Report on every activityBut don’t report every result

28

QUANTITATIVE DATA

Provide totals and percentagesNote statistical significance (where

appropriate)

29

202 (79%) Alta Vista residents were in favor of the policy

EXCEL-GENERATED TABLE

30

A paired T-test comparing 2010 and 2013 shows a statistically significant difference in the number of cigarette butts between 2010 and 2013 (p<0.05)

Table 1. Number of Cigarette Butts in Sacramento Area Parks, by Year

Location 2010 2011 2012 2013

Arden 68 72 81 37

Sunrise 282 343 314 45

Fair Oaks 121 180 196 58

McKinley 312 369 231 111

Goethe 216 248 239 222

Total Citywide *999 1212 1061 *473

31

*Statistically significant difference p<0.05

VISUALLY IMPROVED TABLE

LINE GRAPH

32

BAR CHART

33

QUALITATIVE DATA

Common themesContrastsIndicative quotesAvoid percentages

34

SUMMARIZING CONTENT ANALYSIS

35

Figure 3. Incentives and Barriers to Smoke Free Casino Policy

BALANCING ACT

Focus on what’s most importantReport both + and - resultsMake results transparent

36

ATTENTION GETTERS

Supplement text with visual elements

Draw focus to key findings

Make data digestible Match format to data

type

37

CONCLUSIONS & RECOMMENDATIONS

FER SCORING CRITERIA

Conclusions & Recommendations Maximum Points

Discuss outcome/process evaluation 2

Conclusions are supported by data 2

Includes recommendations for future work 2

Section total 6

39

INTERPRETING THE FINDINGS

Confirm what you expected to find, and didDispel what you thought was true, but isn’tBring to light what you didn’t know, but should

40

LEAVING A ROADMAP

What strategies worked well or poorly?What would you do differently next time?What steps should follow?

41

IT’S YOUR TURN TO SCORE

10-15 minutes:• Read and score the results and conclusions/

recommendations section • Note any questions you have

10-15 minutes:• Reconvene to share scores

42

AS YOU REVIEW THE RESULTS…

1.What questions does it leave unanswered?

2.Do the results address what the project wanted to know?

3.Did the section provide enough information for the reader to assess the findings?

4.Are the most important findings brought out in the narrative and/or the tables?

5.Do the findings appear to be objective?

6.Does the section adequately interpret the results by explaining how the data informed the project?

43

AS YOU REVIEW CONCLUSIONS

1. Do the conclusions evaluate the process and outcome findings in relation to the achievement of the objective?

2. Are the conclusions supported by the data?

3. Do the recommendations provide a clear roadmap for next time?

44

REPORT RATING

Total score: ___/32 possible points

Rating: ____ High (24-32)

____ Medium (16-23)

____ Low (0-15)

Overall Assessment: Recommend report for intervention activities?Recommend report for evaluation?

45

WORKSHOP RECAP

Why final evaluation reports matter Essential components of a report Provide enough detail about your path Make rationale and linkages transparent How reports are scored Use ratings & feedback to improve

46

PARTICIPANT SELF-ASSESSMENT

Help us learn how we’re doing.Complete the self-assessment form

and rate your understanding of writing final evaluation reports

before and after this training.

Thank you!

47

Our team — Diana Cassady, Jeanette Treiber, Robin Kipke, Catherine Dizon, Diana Dmitrevsky

Our website — http://tobaccoeval.ucdavis.edu

Our email — tobaccoeval@ucdavis.edu

Our main line — 530.752.9951

We are the statewide technical assistance center on evaluation for all Prop. 99-funded projects in California

48