what works in multi-site evaluations of nutrition education interventions? andy fourney, andrew...
TRANSCRIPT
![Page 1: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/1.jpg)
What Works in Multi-Site What Works in Multi-Site Evaluations of Nutrition Evaluations of Nutrition Education Interventions?Education Interventions?
What Works in Multi-Site What Works in Multi-Site Evaluations of Nutrition Evaluations of Nutrition Education Interventions?Education Interventions?
Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie KeihnerCalifornia Nutrition Network for Healthy Active Families
Research and Evaluation Unit
![Page 2: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/2.jpg)
California Nutrition Network California Nutrition Network for Healthy, Active Familiesfor Healthy, Active Families
Targets low-income families
Conducts a social marketing campaign to increase: fruit and vegetable
consumption physical activity promote the Food Stamp
Program Contracts over 170
agencies Receives almost 106
million from USDA food stamp program
![Page 3: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/3.jpg)
Strategies
Factors thatInfluence
FVC
Change inBehavior
Evaluation for contractors Evaluation for contractors receiving >$350kreceiving >$350k
Process
Short-term
ImpactLong-term
impact
![Page 4: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/4.jpg)
Network Developed Network Developed MethodologyMethodology
1. SOW Impact evaluation objective2. Compendium of Surveys 3. Impact Evaluation Handbook 4. Trainings and workshops 5. Data entry templates 6. Evaluation report template7. Monitoring
![Page 5: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/5.jpg)
Scope of Work Impact Scope of Work Impact Evaluation ObjectiveEvaluation Objective
Impact objective: By Sep. 30 20xx, a sample of 4th and 5th grade students at intervention sites will report an increase in fruit and vegetable consumption and an increase in one or more factors related to fruit and vegetable consumption, such as knowledge, preferences, outcome expectations and self-efficacy.
![Page 6: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/6.jpg)
Impact Evaluation Impact Evaluation HandbookHandbook
![Page 7: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/7.jpg)
Compendium of SurveysCompendium of Surveys
![Page 8: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/8.jpg)
TrainingsTrainings
WorkshopsWorkshops
![Page 9: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/9.jpg)
One-on-one TAOne-on-one TA
![Page 10: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/10.jpg)
Data Entry TemplatesData Entry Templates
Created in MS ExcelProvides same results as SPSSCalculate pretest mean, posttest
mean, difference and p-value for a paired t-test
Customized for each survey
![Page 11: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/11.jpg)
Data Entry TemplatesData Entry Templates
![Page 12: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/12.jpg)
Data Entry TemplatesData Entry Templates
![Page 13: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/13.jpg)
Evaluation Report TemplateEvaluation Report Template
18 questionsDesignMethodsResultsReflectionSuccesses
![Page 14: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/14.jpg)
Report Template - Report Template - DesignDesign
1. Please type your impact objective. 2. What were the factors you measured?3. What intervention and strategies were implemented
to change the factors? 4. What survey tool did you use? 5. Were there questions on the survey tool that were
not addressed in the intervention? If so, why?6. Describe challenges faced during the intervention
implementation. 7. What type of design was used, e.g., pretest and
posttest, pre-posttest with a control, posttest only or other?
![Page 15: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/15.jpg)
Report Template - Report Template - Evaluation Survey ImplementationEvaluation Survey Implementation
8. What type of design was used, e.g., pretest and posttest, pre-posttest with a control, posttest only or other?
9. Who were the survey respondents? (please give age range)
10. Where were they administered (name and type of locations, e.g., health department, community center, mobile home park, etc.)
11. When were the pretest and posttest surveys administered? (Please give dates.)
12. How many were administered? (If a different number of pretests and posttest please explain.)
13. How long did it take the respondents to answer the survey? (Please give range in minutes.)
![Page 16: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/16.jpg)
Report Template - Report Template - Evaluation ResultsEvaluation Results
14. What were the results? How would you interpret these results? How might these results be used?
15. How would you interpret these results?
16. How might these results be used?
![Page 17: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/17.jpg)
Report Template - Report Template - ReflectionReflection
17. What were the big challenges you encountered during the evaluation process?
18. Now, reflect on the successes you’ve had over the past year. Describe the best moment you had as a nutrition educator this year. Think about a moment when you knew your nutrition education had made a difference in someone’s life or give an example of a time when you were proud to be a nutrition educator.
19. Share a story about a one of the best moments you had while conducting the evaluation this year? Think about one of the highlights?
20. If Aladdin appeared with his magic lamp and offered you a wish to have more of those best moments or highlights what would you ask for?
![Page 18: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/18.jpg)
Monitoring sheet (in MS Monitoring sheet (in MS Excel) includes columns for:Excel) includes columns for:
Contractor Name Contact Name Region Channel Federal Share Program Manager Required/Volunteer Contract External Evaluator Number of years in impact evaluation
![Page 19: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/19.jpg)
Monitoring sheet Monitoring sheet (continued)(continued)
Evaluation Objective Factors (to be)/(that were) measured Design (posttest only, posttest with control, pre-
posttest, pre-post with control) Participant’s age (grade, adults/youth, in years) Survey Used Have a copy of survey Pretest date(s) Posttest date(s) Number of Surveys expected (collected)
![Page 20: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/20.jpg)
Monitoring sheet Monitoring sheet (continued)(continued)
Need Data Entry Template (date) Sent data entry template (date) Received Report (in email from & date) Report Late/Received Draft Extension Granted (notes) Data Received Need Training
![Page 21: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/21.jpg)
What were the indicators of What were the indicators of success?success?
Factor
# that measured
change in factor
03-04 04-05 05-06
Knowledge 7 7 19
Preferences 10 13 22
Self-efficacy 3 4 18
Outcome expectations
2 2
Consumption 1 7 22
PA preferences 0 1
![Page 22: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/22.jpg)
Who has participated?Who has participated?
Channel03-04
n=12 (44%)
04-05
n=24 (67%)
05-06
n=45 (98%)
Schools 8 12 24
Colleges and universities
2 3 4
Health departments
1 6 11
County Offices of Ed.
2 6
Cooperative Extension
1 1 1
First 5 1
![Page 23: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/23.jpg)
Funding Funding (% of total budget)(% of total budget)
Federal Funding of Contractors that Participated in Impact Evaluation
16
30
45
0
10
20
30
40
50
2003-2004 2004-2005 2005-2006
Year
Fu
ndin
g
(In
Million
s o
f D
ollars
)
(19%) (34%) (47%)
![Page 24: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/24.jpg)
ResourcesResources
www.ca5aday.comResearch and Evaluation
See impact evaluation section at the bottom
www.harvestofthemonth.com
![Page 25: What Works in Multi-Site Evaluations of Nutrition Education Interventions? Andy Fourney, Andrew Bellow, Patrick Mitchell, Sharon Sugerman, Angie Keihner](https://reader038.vdocuments.us/reader038/viewer/2022110402/56649e455503460f94b39536/html5/thumbnails/25.jpg)
Hoorah! – More MattersHoorah! – More Matters