innovative synthesis methods
DESCRIPTION
Learn key steps to synthesize numerous evaluation studies that have different target groups, interventions, evaluation designs, statistical techniques and geographical locations. Utilizing principles of the realist synthesis method and multiple case studies contributes to innovative alternatives to the meta-analysis technique.TRANSCRIPT
Innovative Synthesis Methodologies
Donna Smith-Moncrieffe Canadian Evaluation Society Conference
Fairmont Royal York, TorontoJune 9-12, 2013
Innovative Synthesis Methodologies
Donna Smith-Moncrieffe Canadian Evaluation Society Conference
Fairmont Royal York, TorontoJune 9-12, 2013
2
Summary of the PresentationSummary of the Presentation
NCPC Mandate
Government Context Evaluation considerations at the corporate level (Performance) Evaluation considerations at the project’s level (Knowledge products)
Overview of 3 Types of synthesis methods Key points Challenges
Case Study (Step by Step Process) Collating data from projects with different target groups, interventions and outcomes Synthesis tables Providing reporting options for summarizing various statistical results: percentages,
Effect Sizes, T tests and F coefficients
Conclusions
3
Government Context: Evaluation Approaches Government Context: Evaluation Approaches
4
Why did we need to Synthesize the Data?Why did we need to Synthesize the Data?
● We needed to respond to the questions about, “what works in crime prevention”
● We needed to provide findings and conclusive statements about 13 programs (total of 36 project sites) with different:
Target groups;
Interventions;
Outcomes; and
Evaluation designs
● Single site repeated design
● Quasi-experiment design (sometimes with a matched comparison group)
5
Types of Synthesis MethodsTypes of Synthesis Methods
6
Synthesis Method Requirements for quality and RigorSynthesis Method Requirements for quality and Rigor
7
What approach did we use to Synthesize 13 Evaluations?
What approach did we use to Synthesize 13 Evaluations?
8
Step 1: Identify Key QuestionsStep 1: Identify Key Questions
What behaviour related changes were made (i.e. changes in violent offending, police contact and victimization?
What percentage of projects made favorable changes in relation to the criminal justice system (i.e. reduced violent offending, police contacts etc..)?
How much change was made?
What contextual and program related factors contributed to these changes (i.e. geographical location, partnerships, resources etc..)?
9
Step 2: What are the inclusion criteria?Step 2: What are the inclusion criteria?
● In the Government context, we need to use as many evaluation studies as possible in the synthesis; even when they do not meet rigid evaluation standards.
● Evaluation studies that met the following minimum requirements were included in the synthesis:
Study uses quantitative data (descriptive or inferential statistics);
Measures that correspond with key crime related outcomes (i.e. association with anti-social peers, emotional regulation, substance abuse, offending, education, employment etc..);
Study used at least the basic evaluation design (single group pre-post test repeated measures design); and
At least pre and post test measures had been available
10
Step 3: Enter the Data (Partial Sample)Step 3: Enter the Data (Partial Sample)
Name of Study
Description of Variable
Sample Size
Result P Level Type of Change
Level of Study
Project Intervention Toronto
Arrests E=76C=43
F=0.291 P=.823 No Change
Exemplary
Multi-Systemic Therapy
Youth not arrested for an offence
E=28C=12
E=89.3%C=66.7%
N/A Favorable Basic
Towards No Drugs
Weapon Carrying
E=847C=54
F=0.63 P<0.01 Favorable Notable
Velocity Police contacts
E=87 69%(reduction)
N/A Favorable Notable
Interpretation of Results
11
Step 4: Analysis and InterpretationStep 4: Analysis and Interpretation
Synthesis for 9 evaluation studies measuring changes in criminal behaviours
Overall interpretation
Background Information•Number of evaluation studies: 9•Number of sites: 11•Common measures include: 1) Arrests; 2) Non-Violent Offending; 3) Violent offending; 4) Criminal Victimization; and 5)Weapon Carrying
Quantitative results•% of change by sub-measure: 50%•% of projects demonstrating favorable change (showing at least one stat. significant result at the project level: 75%•% of projects demonstrating unfavorable change:0%•% of projects demonstrating no change: 25%
Qualitative results•Provide qualitative results by model, site or project level to qualify the quantitative findings 12
Step 5 : Reporting Aggregate DataStep 5 : Reporting Aggregate Data
13
Step 5: Reporting Aggregate Data (Sample of Approach # 1)Step 5: Reporting Aggregate Data (Sample of Approach # 1)
14
Step 5: Reporting (Sample of Approach #2 reporting on magnitude of change ) Step 5: Reporting (Sample of Approach #2 reporting on magnitude of change )
15
ConclusionsConclusions
● What synthesis method works best when the evaluation studies have a variety of interventions, outcomes and statistical measures?
Elements of the Realist Synthesis review and Multiple-case Study Synthesis are recommended; and
NCPC utilized elements of both of these synthesis methods but could benefit from responding to the Why questions incorporated in the Multiple- Case Study Synthesis
● What reporting method works best to summarize non-standard data?
Approach #1 and # 2 are both useful; and
Assess your audience to determine the best reporting method
16
Conclusions cont’dConclusions cont’d
● What planning activities would be required to utilize both Realist and Multiple Case study Methods?
Develop strategic direction (strategic plan to help focus the direction of the evaluation);
Prioritize key areas to minimize complexities;
Ensure that the information management system is designed to:
● Clarify the unit of analysis (by model, by project, evaluation questions, etc..);
● Respond to How and Why questions such as: How does the program produce these effects or How does the intervention contribute to change?; and
● Collect data about the context for each model, site or project.
17
References for Recommended Synthesis MethodsReferences for Recommended Synthesis Methods
18
19
Thank YouThank You
Contact:
Donna Smith-Moncrieffe, BSc. Crim Dip, MSc.
Senior Evaluation Advisor
Public Safety Canada, National Crime Prevention Centre
Policy, Research and Evaluation
Tel: 416-952-0423
Fax: 416-952-0483
Email: [email protected]
Web : www.PublicSafety.gc.ca | www.SecuritePublique.gc.ca