july 20, 2020 key concepts

July 20, 2020

Key ConceptsEvidence Building Academy

OPRE Report # 2021-108

2

Learning Objectives

You will be able to state the four key analytic concepts that will be used throughout the course.

You will be able to apply the theory of change to yourindividual/team project.

You will be able to identify who is included and excluded from an entry cohort, point-in-time, and exit cohort population.

First, quick review from June

4

1. Making better decisions requires evidence. Evaluation—particularly rigorous evaluation—is an

important way to produce evidence.

2. Rigorous evaluation follows scientific principles. Shows cause and effect, when relevant

Uses best practices in measurement

3. Rigorous evaluation is worth the investment. If you don’t evaluate, you can’t know if it works.

JUNE REV IEW

5

OVERVIEW

The Four Key Evaluation Concepts We Are Going to Talk About Today:Concept 1: Have a clear theory of change.

Concept 2: Ask a good question.

Concept 3: Think longitudinally.

Concept 4: Know your population.

Concept 1:

Have a Theory of Change

7

Your Theory of Change Will Guide Your Entire Evaluation A theory of change logically describes how and why

an intervention will cause a change. It is a compass, helping you stay on the path to get where

you want to go.

A logic model visually displays the theory of change. It is a map, showing the road that you plan to follow.

THEORY OF CHANGE

8

THEORY OF CHANGE

Four Statements to Complete before Starting Your Theory of Change:

I observe [some outcome I want to

improve].

I think it’s because [of this reason].

So I plan to [carry out this intervention],

Which I think will result in [an improved

outcome].

Source: L. A. Alpert, J. M. Haight, B. O. Orlebeke, and F. H. Wulczyn, EDGE: Evidence-Driven Growth and Excellence (Chicago: Chapin Hall at the University of Chicago, Center for State Child Welfare Data, no date).

9

THEORY OF CHANGE


You Need Evidence That Supports Your Theory of Change:

Which I think will result in…

How will you

know?

So I plan to…

How do you

know?

I think it’s because…

How do you

know?

I observe that…

How will you

know?

10

THEORY OF CHANGE


You Need Evidence, cont’d

Use an RCT to compare changes in use of corporal punishment and rates of

physical abuse substantiation after 1 year.

which I think will result in decreased use of

corporal punishment, and in turn, prevent physical

abuse.

Contact logs from our information system. Copies of parents’ action

plans for each session in case files.

So I plan to implement a

parent education & skill building program...

I did a qualitative analysis of parents’ responses to the family assessment tool and reviewed published studies

about why parents use corporal punishment.

I think it’s because parents do not have the knowledge and skills to apply age-appropriate

discipline.

I collected data from recent physical abuse investigations. I reviewed published studies on

what leads to the physical abuse of young children.

I observe that parents who physically abuse young children use

corporal punishment.

11

THEORY OF CHANGE


Use the Four Statements to Develop a Theory of Change for Your Project

How will you

know?

Which I think will result in…

How will you

know?

So I plan to…

How do you

know?

I think it’s because…

How do you

know?

I observe that…

12

Share Your Experience!

Was this easy or difficult?Were any gaps identified?

THEORY OF CHANGE

Concept 2:

Ask a Good Question

14

KNOW YOUR QUEST ION

Lery, B. Haight, J. M., Alpert, L. (2016). Four principles of big data practice for effective child welfare decision-making. Journal of Public Child Welfare, 10(4), 466-474, DOI: 10.1080/15548732.2016.1209149

A Good Question Is Key to Building Evidence for Your Program

Claim,Decision, or

Action

to support aEvidence

which may be understood as

Findingsresults in

Data

which when applied toDesign and

Analysis

which decides the

QuestionOne expresses what they care

about in the form of a

15

Four Components of a Good Research Question:

1. Make sure it’s a question.2. It’s narrow and specific.3. The answer can be seen.4. It’s appropriate for the intervention: Aligned with the theory of change;

Matches its current phase of development.

KNOW YOUR QUEST ION

Concept 3:

Think Longitudinally

17

Two Aspects to Thinking Longitudinally:

1. Time itself is a factor that can impact outcomes.2. The impact of time can be analyzed using longitudinal

data.

THINK LONGITUDINALLY

18

Time Itself Is a Factor That Can Impact Outcomes

Time-dependent means the amount of time that has passed influences whether or not something happens.

Examples of time-dependent outcomes are: Placement moves while in foster care;

Likelihood of reentry after reunification.


19


Longitudinal data collects information more than once about the same group of people over a period of time.

Measuring something year to year, but from different people, is not longitudinal data.

The Impact of Time Can Be Analyzed Using Longitudinal Data

Time Start Year 1 Year 2 Year 3

Annual Measures

Longitudinal Data

20

Longitudinal Data Can Answer Questions about Whether and When Something Happened

Did our program reduce the time to reunification? Did our program reduce the likelihood of abuse? Did our program increase placement stability?

What are some examples of longitudinal questions from your evaluation projects?


Concept 4:

Know Your Population

22

Your Theory of Change Will Guide the Selection of Your Target Population The target population is the group of people who the

intervention is created for. The study population is the group of people from the

target population who will be in your evaluation. A sample is when the study population includes some, but

not all, of the target population.

KNOW YOUR POPULAT ION

23

Three Common Ways to Get Your Study Population in Child Welfare1. Point-in-time: only people in the program at a specific time,

or the active caseload.

2. Entry cohort: all people who entered the program during a period of time.

3. Exit cohort: only people who left the program over a period of time.

These are not “three views of the same population.”

They are three very different populations.


24


Who gets counted Who is ignored

Point-in-time Only those in the program that day Everyone who already left the program

Exit cohort Only those who exited during the period

Everyone who is still in the program and everyone who left early

Entry cohort Everyone starting program during the period

Nobody

Only Entry Cohorts Include Everyone You Intend to Treat

25


Let’s Practice! What Sample Was Used for Each Analysis? 1. 60% of parents who completed the program were reunified with

their children.2. Of those assigned to the treatment group, 15% had an overdose

one year later compared with 25% of the comparison group.

Exit

Entry

3. More children were living with family in April 2020 compared with April 2019, the month before family meetings were implemented.

Point-in-time

4. At one-year follow-up, young people from the treatment group were more likely to have completed school than those from the comparison group.

Entry

5. On the last day of the fiscal year, 70% of young people were housed.

Point-in-time

6. Parents who graduated from the program reported less use of corporal punishment six months later.

Exit

26

OVERVIEW

Review: The Four Key Evaluation Concepts You Will Use Throughout This Course Are:Concept 1: Have a clear theory of change.

Concept 2: Ask a good question.

Concept 3: Think longitudinally.

Concept 4: Know your population.

Questions?

This report is in the public domain. Permission to reproduce is not necessary. Suggested citation: Urban Institute et al. (2021). Slide Deck Session 2: Key Concepts - Child Welfare Evidence-Building Academy. OPRE Report 2021-108, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

july 20, 2020 key concepts

Documents