what’s all the talk about data- driven decision-making? a practical guide to bridge data users and...

W H A T ’ S A L L T H E T A L K A B O U T D A T A -D R I V E N D E C I S I O N - M A K I N G ?

A Practical Guide to Bridge Data Users and Producers

Patricia Langan & Andrea Lozano

Save the Children US

Arief Firdaus

Save the Children Indonesia

September 22, 2015

2

Skills to Succeed

Agenda

1. Rationale2. The Structured Experiential Learning (SEL)

approach3. Case study4. Applying SEL in Indonesia5. Conclusions6. Questions & discussion

3

Skills to Succeed

Today’s Objectives

Share the SEL approach Present SEL in a real-world context Gather feedback Identify ways to improve the SEL

approach

4

Skills to Succeed

Agenda

1. Rationale2. The SEL approach3. Case study4. Applying SEL in

Indonesia5. Conclusions6. Questions &

discussion

5

av

Skills to SucceedMonitoring Data Gets No Respect!

Monitoring data is only useful for accountability Data only shows the problem Goals are fixed so I can’t change the workplan Let’s wait to learn when the project is over

But needs to think this: There’s more ways to use monitoring data, e.g. to find root causes and

solutions Data can help chart new course toward fixed goal Learning at project end is too late – use data to learn and adapt now

And do this: Create processes and habits for program and institutional learning through

systemic approaches to experimentation – the SEL process

The typical S2S program manager thinks this:

6

av

Skills to SucceedStructured Experiential Learning (SEL)

Note: Based on “Adaptive Design” (WB 2015), “MeE” (Pritchet et al) Behavioral economics (ideas42), among others

Structured Experienti

al Learning

The collection and use of monitoring data across alternative project designs to provide real-time performance information with direct feedback into design and implementation.

Goal Generate and use evidence for internal and external learning

7

Skills to Succeed

Agenda



discussion

8

av

Skills to Succeed

The SEL Approach

DEFINEIdentify the

problem

DIAGNOSEIdentify root causes of the

problem

DESIGNDesign and

pilot potential solutions

TESTCompare different

approaches and select best

solution

9

av

Skills to Succeed

The SEL Approach

DEFINEIdentify the

problem

DIAGNOSEIdentify

potential root causes of the

problem

DESIGNDesign and




solutionDefine a specific, actionable problem of programmatic relevance

10

av

Skills to Succeed

The SEL Approach

DEFINEIdentify the

problem

DIAGNOSEIdentify


problem

DESIGNDesign and




solution▪ Gather internal and external data▪ Transform data into information using visual graphics▪ Brainstorm root causes of the problem with various

stakeholders

11

av

Skills to Succeed

The SEL Approach

DEFINEIdentify the

problem

DIAGNOSEIdentify


problem

DESIGNDesign and




solution▪ Brainstorm potential solutions for each root cause▪ Select the 5 most relevant & less costly potential solutions▪ Design time-bound, area-bound tests for 2-3 potential

solutions

12

av

Skills to Succeed

The SEL Approach

DEFINEIdentify the

problem

DIAGNOSEIdentify


problem

DESIGNDesign and




solution▪ Select performance variables to gauge effectiveness▪ Track performance variables for each test▪ Select the best solution based on pre-specified criteria

13

Skills to Succeed

Agenda



discussion

14

av

Skills to SucceedCase Study: applying SEL in a WFD program

STEM: science, technology, engineering, and mathematics

Labor market

assessment

High demand

for STEM jobs

Vocational

training

Computer programmi

ng

3D printing

Only 1 out of 6

participants enrolled in STEM are

girls

After a few months……

15

av

Skills to SucceedStep 1: Define the problem

Example 1The program is not popular with girls

Which problem statement is narrow enough and actionable?

Example 3Too few girls enroll in STEM courses to meet the project targets

Example 2

Gender disparity in primary education limits girls ability to enroll in technical courses later in life

16

av

Skills to Succeed

Step 2: Diagnose (1/4)

1. Review routine monitoring data2. Gather secondary data 3. Data visualization 4. Data assessment 5. Identify key stakeholders6. Lead brainstorming session

Gather data to better understand the root causes of the problem

17

Skills to Succeed

Opportunity for iteration to re-define the problem


What does the data show?

Enrolled in employability skills training

Male 51.2%

Female 49.8%

Difference 1.4%

Average age

Male 18

Female 19

Difference 1

Average years of education

Male 9.4

Female 8.9

Difference 0.5

Enrolled in vocational training

Male 47.5%

Female 22.0%

Difference 25.5%

18

av

Skills to Succeed

Tips:

1. Fast-paced brainstorming sessions2. Key stakeholders from diversity of backgrounds 3. Always include youth4. Always stick to the time5. No idea is bad, the more the better at this stage6. Entire group decides top 5 root causes


Brainstorm root causes of the problem

19

av

Skills to Succeed


Root cause 1

Root cause 2

Root cause 3

Root cause 4

Root cause 5

STEM courses are not interesting to girls because they think it’s boring

Parents disapprove of girls working alongside boys

STEM courses are held late at night and girls feel unsafe traveling to the vocational school

Girls think that STEM workplaces are unsafe

Girls have a negative opinion of STEMs courses because they think they will be perceived as less feminine

20

av

Skills to Succeed

Step 3: Design (1/2)

Tips:

1. Each small group assigned a root cause and tasked to come up with as many potential solutions as they can

2. No idea is bad, the more the better at this stage3. Entire team chooses top 5 potential solutions

Opportunity for iteration to re-define the root causes

Brainstorm potential solutions

21

av

Skills to Succeed

Step 3: Design (2/2)

Potential solution to cause 1

Potential solution to cause 2

Potential solution for cause 3



Cause: STEM courses are not interesting to girls because they think it’s boring

Solution: Work with employers and vocational schools to develop a STEM campaign focused on girls.

Identify employers who offer gender-separate workspaces and invite them to speak at trainings and job linkage events where youth and their parents are present.

Work with vocational schools to revise the scheduling program for their training offerings.

Work with employers to develop workplace improvement plans and share this information at trainings and job linkage events at the school.

Cause: Girls have negative opinion of STEM

Solution: Identify women who work in STEM and convince them to become mentors for young girls in the program.

22

av

Skills to Succeed

Step 4: Test (1/5)






Work with employers and vocational schools to develop a STEM campaign focused on girls.

Identify employers who offer gender-separate workspaces and invite them to speak at trainings and job linkage events where youth and their parents are present. – TEST A

Work with vocational schools to revise the scheduling program for their training offerings.

Work with employers to develop workplace improvement plans and share this information at trainings and job linkage events at the school.TEST B

Identify women who work in STEM and convince them to become mentors for young girls in the program.

TEST C

23

av

Skills to Succeed

Step 4: Test (2/5)

Criteria to choose

potential solutions to

test

Criteria to choose

performance indicators

How to compare solutions

▪ Feasible to implement in short timeframe

▪ Cost-effective

▪ Feasible to implement alongside regular program operations

▪ Can move in short timeframe

▪ Feasible to collect & analyze

▪ Relevant

▪ Cross-cutting

▪ Team decision

▪ Are all performance indicators equally important to you?

24

av

Skills to Succeed

They chose to measure 3 performance indicators that fulfilled criteria:

Step 4: Test (3/5)

How to compare solutions?

Valuable

Safe

Appealing

Girls’ perception of how valuable STEM courses are

Girls’ perceptions of how safe STEM jobs are

Girls’ perceptions of how appealing they find STEM jobs

25

av

Skills to Succeed

Step 4: Test (4/5)

Performance indicator

#1 #2 #3

What is performance indicator?

Value Safety Appeal

How will you track performance variable across all test groups and the business as usual group?

Call participants and ask them to answer survey

26

av

Skills to Succeed

Step 4: Test (5/5)

27

Skills to Succeed

Agenda


Indonesia5. Conclusions6. Questions & discussion

28

av

Skills to Succeed

Applying SEL in Indonesia

Background

Literacy

Boost, Youth Employabilit

y Skills

Similar interventions across country

Some interventions focus on specific approaches

Need

▪ Build data culture

▪ Ongoing learning culture

▪ Advance research methodology

29

av

Skills to SucceedOur progress in Indonesia (1/4)

Develop criteriaIdentify promising interventions

Identify resources & opportunitiesPave the wayStep 1

Step 2

Step 3

Step 4

30

Skills to Succeed

Results from SEL learning workshop

Our progress in Indonesia (2/4)

No significant differences between intervention and control sites at project x

Problem

Narrowed Statement

Not enough teachers are applying improved teaching method

Initial discussions led to too broad causes of the problem

Iteration to redefine problem statement

31

Skills to Succeed

To be continued…


Narrowed Statement

Not enough teachers are applying improved teaching method


Root cause 1

Root cause 1

Lack of ongoing support for teachers to implement improved teaching methodNo clear benefit for teachers to adopt improved teaching method

32

Skills to Succeed

Despite this is complex question, the process allowed staff to:

1. Be critical of how they present an issue and reflect back on how they can narrow the issue to an actionable, relevant problem

2. Use data to support/contrast their statement

3. Gain understanding on SEL along the way



Critical

Data

Understanding

33

Skills to Succeed

Agenda

1. Rationale2. The SEL approach3. Case study4. Applying SEL in Indonesia5. Conclusions6. Questions & discussion

34

av

Skills to Succeed

Conclusions

1. SEL is a 4-step process2. Narrow problem statement3. Brainstorm root causes and potential

solutions4. Choose potential solutions to test and

indicators to compare performance5. Gather data and identify best solution6. Rollout best solution to all the program

35

av

Skills to Succeed

Limitations to SEL

1. Work in progress2. Most relevant for projects with quality

routine monitoring systems3. Most relevant for projects with longer

timeframe that can allow for learning and adaptation

4. Suited for narrow, operational questions5. Donors openness’ to mid-course

corrections

Word of caution

36

Skills to Succeed

Agenda



discussion

37

Skills to Succeed

Thank you!

Patricia Langan, Program Director, Skills to Succeed

Andrea Lozano, Senior Specialist, Research and M&E, Skills to Succeed

Arief Firdaus, National MEAL Specialist, Save the Children Indonesia

For questions/comments, please contact Andrea at [email protected]

mailto:[email protected]

what’s all the talk about data- driven decision-making? a practical guide to bridge data users and...

Documents

sel approach present

late use data

succeed4monitoring data

external learning

bridge data users

project end

datadriven decisionmaking

alternative project