W H A T ’ S A L L T H E T A L K A B O U T D A T A -D R I V E N D E C I S I O N - M A K I N G ?
A Practical Guide to Bridge Data Users and Producers
Patricia Langan & Andrea Lozano
Save the Children US
Arief Firdaus
Save the Children Indonesia
September 22, 2015
2
Skills to Succeed
Agenda
1. Rationale2. The Structured Experiential Learning (SEL)
approach3. Case study4. Applying SEL in Indonesia5. Conclusions6. Questions & discussion
3
Skills to Succeed
Today’s Objectives
Share the SEL approach Present SEL in a real-world context Gather feedback Identify ways to improve the SEL
approach
4
Skills to Succeed
Agenda
1. Rationale2. The SEL approach3. Case study4. Applying SEL in
Indonesia5. Conclusions6. Questions &
discussion
5
av
Skills to SucceedMonitoring Data Gets No Respect!
Monitoring data is only useful for accountability Data only shows the problem Goals are fixed so I can’t change the workplan Let’s wait to learn when the project is over
But needs to think this: There’s more ways to use monitoring data, e.g. to find root causes and
solutions Data can help chart new course toward fixed goal Learning at project end is too late – use data to learn and adapt now
And do this: Create processes and habits for program and institutional learning through
systemic approaches to experimentation – the SEL process
The typical S2S program manager thinks this:
6
av
Skills to SucceedStructured Experiential Learning (SEL)
Note: Based on “Adaptive Design” (WB 2015), “MeE” (Pritchet et al) Behavioral economics (ideas42), among others
Structured Experienti
al Learning
The collection and use of monitoring data across alternative project designs to provide real-time performance information with direct feedback into design and implementation.
Goal Generate and use evidence for internal and external learning
7
Skills to Succeed
Agenda
1. Rationale2. The SEL approach3. Case study4. Applying SEL in
Indonesia5. Conclusions6. Questions &
discussion
8
av
Skills to Succeed
The SEL Approach
DEFINEIdentify the
problem
DIAGNOSEIdentify root causes of the
problem
DESIGNDesign and
pilot potential solutions
TESTCompare different
approaches and select best
solution
9
av
Skills to Succeed
The SEL Approach
DEFINEIdentify the
problem
DIAGNOSEIdentify
potential root causes of the
problem
DESIGNDesign and
pilot potential solutions
TESTCompare different
approaches and select best
solutionDefine a specific, actionable problem of programmatic relevance
10
av
Skills to Succeed
The SEL Approach
DEFINEIdentify the
problem
DIAGNOSEIdentify
potential root causes of the
problem
DESIGNDesign and
pilot potential solutions
TESTCompare different
approaches and select best
solution▪ Gather internal and external data▪ Transform data into information using visual graphics▪ Brainstorm root causes of the problem with various
stakeholders
11
av
Skills to Succeed
The SEL Approach
DEFINEIdentify the
problem
DIAGNOSEIdentify
potential root causes of the
problem
DESIGNDesign and
pilot potential solutions
TESTCompare different
approaches and select best
solution▪ Brainstorm potential solutions for each root cause▪ Select the 5 most relevant & less costly potential solutions▪ Design time-bound, area-bound tests for 2-3 potential
solutions
12
av
Skills to Succeed
The SEL Approach
DEFINEIdentify the
problem
DIAGNOSEIdentify
potential root causes of the
problem
DESIGNDesign and
pilot potential solutions
TESTCompare different
approaches and select best
solution▪ Select performance variables to gauge effectiveness▪ Track performance variables for each test▪ Select the best solution based on pre-specified criteria
13
Skills to Succeed
Agenda
1. Rationale2. The SEL approach3. Case study4. Applying SEL in
Indonesia5. Conclusions6. Questions &
discussion
14
av
Skills to SucceedCase Study: applying SEL in a WFD program
STEM: science, technology, engineering, and mathematics
Labor market
assessment
High demand
for STEM jobs
Vocational
training
Computer programmi
ng
3D printing
Only 1 out of 6
participants enrolled in STEM are
girls
After a few months……
15
av
Skills to SucceedStep 1: Define the problem
Example 1The program is not popular with girls
Which problem statement is narrow enough and actionable?
Example 3Too few girls enroll in STEM courses to meet the project targets
Example 2
Gender disparity in primary education limits girls ability to enroll in technical courses later in life
16
av
Skills to Succeed
Step 2: Diagnose (1/4)
1. Review routine monitoring data2. Gather secondary data 3. Data visualization 4. Data assessment 5. Identify key stakeholders6. Lead brainstorming session
Gather data to better understand the root causes of the problem
17
Skills to Succeed
Opportunity for iteration to re-define the problem
Step 2: Diagnose (2/4)
What does the data show?
Enrolled in employability skills training
Male 51.2%
Female 49.8%
Difference 1.4%
Average age
Male 18
Female 19
Difference 1
Average years of education
Male 9.4
Female 8.9
Difference 0.5
Enrolled in vocational training
Male 47.5%
Female 22.0%
Difference 25.5%
18
av
Skills to Succeed
Tips:
1. Fast-paced brainstorming sessions2. Key stakeholders from diversity of backgrounds 3. Always include youth4. Always stick to the time5. No idea is bad, the more the better at this stage6. Entire group decides top 5 root causes
Step 2: Diagnose (3/4)
Brainstorm root causes of the problem
19
av
Skills to Succeed
Step 2: Diagnose (4/4)
Root cause 1
Root cause 2
Root cause 3
Root cause 4
Root cause 5
STEM courses are not interesting to girls because they think it’s boring
Parents disapprove of girls working alongside boys
STEM courses are held late at night and girls feel unsafe traveling to the vocational school
Girls think that STEM workplaces are unsafe
Girls have a negative opinion of STEMs courses because they think they will be perceived as less feminine
20
av
Skills to Succeed
Step 3: Design (1/2)
Tips:
1. Each small group assigned a root cause and tasked to come up with as many potential solutions as they can
2. No idea is bad, the more the better at this stage3. Entire team chooses top 5 potential solutions
Opportunity for iteration to re-define the root causes
Brainstorm potential solutions
21
av
Skills to Succeed
Step 3: Design (2/2)
Potential solution to cause 1
Potential solution to cause 2
Potential solution for cause 3
Potential solution for cause 4
Potential solution for cause 5
Cause: STEM courses are not interesting to girls because they think it’s boring
Solution: Work with employers and vocational schools to develop a STEM campaign focused on girls.
Identify employers who offer gender-separate workspaces and invite them to speak at trainings and job linkage events where youth and their parents are present.
Work with vocational schools to revise the scheduling program for their training offerings.
Work with employers to develop workplace improvement plans and share this information at trainings and job linkage events at the school.
Cause: Girls have negative opinion of STEM
Solution: Identify women who work in STEM and convince them to become mentors for young girls in the program.
22
av
Skills to Succeed
Step 4: Test (1/5)
Potential solution for cause 1
Potential solution for cause 2
Potential solution for cause 3
Potential solution for cause 4
Potential solution for cause 5
Work with employers and vocational schools to develop a STEM campaign focused on girls.
Identify employers who offer gender-separate workspaces and invite them to speak at trainings and job linkage events where youth and their parents are present. – TEST A
Work with vocational schools to revise the scheduling program for their training offerings.
Work with employers to develop workplace improvement plans and share this information at trainings and job linkage events at the school.TEST B
Identify women who work in STEM and convince them to become mentors for young girls in the program.
TEST C
23
av
Skills to Succeed
Step 4: Test (2/5)
Criteria to choose
potential solutions to
test
Criteria to choose
performance indicators
How to compare solutions
▪ Feasible to implement in short timeframe
▪ Cost-effective
▪ Feasible to implement alongside regular program operations
▪ Can move in short timeframe
▪ Feasible to collect & analyze
▪ Relevant
▪ Cross-cutting
▪ Team decision
▪ Are all performance indicators equally important to you?
24
av
Skills to Succeed
They chose to measure 3 performance indicators that fulfilled criteria:
Step 4: Test (3/5)
How to compare solutions?
Valuable
Safe
Appealing
Girls’ perception of how valuable STEM courses are
Girls’ perceptions of how safe STEM jobs are
Girls’ perceptions of how appealing they find STEM jobs
25
av
Skills to Succeed
Step 4: Test (4/5)
Performance indicator
#1 #2 #3
What is performance indicator?
Value Safety Appeal
How will you track performance variable across all test groups and the business as usual group?
Call participants and ask them to answer survey
26
av
Skills to Succeed
Step 4: Test (5/5)
27
Skills to Succeed
Agenda
1. Rationale2. The SEL approach3. Case study4. Applying SEL in
Indonesia5. Conclusions6. Questions & discussion
28
av
Skills to Succeed
Applying SEL in Indonesia
Background
Literacy
Boost, Youth Employabilit
y Skills
Similar interventions across country
Some interventions focus on specific approaches
Need
▪ Build data culture
▪ Ongoing learning culture
▪ Advance research methodology
29
av
Skills to SucceedOur progress in Indonesia (1/4)
Develop criteriaIdentify promising interventions
Identify resources & opportunitiesPave the wayStep 1
Step 2
Step 3
Step 4
30
Skills to Succeed
Results from SEL learning workshop
Our progress in Indonesia (2/4)
No significant differences between intervention and control sites at project x
Problem
Narrowed Statement
Not enough teachers are applying improved teaching method
Initial discussions led to too broad causes of the problem
Iteration to redefine problem statement
31
Skills to Succeed
To be continued…
Our progress in Indonesia (3/4)
Narrowed Statement
Not enough teachers are applying improved teaching method
Results from SEL learning workshop
Root cause 1
Root cause 1
Lack of ongoing support for teachers to implement improved teaching methodNo clear benefit for teachers to adopt improved teaching method
32
Skills to Succeed
Despite this is complex question, the process allowed staff to:
1. Be critical of how they present an issue and reflect back on how they can narrow the issue to an actionable, relevant problem
2. Use data to support/contrast their statement
3. Gain understanding on SEL along the way
Results from SEL learning workshop
Our progress in Indonesia (4/4)
Critical
Data
Understanding
33
Skills to Succeed
Agenda
1. Rationale2. The SEL approach3. Case study4. Applying SEL in Indonesia5. Conclusions6. Questions & discussion
34
av
Skills to Succeed
Conclusions
1. SEL is a 4-step process2. Narrow problem statement3. Brainstorm root causes and potential
solutions4. Choose potential solutions to test and
indicators to compare performance5. Gather data and identify best solution6. Rollout best solution to all the program
35
av
Skills to Succeed
Limitations to SEL
1. Work in progress2. Most relevant for projects with quality
routine monitoring systems3. Most relevant for projects with longer
timeframe that can allow for learning and adaptation
4. Suited for narrow, operational questions5. Donors openness’ to mid-course
corrections
Word of caution
36
Skills to Succeed
Agenda
1. Rationale2. The SEL approach3. Case study4. Applying SEL in
Indonesia5. Conclusions6. Questions &
discussion
37
Skills to Succeed
Thank you!
Patricia Langan, Program Director, Skills to Succeed
Andrea Lozano, Senior Specialist, Research and M&E, Skills to Succeed
Arief Firdaus, National MEAL Specialist, Save the Children Indonesia
For questions/comments, please contact Andrea at [email protected]