combining utilization focused and developmental evaluation: … · 2014. 3. 26. · • utilization...

Combining Utilization Focused and Developmental Evaluation: The Case

of the Aflateen Social and Financial Education Program for Youth

Daniel Shephard & Simon Bailey

Aflatoun, Child Social & Financial Education

The Netherlands

Background: Context

• Aflatoun

• Provides Social and Financial Education to Other Organizations (Social Franchise)

• History

• Started as a small education programme in Mumbai in early 90s

• Secretariat for Global Expansion started in Amsterdam in 2005

• Focus on primary education

• Has partners in over 100 countries and work with over 2 million children per year

• Aflateen

• New curriculum and first experience with age group

• Five year partnership with MasterCard Foundation to roll out to 50 countries

• Period of curriculum development, then piloting, then re development, then roll out

• Mid Term Evaluation

• MC requested a mid term evaluation, not part of original contract

• Provided opportunity for Secretariat to co design evaluation

• Secretariat Perspective

• Wanted approach that reflected the stage of programme we were in (re development)

• Interested in how the process of supporting organizations had worked

• Settled on utilization focused evaluation as a potential option

1

Background: Process and Level of Assessment

• Selected Partnership for Evaluation • External

• Partnership of Utilization Focused External Evaluation team of two

• Internal

• Research Manager

• Partners & Consultants from 8 countries

• Secretariat level: • Assessment of the Aflateen curriculum development and pilot process to help the

Secretariat review and improve its rollout and support of the program

• 3 visits by Evaluator(s)

• (Co)writing

• 2 field visits by evaluators (focus groups with country/regional partners)*

• Surveys and Phone Interviews

• Implementation level (not covered in presentation) • Generating lessons for current and future partners.

• 2 field visits by evaluators (focus groups with country/regional partners)*

• 8 commissioned research projects (2 to 5K projects)

*This element provided data on both the secretariat functioning and program implementation

2

Background: Methodology Combination

• Utilization focused evaluation – Focus on Users, uses, key evaluation questions, and

facilitating use

• Developmental evaluation – Documenting a model in the making and learning from it

• UDFE instead of UFE

3

Simplified Model of Secretariat Social Franchise

4

Brainstorming & Clustering

5

Intended Uses: Course Correction and Improvement

Secretariat (By Topic)

1. The curriculum development process

• Define our desired curriculum development process for the future

2. E-learning

• Create a plan to scale up and broaden e-learning.

3. Partners / dissemination

• Align Aflateen partner support practices with strategic goals

4. Training

• Differentiate training process for Aflateen vs. Aflatoun as needed

5. Evaluation

• A methodology for research calls that improves their program

6

What was the Model Used?

7

Intended Users

Primary Intended USERS

• Curriculum Manager* and Programme Head

• Specialized Staff, Department Head, and Research manager*

• Programme Head, 2 Programme Staff, and Research Manager*

• Training Manager, 2 Programme Staff and Research Manager*

• Department Head, and Research manager*

Other USERS

• Programme Staff

• Are what we doing effective? How to improve?

• Management

• How do we manage programme expansion? How do we change? How do we

secure more funding?

• MasterCard Foundation

• Is this approach applicable to our portfolio of grantees?

*This is the same staff member

8

Actual Users

Primary Intended Users

• Not all primary Users available or full participants, traveling challenges

• Research Manager often mediator or primary participant

• Time pressure reduced input beyond research team

• Evaluators did think contribution was better than most cases!

Programme Staff

• Importance of in person discussions

• Team contributed (even beyond primary Users) and were affected

• Data collection and use process merged

• Often brought out and document tacit processes, perceptions, and practices

Management

• Process Ownership

• Reputation Management

• Follow Up on Recommendations

Mastercard

• Learning about methodology

• Interest in how organization used the approach and changed

9

Finalization of Uses: Process

• External researcher provided recommendation domains based on findings

which were then adapted, replaced or expanded by the Users.

• The final recommendations for use were drafted by Users prior to a final visit by

the researcher

• Recommendations were finalized during the last visit with involvement of the

management and full staff.

• Management determined much of the wording of the final recommendations.

• Donor asked for formal timelines for implementation

10

Finalization of Uses:

Curriculum Development

• Develop a Secretariat strategy to support country-level curriculum integration, including

national events and a guide for partners

• Modify and simplify the curriculum Contextualization Manual through the creation of simple

checklists, a list of most frequent adaptations, and FAQs, to increase use

• Document the process for the development of new curriculum based on experience from

Aflateen, Aflatoun and Aflatot as part of the Quality Assurance process

E-learning

• Disseminate Aflateen Digital learning to existing Aflateen partners

• Disseminate Aflateen Digital learning to youth organizations and other stakeholders who

are not currently partners

• Identify ‘Aflateen Digital Champions’ from amongst the youth who are actively using and

promoting the platform

• Optimize the platform on an ongoing basis via usability testing

Programme Support

• Strengthen national networks and partners that function as country level secretariats for

scale-up of Aflateen

• Strengthen direct linkages between partners regionally and globally by leveraging partner

skills and experience more strategically

• Ensure that Aflateen partners get more differentiated support and Aflateen specific partner

communication

11

Cont’d

Training

• Update the Aflateen training to strengthen content related to facilitation skills for training

youth and then conduct new regional master trainings in 2014

• Expand and strengthen the pool of Aflateen specific trainers

• Revise monitoring and evaluation tools for both Secretariat and Partner led trainings to

facilitate learning

Evaluation (Changed question)

• To prepare and implement a research plan that privileges proof of concept, scaling and

performance while linking partners to external capacity development opportunities

• Complete the creation of the Implementation & Monitoring Guide for monitoring

programme numbers. This should also capture variability of programme implementation

• Produce user-friendly guides and integrate them into programmatic support for

conducting research of various types

12

Actual Implementation of Uses

General Findings

• Some automatic or based on past understanding of issues

• Some instigated via the process itself

• Some came late in process after dealing with new information or having it presented to us

by the evaluator in challenging ways

Process

• Responsibility for driving the process shifted to Management level from research level

• Transition from recommendations with clear timelines to non-timelines in the final report

affected the actualization of recommendations

• To date, more likely to be completed if started in the evaluation or shortly thereafter than if

it was to be started at some time in the future

13

Contributing Factors to Use

Supporting

• Willingness of funder to consider learning as end goal of evaluation

• Allow Secretariat to define its own key questions

• Ability of Lead Evaluator to communicate with both team and funder (in person)

• Dedicated time for reflection for staff to consider what they were doing as a group

• Facilitated time as well as internal discussions

• Length of time allowed for continued progress and incorporation of findings

• Ability to modify evaluation questions to use (ex. evaluation) while maintaining evaluation’s

focus on uses

• Ability to have internal evaluation staff as part of process helped mediate concerns on

both sides and keep process going

Challenging

• Balancing demands of outcome evaluation with aim of UFD evaluation

• Transition from evaluation to use after end of report

• Key responsibilities had moved within the organization

• Keeping focus on the programme vs expansion to the whole organization

Point of Discussion

• Developmental approach as a key requirement for Utilization Focused Evaluation to

achieve change?

14

Aflatoun, Child Social & Financial Education

Stichting Child Savings International | Po Box 15991 | Spaklerweg 14 | 1096 BA Amsterdam

The Netherlands

P: +31 20 760 1340 | F: +31 20 626 2118 | www.Aflatoun.org

Thanks…. Questions, Clarifications, Discussions

For more information on all Aflatoun evaluation work visit

www.aflatoun.org/evaluation