summary annual progress reportpdf.usaid.gov/pdf_docs/pbaaf528.pdf · summary annual progress report...

AID

EdData II

Task Order 15: Data for Education Programming in Asia and the Middle East (DEP-AME)

Summary Annual Progress Report October 1, 2013- September 30, 2014

REDACTED VERSION

Education Data for Decision Making (EdData II) Technical and Managerial Assistance Contract Number AID-EHC-E-00-04-00004 Task Order Number AID-OAA-BC-11-00001, RTI Task 15 01 October 2013 - 30 September 2014

This report was produced for review by the United States Agency for International Development. It was prepared by RTI International.

Task Order 15: Data for Education Programming in Asia and the Middle East (DEP-AME)

Summary Annual Progress Report October 1, 2013 - September 30, 2014

EdData II Technical and Managerial Assistance, Task 15 EdData II Contract Number EHC-E-00-04-00004-00 Task Order Number AI D-OAA-BC-1 1-00001 RTI Project No. 09354.015

Prepared for Mitch Kirby USAID/Asia Bureau, Senior Education Advisor 1300 Pennsylvania Ave , NW Washington , DC 20523

Prepared by Michelle Ward-Brent RTI International* 3040 E. Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2 194

RTI International is one of the worlds 1ead1ng 1·esearch mstltutes dedicated to 1mprovmg the human cond1tior by turnmg knowledge mto pract1ce Our staff of more than 2.800 pro 'Ides research and techn1cal expert1se to governments and bus nesses 111 more than 40 countries 111 the areas of health and pharmaceuticals. education and tra1nmg. surveys and statistics. advanced technology, international developMent, econom1c and social pol1cy. energy and the environment. and laboratory and chemistry services. For more mformat1on 'is1t www. rt1.org

RTI lnter1ational is a trade name of Research Tr,angle Institute.

The authors views expressed 111 th1s publ1cat1on do not necessanly reflect the 11ews of the United States Agency for International Development or the Un1ted States Governrr1ent.

EdData II Task Order 15 (OEP-AME)

Abbreviations Allegro

AME

CLA

CMS

DEP-ASIA/ME

DEO

DepED

DD

DM

DP

ECD

Ed-Data

EMES

EGR

EGRA

ETC

FY

GBAO

GOB

IPD

LOI

MENA

MOE

MOE HE

MTB-MLE

NA

PA

PT

RC

RTI

SES

SMC

sow TBD

TOS

US AID

Allegro Global Procurement Solutions, Inc.

Asia/Middle East

central level agencies

content management system

Data for Education Programming in Asia and Middle East

district education officers

Department of Education of the Philippines

Deputy Director

Deputy Minister

development partner

early childhood development

Education-Data

Education Management Efficiency Study

early grade reading

early grade reading assessment

education training center

Fiscal Year

Gorno-Badakshan Autonomous Province

Government of Bangladesh

Institute of Professional Development

language of instruction

Middle EastiN orth Africa

Ministry of Education

Ministry of Education and Higher Education

Mother Tongue Based-Multilingual Education

not applicable

Palestinian Authority

parent-teacher association

research centers

Research Triangle International

socio-economic status

school management committee

Scope of Work

to be determined

Teacher Observation Study

U.S. Agency for International Development

Annual Reporl Year 3 (01 October 2013 - 30 September 2014) iii

EdData II Task Order 15 (DEP-AME)

Table of Contents Page

Abbreviations .... ............... ............... ... .. .... .. ........ .. ...................................... .......... ii i

Executive Summary .... ..... ................ .... .... ........................................ ..................... 1

Programming and Project Support ... .. .. .... ... ...................... ............................ .... .... 1

Activity Status ................................... ........ ......................... ............................ ........ 2

Report Summary .............................. .................. .. ............. .................................... 2

Operational Activities ........................ ................................. ............................ .. ...... 6

USAI D Communication and Meetings ....................... ... ..... ... ................................. 6

Reports and Deliverables ................ ......... .......... .. ............. .................................... 6

Overall Progress: Results, Requirements, Standards ........ .............. ........ .............. 8

Progress toward Objectives (by Result) ... .... ....... .............. .................................. 14

Result 1: AME Mission Strategy-Related Data Needs Met ............................ 14

Result 2: Availability of AME Education Data and Trends Expanded ............. 29

Result 3: Measurement Tools fo r Cross-Country Use ............................ ........ 32

Result 4: US AID Staff Knowledge Strengthened ............................................ 44

Annex A: Financial Summary ......... ................................................................... A-1

iv EdOata/1 TO 15: DEP-AME Annual Report Year 3 (01 October 2013- 30 September 2014)


Executive Summary

Task Order 15 (TO 15), Data for Education Programming in Asia and Middle East (DEPASIA/ME), is a centrally-funded Time and Materials Task Order under the Ed-Data II Indefmite Quantity Contract (IQC) between the U.S. Agency for International Development (USAID) and RTI International, with support from Allegro Global Procurement Solutions, Inc. (Allegro ). 1 The contract was awarded with a Task Order ceiling price of [redacted]. The project began on September 26, 2011. A September 2014 contract modification raised the level of obligated funding from [redacted] to [redacted]. The contractual end date is presently November 30, 2014. A no-cost extension request has been submitted to enable DEP-AME to complete the activities begun and within the scope of work. Labor categories use fixed daily rates.

Support activities under the Task Order are intended to provide technical assistance and training services in direct support ofUSAID's 2011 Education Strategy in Asia and the Middle East, in 17 Missions implementing or planning basic education programs, including Afghanistan and Pakistan. Specific focus is placed on ensuring that education data needs are met in the region. It is designed to support Mission responses to USAID's 2011 Education Strategy, and its three specific goals:

• Goal 1: Itnproved reading skills for 100 million children in primary grades by 2015

• Goal2: Improved ability of tertiary and workforce development programs to generate workforce skills relevant to a country's development goals

• Goal 3: Increased equitable access to education in crisis and conflict environments for 15 million learners by 20 15

Programming and Project Support DEP-ASIA/ME is intended to generate regional and country-specific education dataand analysis of those data-that can be used by Asia and Middle East (AME) bureaus, missions, and partner countries to prioritize education needs and corresponding investments. The project is also intended to strengthen local skill in design, evaluation, and management of education progran1s, and quality data capture and analysis to support them, across the region. Activities have been designed for four core Result Areas.

• Result 1: AME Mission Strategy-Related Data Needs Met

• Result 2: Availability of AME Education Data and Trends Expanded

• Result 3: Measurement Tool with Applicability across Countries Developed

• Result 4: Education Personnel Knowledge of Selected Education and Evaluation Topics and Processes Strengthened

1 Contract number EHC-E-00-04-00004-00; Order Number AID-OAA-BC-11-0000.

Annual Report Year 3 (01 October 2013-30 September 2014)


Activity Status This report covers activities implemented in four Result areas across a 12 month period from October 1, 2013-September 30, 2014. Activity status updates are summarized in the table below.

Resu Its Activity Status

Result 1 Egypt: Grade 3 Early Grade Reading Assessment Baseline Completed

Result 1 Egypt: EGRA Baseline Comparison Electronic vs. Paper Completed

Result 1 Philippines: Analytic Support Services for Early Grade Reading Completed

Result 1 Nepal: EGRA, Teacher Observation Study, Management Effectiveness Studies Completed

Result 1 Bangladesh: Institutional Assessment of Early Grade Reading Underway

Result 1 West Bank: Early Grade Reading Assessment Underway

Result 1 Morocco: Reading Research Studies Completed

Result 1 Tajikistan (GBAO) Early Grade Reading Assessment Underway

Result 2 Topical Analysis: Scale-up and Sustainability Submitted

Result 2 Topical Analysis: Early Grade Reading Instruction in Arabic Completed

Result 2 Translation of Topical Analysis : Early Grade Reading Instruction in Arabic Completed

Result 3 Reading Improvement Modeling Tool (EGRA Barometer) Underway

Result 4 Mission Education Staff Training Modules on Impact Evaluation Completed

Result 4 Evaluation for Education Projects Workshop (Bangkok) Underway

Reports Submission of Yearly Report #11 Completed

Report Summary

Quarterly Activity Overview

Activities undertaken under each of the four Result Areas during Fiscal Year (FY) 2014 proceeded at a rapid pace this year with the largest volume of analytic analysis and technical assistance support occurring under Result 1. This included direct support to six USAID Mission programs. Several activities concluded this year under Result 1 and

2 Annual Report Year 3 (01 October 2013- 30 September 2014)


Result 2. Significant increases in technical support also occurred under Result 3 and Result 4.

First quarter activity included continuation of existing activities and the launch of several ones, including support for the Bangladesh: Institutional Assessment of Early Grade Reading. This field-based activity had been delayed several times due to security and scheduling considerations. On October 21, the Mission requested that the assessment team be fielded during November. On November 15, DEP-AME was informed that the November timeline would need to be postponed until January for security reasons. In response, DEP-AME conducted desk review work during the intervening months through December in preparation for eventual field mobilization in January, 2014.

Second quarter activity continued analytic support for activities in the West Bank, the Philippines, Nepal, Bangladesh, Morocco and Tajikistan. This included the field launch of the Bangladesh: Institutional Assessment of Early Grade Reading; planning and launch of the Morocco: Reading Research Studies; completion of survey data collection for the Nepal: EMES, TOS, and EGRA ; implementation of the Philippines: Analytic Support Services for Early Grade Reading; planning for the Tajikistan : Early Grade Reading Assessment in Tajikistan/GBAO; and implementation of the West Bank: Early Grade Reading Assessment implementation in Palestinian schools.

Third quarter technical assistance and analytic support further advanced these activities through field-based research, data analysis, and policy dialogue workshops. This included the revision and completion of the Bangladesh: Institutional Assessment of Early Grade Reading report draft based upon written feedback from USAID and the field Mission; field research and data collection for the Morocco: Reading Research Studies; completion of the report and conduct of a policy dialogue workshop in Katmandu with USAID, the MOE, and stakeholders at which the Nepal TOS, EMES, and EGRA studies' fmdings were shared; continued implementation under the Philippines Analytic Support Services for Early Grade Reading in preparation for an August Policy Dialogue Workshop; mobilization and field launch of the Tajikistan/GBAO Early Grade Reading Assessment in GBAO with meetings with USAID, the MOE, the Institute for Professional Development (IPD), M-Vector, implementers and donors, revision of the SOW and budget based upon fmdings , and planning for an August-September instrument adaptation, assessor training and data collection; and drafting of the West Bank: Early Grade Reading Assessment report and a Policy Dialogue workshop in Ramallah with USAID, MOE counterparts and stakeholders.

Fourth quarter activities included continued technical support to the USAID Missions in West Bank, Bangladesh, Tajikistan (Gorno-Badakshan [GBAO] region), Morocco and the Philippines for Early Grade Reading Assessments (EGRAs) and the completion of some activities. DEP-AME translated into Arabic the Topical Analysis: Early Grade Reading Instruction in Arabic, and continued robust activity on the EGRA Barometer pilot, including a focus on data integration, tracking barometer use, and design of country-level piloting. Design and refmement of the Evaluation for Education

Annual Report Year 3 (01 October 2013- 30 September 2014) 3


4

Programming training course for USAID Education Officers continued, including the identification of a replacement trainer due to scheduling conflicts with the original candidate.

Result Area Support by Quarter

• Quarter 1: October 1-December 31, 2013-DEP-AME activities during the first quarter included assistance to the USAID Missions in Bangladesh, Morocco, Nepal, the Philippines, Tajikistan (GBAO) and the West Bank, as well as analytic research and training development.

• Result 1 activity included the launch of the USAID/West Bank EGRA with launch meetings in Ramallah, and the completion of the scope of work for the Philippines: Analytic Support Services for Early Grade Reading. Data from the latter, designed to build local capacity on mother tongue early grade reading assessments, was presented at the November 2013 All Children Learning Workshop in Rabat. In Nepal, adaptation of the Education Management Efficiency Study (EMES) and Teacher Observation Study (TOS) instruments took place, along with training of the EMES/TOS assessors for data collection. Planning for the Bangladesh: Institutional Assessment of Early Grade Reading also launched.

• Result 2 activities supported research and drafting of a Topical Analysis: Early Grade Reading Instruction in Arabic. This too was presented at the All Children Learning Workshop in Rabat, Morocco. DEP-AME research supported preparation of a Topical Analysis on Scale-up and Sustainability.

• Result 3 activity included acceleration of the analytic design of the Beta version leading to the demonstration of the Reading Improvement Modeling Tool (EGRA Barometer). Planning for the launch of Phase 2 piloting of the EGRA Barometer occurred, along with integration of data from selected EGRA programs.

• Result 4, activity focused on the design, subcontracting, and planning for an anticipated February 2014 training workshop entitled Evaluation for Education Programming for USAID Education Officers. This was subsequently delayed until November 2014.

• Quarter 2: January 1-March 30, 2014-During the second quarter, multiple activities were underway, including field-based Early Grade Reading Assessments (EGRAs) and launches of institutional assessments and reading research studies.

• Result 1 activity continued support to the USAID Missions in West Bank, Philippines, and Nepal, as well as the addition of new activities in Bangladesh, Tajikistan (Gorno-Badakhshan Autonomous Province [GBAO] region) and Morocco.

Annual Report Year 3 (01 Odober 2013- 30 September 2014)


• Result 2 activity fmalized the Topical Analysis: Early Grade Reading Instruction in Arabic, complete with all Annexes, as well as a draft of the Topical Analysis: Scale up and Sustainability.

• Result 3 activity focused on the Beta version of the reading improvement modeling tool launch, the EGRA Barometer, and the refmement of a Phase 2 plan to pilot the tool in two countries, to integrate country EGRA data, and to continually refine functionality.

• Result 4 activities included dialogue with the subject matter expert on the Evaluation for Education Programming training workshop planned for mid-late 2014). Refmements to the Analytic Agenda and workshop design occurred.

• Quarter 3: Aprill-June 30, 2014-Activities during the third quarter included assistance, for analytic support, EGRA instrument adaptation, assessor training and data collection; and Policy Dialogue workshops.

• Result 1 activity support was provided to the US AID Missions in West Bank, Bangladesh, Tajikistan (Gorno-Badakshan [GBAO] region) , Morocco and the Philippines for EGRAs, continued, and completion of some activities.

• Result 2 activity completed the translation into Arabic of the Topical Analysis: Early Grade Reading Instruction in Arabic.

• Result 3 activity continued a strong emphasis on the EGRA Barometer pilot, including a focus on data integration, tracking barometer use, and design of country-level piloting.

• Result 4 activity focused on the continued design and refmement of the Evaluation for Education Programming training course for US AID Education Officers and the identification of a new subject matter expert due to scheduling conflicts with the original trainer.

Quarter 4: July !-September 30, 2014: During the fourth quarter of Fiscal Year 2014 activities largely focused on country-level support and piloting of the EGRA Barometer.

• Result 1 activity included support to the USAID Missions in West Bank, Bangladesh, Tajikistan (Gorno-Badakshan [GBAO] region), Morocco and the Philippines for Early Grade Reading Assessments (EGRAs), continued, and completion of some activities.

• Result 2 activity was limited this quarter.

• Result 3 activity focused on the EGRA Barometer pilot, including a focus on data integration, tracking barometer use, and design of country-level piloting.

• Result 4 activity focused on the continued design and refmement of the Evaluation for Education Programming training course for US AID Education Officers and the identification of a new subject matter expert due to scheduling conflicts with the original trainer.



Operational Activities No significant operational activities or changes have occurred.

USAID Communication and Meetings Communications and Meeting: Regular meetings, phone calls, and written communication occurred between RTI and USAID'S Senior Education Advisor Mitch Kirby and Education Development Officer Chris Capacci-Carneal, as well as with subcontractor Allegro Consulting. Social Impact, and varied consultants. DEP-AME staff also participated in meetings where the EGRA Barometer was presented and showcased, and in the November 2014 All Children Learning Workshop in Rabat, Morocco. These are referenced below.

Reports and Deliverables

6

Documentation and Reporting: The Fiscal Year 2014 Financial Summary is included at the end of this report. It is inclusive of the obligated amount to date; total expenditure, outstanding commitments, total expended to date, balance remaining of obligated funds, and the baling remaining of the total estimated cost. Final reports, proposals or deliverables submitted to US AID over the course of this year have been submitted previously under separate cover. They are listed below for reference.

Quarter 1-Reports and Deliverables:

• Financial Summary

• Result 1: GBAO Trip Report

• Result 1: DEP-AME Morocco Component 2

• Result 1: EGR Assessment Bangladesh Report

• Result 1: West Bank Grade 2 Results Presentation

• Result 2: Topical Analysis of Arabic Reading Instruction 4 April 2014

• Result 3: Early Grade Reading Barometer Usage Reports (April-June)

• Result 4: Impact Evaluation Training Annotated Agenda



• Result 1: Philippines: Analytic Support Services for Early Grade Reading

• Result 1: Nepal: EGRA, TOS, and EMES

• Result 1: Bangladesh: Reading/Education Sector Support/ Assessment

• Result 1: West Bank: Early Grade Reading Assessment

Annual Reporl Year 3 (01 October 2013-30 September 2014)


• Result 2: Draft 4 Topical Analysis: Scale-up and Sustainability

• Result 2: Topical Analysis: Early Grade Reading Instruction in Arabic



• Result 1: GBAO Trip Report and Presentation in Tajik

• Result 1: DEP-AME Morocco Component 2


• Result 1: Nepal AME Trip Report and EMES Report

• Result 1: West Bank Grade 2 Results Presentation (and Arabic)

• Result 1: Philippines Trip Report

• Result 2: Topical Analysis Early Grade Reading- Updated report

• Result 3: Early Grade Reading Barometer Usage Reports (April-June)

• Result 4: Castleman- USAID training Annotated Agenda



• Result 1: PhilED Data January Trip Report

• Result 1: West Bank EGRA Memorandum ofUnderstanding

• Result 1: GBAO Proposal


• Result 1: Nepal AME Trip Report (November, submitted January)

• Result 3: Early Grade Reading Barometer Overview

• Result 4: Impact Evaluation Training Annotated Agenda

The table of Overall Progress: Results, Requirements, Standards in the next section

serves as a complete summary ofDEP- AME activities planned, underway or completed since inception.

Annual Report Year 3 (01 Odober 2013-30 September 2014) 7


Philippines: Analytic Support Services for Early Grade Reading Training

Overall Progress: Results, Requirements, Standards

8

This table captures all DEP-AME activities and status to date at the close of this reporting period.

Result 1: AME Mission Strategy-Related Data Needs Met

Requirement 1.1: Determine Feasibility of Mission-Specific Requests (Data Support) Status

Nepal : Early Grade Reading Institutional Assessment Completed

Burma: Education Sector Desk Review Completed

MENA: Region Cost Comparisons of Early Grade Reading Completed

Egypt: Early Grade Reading Case Study Completed

Egypt: Grade 3 Early Grade Reading Assessment Baseline Completed

Philippines: Analytic Support Services for Early Grade Reading Completed

Bangladesh: Institutional Assessment of Early Grade Reading Underway

West Bank: Early Grade Reading Assessment Underway

Nepal: EGRA, Teacher Observation Study, Management Effectiveness Studies Completed

Morocco: Reading Research Studies Underway

Annual Report Year 3 (01 Odober 2013-30 September 2014)



Tajikistan (GBAO) Early Grade Reading Assessment Underway

Requirement 1.2: Develop Plans for Country-Specific Data Support Status






Bangladesh: Institutional Assessment of Early Grade Reading Underway


Nepal : EGRA, Teacher Observation Study, Management Effectiveness Studies Completed



Requi rement 1.3: Adapt or Develop Country-Specific Education Tools Status






Bangladesh : Institutional Assessment of Early Grade Reading Underway


Nepal : EGRA, Teacher Observation Study, Management Effectiveness Studies Completed



Requirement 1.4: Produce Reports on Country-Specific Support Status


Annual Reporl Year 3 (01 October 2013 - 30 September 2014) 9



Burma: Education Sector Desk Review

MENA: Region Cost Comparisons of Early Grade Reading

Egypt: Early Grade Reading Case Study

Egypt: Grade 3 Early Grade Reading Assessment Baseline

Philippines: Analytic Support Services for Early Grade Reading

Bangladesh : Institutional Assessment of Earl y Grade Reading

West Bank: Early Grade Reading Assessment

Nepal : EGRA, Teacher Observation Study, Management Effectiveness Studies

Morocco: Reading Research Studies

Tajikistan (GBAO) Early Grade Reading Assessment

Result 2: Availability of AME Education Data and Trends Expanded

Requirement 2.1: Develop AME Regional and Country Education Trends Analyses

Research Conducted and Data Compiled

Graphic Presentations Designed

Literature Review on Early Grade Reading and MTI Trends

Regional report with Country Data Submitted on MENA Region

Regional report with Country Data Submitted on EA Region

Regional report with Country Data Submitted for CA Region

Requirement 2.2: Conduct Identification and Analyses of U.S. and International Evidence-Based Strategy-Related Education Programs and Interventions

Potential Themes Identified

Review and Analysis on First Theme (Reading)

Literature Review of the Scale-up and Sustainability

Completed

Completed

Completed

Com pleted

Com pleted

Underway

Underway

Com pleted

Underway

Underway

Status

Compl eted

Compl eted

Com pleted

Completed

Completed

Completed

Status

Completed

Completed

Completed

10 Annual Report Year 3 (01 October 2013 - 30 September 2014)



Requirement 2.3: Conduct AME Workshops/Seminars

Traini ng Summer 2013 in Washington DC

Traini ng Impact Evaluation, November 2014 (Bangkok, Thailand)

Result 3: Measurement Tool with Applicability Across Countries Developed

Requirement 3.1: Identify Areas Where Measurement Tools Are Needed

Concept Notes Drafted for USAID Review

Concept Note Review by USAID

Refine Concept Note

Requirement 3.2: Develop the Selected Tool

Reading Achievement Data and Set Standards

Generate Learning Outcome Projections

Too/1 : Use data to: understand existing reading achievement levels, factors that influence or constrain performance; set reasonable standards for reading performance, consider the scope for and likely trajectory of improvement

Tool 2: Use data to: generate projections for how reading interventions could impact learning outcomes on an increasing scale; examine relationship between the variation in the quality and fidelity of implementation and variation in learning outcomes achieved; and mode/likely impacts of national programs at scale

Tool 3: Use data to: measure impact of reading programs by comparing treatment and control schools and investigate the variations within treatment schools; evaluate cost- effectiveness of an intervention; help improve implementation; re-examine initial assumptions on standards for reading performance; and inform policy decision-making and management of national scale programs

Requirement 3.3: Pilot the Tool

Review Variations in Quality and Fidelity

Measure Treatment and Control Schools

Evaluate Cost-Effectiveness and Improve Implementation

Policy Dialogue

Annual Report Year 3 (01 October 2013-30 September 2014)

Status

Cancelled

Underway

Status

Completed

Completed

Completed

Status

Completed

Completed

Ongoing

Ongoing

Ongoing

Status

Pending

Pending

Pending

Pending

11


Result 4: Education Personnel Knowledge of Selected Education and Evaluation Topics and Processes Strengthened

Requirement 4.1: Identify Education Strategy Goal 1 Topics for Classroom Training, Online Training, Online Information/Resource Material, or Future Training

Survey Topics Identified

Train ing Plan Goal 1

Requi rement 4.2: Identify Education Strategy Goal 2 Topics for Classroom Training, Online Training, Online Information/Resource Material, or Future Training


Training Plan Goal 2

Requirement 4.3: Identify Education Strategy Goal 3 Topics for Classroom Training, Online Training, Online Information/Resource Material , or Future Training


Traini ng Plan Goal 3

Requi rement 4.4: Conduct a Training Needs/Preference Survey of Education Personnel in Asia and Middle East Regions

Conduct Survey and Analysis

Requi rement 4.5: Produce a Training Plan

Full Training Plan

Requirement 4.6: Develop Goal 1 Training Modules or Courses for Classroom and Online Training

Research Goal 1 Impact Evaluation

Design Training Modules On-line Course

Requirement 4. 7: Develop Goal 2 Training Modules or Courses for Classroom, Online, and/or Resource Materials

Rolled into Goal 1 Impact Evaluation above in 4. 6

Requirement 4.8: Develop Goal 3 Training Modules or Courses for Classroom and Online Training

Rolled into Goal 1 Impact Evaluation above in 4.6

Requirement 4.9: Deliver Classroom Training for Goals 1, 2, and 3

Research Completed

Status

Completed

Completed

Status

Completed

Completed

Status

Completed

Com pleted

Status

Com pleted

Status

Completed

Status

Completed

Completed

Status

NA

Status

NA

Status

Completed



Result 4: Education Personnel Knowledge of Selected Education and Evaluation Topics and Processes Strengthened

Course Designed Underway

Training Summer 2013 in Washington DC Cancelled

Traini ng Planned November 2014 (Bangkok, Thailand) Pending

Requirement 4.10: Deliver Online Training Modules or Courses Status

Research Completed Completed

Course Designed Completed

Traini ng On-Li ne Completed

Requirement 4.11: Package Informational/Resource Material for Online Access Status

Design Completed Completed

Record ing and Packaging Completed

Transfer to USAID University Completed


EdOata II Task Order 15 (OEP-AME)

Progress toward Objectives (by Result)


14

Overview of Activities and Achievements

DEP-AME activities this year focused on six USAID Mission analytic activities under Result 1.

Bangladesh Institutional Assessment of Early Grade Reading

In May 2013 USAID/Bangladesh asked RTI to assist with analyzing programs, policies, practices, and institutional capabilities for improving early grade reading in Bangladesh. The resulting Institutional Assessment of Early Grade Reading, launched this fiscal year after delays due to security concerns. Once fieldwork was approved, DEP-AME mobilized the team in January 2014 to Dhaka to conduct the institutional assessment of the early grade reading sector.

The Institutional Assessment of Early Grade Reading served as a rapid education sector assessment and related desk study. It examined major aspects of the education sector (government and nongovernment), and relevant elements of the Government of Bangladesh (GOB) outside the education sector that impact upon early grade reading either directly or indirectly. The objective was to identify specific strengths, weaknesses, and leverage points, and to offer a set of recommendations and an overarching conclusion regarding the Bangladeshi education sector ' s potential to scale-up and sustain an early grade reading (EGR) program.

The assessment was conducted by a team of researchers formed by R TI. The team was comprised of Dr. Colette Chabbott (Team Leader) , Dr. F. Henry Healey (Lead Writer) , Dr. Shahnaz Karim (Technical Expert on education governance and related issues), and Dr. Nazmul Haq (Technical Expert on curriculum and teacher training). The assessment involved both a desk study and a field visit during which a number of guided interviews of key informants took place over the calendar period January 26-February 14, 2014. The report reflected the fmdings of the assessment. The desk study itself was largely undertaken before the field study took place.

Through the field study, the team interviewed key officials in the Ministry of Primary and Mass Education (MOPME), the Ministry of Education (MOE), and their central and lower level departments, offices, institutes, and centers. The team also visited schools where they had an opportunity to talk with teachers, Head Teachers, and members of the school management committee (SMC). Additionally, a focus group session involving eight teachers from various schools in Dhaka was conducted. The field study also allowed the team to meet with people from a number of non-government organizations (NGO), international NGOs, research centers (RC), and development partners (DP).

Annual Report Year 3 (01 Odober 2013- 30 September 2014)


• To ensure that the information presented in this report was as accurate as possible, measures were taken to cross check the information gathered from one informant by asking for the same information from a number of other relevant informants. When a number of different informants provided the same answer, the team felt confident that the information fairly accurate, realizing of course that in most instances our sample size was exceedingly small. In those instances where the team was unable to find a clear answer, it is noted in the report. Results of the assessment were summarized and analyzed in consultation with the USAID/Bangladesh team and USAID Asia and Middle East Bureau Senior Education Advisor Mitch Kirby. The draft report was submitted to USAID in March for comment along with a summary brief for external audiences. A revised version was submitted later in the year.

Next Steps

• Pending revisions from USAID/Bangladesh Mission, the report will be fmalized. This will conclude this activity in the coming year.

Morocco: Reading Research Studies

DEP-AME began work on the Morocco: Reading Research Studies series this year, submitting a proposal to USAID in March, as requested, for funding three major studies on reading to provide the Ministry of Education and Vocational Training the needed evidence for its upcoming reform strategy. Within weeks, DEP-AME mobilized to conduct the research for what are the first three of an anticipated five Reading Research Studies. Field activities drew on the technical expertise of the VarlyProject and other experts to carry out the research.

The instrument design was conducted in April and fieldwork training, instrument piloting and fieldwork preparation occurred during April and May. Data cleaning and analysis occurred in May and report writing began in June. Translation of the fmal deliverables occurred in July and the instruments, survey documents and fmal report were subsequently submitted, concluding Phase 1. Translation of the first three deliverables has been completed and submitted, including instruments, survey documents and the Final Report. Dissemination workshops on the results fmdings were also conducted by DEP-AME and included outreach to other implementers to facilitate wide access to the fmdings. The three studies, conducted by RTI, covered the following topics:

• Curriculum and Textbook Analysis: This study reviewed the official curriculum and a selection of textbooks to determine whether they reflect international best practices in the teaching of Arabic, and whether they provide the framework for improving reading instruction in the early grades. The study looked at the gaps between the official curriculum and evidence-based practices in teaching Arabic reading. The analysis revealed the need to improve the presentation and pedagogical quality of primary school textbooks. The study

Annual Reporl Year 3 (01 October 2013- 30 September 2014) 15


16

showed that certain key competencies in reading acquisition-for example, phonemic awareness, letter-sound identification and incremental decoding-were not addressed sufficiently in the textbooks before requiring children to do more complex tasks like reading phrases and understanding of texts. Monitoring of student learning focused mostly on recitation and oral comprehension skills, and much less on reading activities that require a significant linguistic and cognitive effort such as decoding and free writing. Teacher guides were not formally analyzed in this study; however, they were used as reference documents to the textbooks to see what the teacher guides were recommending the teacher to do with the textbook.

• Pre-Service Teacher Education Analysis: The objective of this study was to look at how teachers in training learn to teach reading in the early grades. The key research questions revolved around whether reading is taught as something explicit or implicit in the teacher preparation curriculum, and whether the methods promoted in teacher training reflect best practices in the teaching of Arabic to early readers. Several factors limit the effectiveness of pre-service teacher training to prepare teachers to teach Arabic reading in the early grades. The main factor is the limited amount of time required for teacher certification in the CRMEF (about 9 months). There is not sufficient time dedicated to providing in-depth strategies for teaching a specific subject area like reading alongside all of the other general professional knowledge and skills required by the curriculum (lesson planning, classroom management, etc.). The one-year program does provide a range of courses, but not to the depth that a full three- or four-year program would. This is compounded by the fact that teachers in training can come from undergraduate programs in any subject area, and therefore do not always have pre-requisite content knowledge in the subject to be taught. Finally, the study found an important gap between theoretical know ledge and practical skills, which needs to be addressed in order to better prepare teachers for their practice.

• Teacher Attitudes, Beliefs, and Practices: This study examined the perceptions of teachers towards the teaching of reading in Arabic and the way these perceptions influence practice. The principal research questions were: What beliefs have the most influence on the teaching methods used in early reading lessons? How do these perceptions impact the professional growth of teachers? How to teachers react to questions related to language of instruction? Are they receptive to using local dialects to support teaching reading in Arabic?

The analysis found that teachers express a relatively optimistic view of their motivation to teach in primary grades and to teach reading in Arabic. However, many factors make it difficult for them. First, they do not feel they have the resources, in general, to teach in crowded or multi-grade classrooms, regardless of pedagogical expertise. Next, there is no consensus on the best method to use for reading instruction; some teachers maintain the government's recommended whole language method, while others opt for a mixed or

Annual Report Year 3 (01 October 2013- 30 September 2014)


"syllabic" approach to teaching reading. Very few are aware of recent evidence of best practice in teaching reading in Arabic in the early grades (for example, use of morphology and a phonetic approach), but they do recognize that the current textbooks are not aligned with the ability of the children. Teachers fmd themselves adapting on their own to the sociolinguistic environment and to teaching materials that are too difficult for most students especially those with no preschool education.

Phase 2 Planning and design for two remaining Reading Research Studies is underway for the coming fiscal year. These are slated to be completed around March or April 2015.

Next Steps

• Launch Phase 2 Reading Research Studies ' activities in the next reporting period.

Nepal: Early Grade Reading Assessment (EGRA), Education Management Efficiency Study and Teacher Observation Study (EMES-TOS)

The Nepal Early Grade Reading Assessment (EGRA), Education Management Efficiency Study and Teacher Observation Study (EMES-TOS) was launched in the prior reporting year and concluded its activity this Fiscal Year. An initial delay arose due to electionrelated security concerns which necessitated a scheduling change. Once mobilized, the team moved rapidly to launch this three-part analytic study, intended to strengthen local skills in design, evaluation, and management of education programs, and quality data capture and analysis to support them.

On October 1, fmal approval was received from US AID for the study. Three weeks later DEP-AME was informed of a delay to the anticipated November 10-14 workshop. This was then rescheduled until after the election. An updated schedule for the Nepal activities was provided to USAID and field work commenced.

In November DEP-AME staff traveled to Nepal to facilitate the adaptation of the EMES/TOS instruments and train the assessors. The full field data collection of the EMES/TOS was completed by December 31. During the second quarter DEP-AME completed the field data collection for the EMES, TOS and EGRA and a draft report was submitted for USAID comment.

The focus of the EGRA study provided a nationally representative assessment of Grade (G) 2 and Grade 3 students in Nepal public schools. In January/February of2014, the EGRA study was conducted, which included an EGRA for Grades 2 and 3 students in the Nepali language, a student interview, and a teacher passage about the upcoming Early Grade Reading Program. The study was designed to provide a useful baseline or benchmark by which the effectiveness of the upcoming national reading program could be measured. The report provided key results from a national study measuring the reading skills of students in Grades 2 and 3 in Nepali across all ecobelts.

The full field data collection of the EMES/TOS was completed at the end of December. RTI and its local partner New ERA, began the process of entering and cleaning the data



18

during the next quarter. RTI/New ERA visited 104 community schools across 13 districts, interviewing at least one head teachers, one Grade 2 teacher, one SMC member, and observing one Grade 2 classroom in each school. In addition, 13 District Education Officers (DEOs ), 13 Regional Centers (RCs) and 6 ETCs were interviewed. Informational interviews were also conducted with the Ministry of Education and it 's Central Level Agencies (CLAs) .

During the November trip, several important agreements were reached with respect to the scope of the upcoming EGRA, namely that Grade 2 and Grade 3 learners were to be assessed in Nepali language only; community (public) schools were to be included in the formal study; and a small sample of private schools would be included in an annex to the report to provide anecdotal information to the Ministry. To the extent possible, Ministry officers from the ETCs would also be included as field assessors or supervisors. It was agreed with the Ministry that the adaptation and training of the EGRA would take in January and pilot data collection the first week of February. This was followed by the full field data collection in February and March. Four key research questions were addressed in the Final Report.

Research question 1 asked, to what extent are students in Grade 2 and 3 learning to read in Nepali? In general, students were demonstrating emergent reading skills by Grade 3. There were increases in mean scores between Grades 2 and 3 on all subtasks and in all ecobelts. Girls and boys performed equally well across all subtasks, showing no significant gender disparity in teaching and learning in typical Nepali classrooms in the early grades. However, the mean scores achieved in Grade 3 are more indicative of emergent reading ability rather than the more secure level of skill one would expect after three years of schooling. The levels of reading ability this EGRA revealed in Grade 3 were what should ideally be seen at the end of Grade 1 or beginning of Grade 2. These results suggest that more focused instruction is needed early in the primary cycle to more quickly build literacy-related skills and reading ability. The answers to the second research question shed additional light on this need.

Research question 2 asked, what reading-related skills are students in Grades 2 and 3 acquiring? Students in Grades 2 and 3 performed well on the letter sound know ledge subtask, which means they were acquiring the understanding of the alphabetic principle that is the foundation for reading. However, student performance in both grades on matra

and nonwords subtasks was lower than expected for the grade level. In addition, there remained large percentages of students, especially in Grade 2, that were scoring zero on the matra and nonword subtask. Poor performance in these skill areas usually indicates that instruction does not sufficiently emphasize syllable (matra) reading or decoding (nonword reading). In light of the results for the first two research questions, effort might best be directed towards all students in Grade 1, as well as struggling students in Grades 2 and 3. The results show that once students gain the foundational skills needed to begin reading, general achievement rises. Instruction in Grades 1 and 2 should target these basic skills, with particular attention paid to learning the matras and learning how to

Annual Reporl Year 3 (01 October 2013-30 September 2014)


decode unfamiliar words. Students may benefit from explicit instruction in the sounds of the matras, using similar strategies that have been used to teach the letter sounds, which, results indicate, have been largely successful. In addition, students should receive instruction in comprehension strategies beginning in Grade 1 through oral comprehension stories.

There are large disparities in student achievement by ecobelt, with Grade 3 students in the Terai region performing at levels similar to Grade 2 students in the Mountain, Hill, and Kathmandu Valley ecobelts. For students in the Terai ecobelt, home language was an important factor that may have held back their acquisition of literacy in Nepali. The rna jority of students in the Terai ecobelt reported speaking a language other than Nepali. Those students read fewer words correctly per minute of the reading passage and answered fewer questions correctly on the listening and reading comprehension subtasks. This suggested that their lack of familiarity with the Nepali language was not adequately addressed during the first few years of primary school. Although not the only factor in the different scores across ecobelts, home language is an important consideration for the Terai ecobelt. Given the differences in results, extra support and attention should be paid to students in the Terai ecobelt. In particular, students' oral language skills in Nepali must be strengthened early in their school career so they can learn to read fluently and with comprehension inN epali. Strategies that explicitly help children bridge from their maternal languages to Nepali may be needed.

Research question 3 explored the factors, both in-school and out-of-school, that help explain student performance on the EGRA. As mentioned, an important out-of-school factor is where students reside/go to school, with Terai ecobelt demonstrating lower scores across all subtasks and grades. Linked to ecobelt residency is the other issue already mentioned-home language. While this may be of particular concern in Terai, in general students whose first language was not Nepali were not performing as well as those who came to school speaking the language of instruction. The other out-of-school factor that correlates with reading performance is the relative socio-economic status (SES) of a child's family. Grade 2 students in the lowest socioeconomic quartile had a mean oral reading fluency score equivalent to half of the mean score of those in the highest quartile. The difference in those mean oral reading fluency scores was lower in Grade 3. While it is a good sign that more time in school was helping students from less wealthy families begin to catch up to better off students, schools may need to recognize the needs of disadvantaged students earlier and intervene to help address challenges those students may face.

While the gap in reading scores between students from the lowest and highest SES quartiles diminished between Grades 2 and 3, the gap between Nepali and non-Nepali speakers seemed more persistent. With that in mind, in addition to intervening early or providing instruction in mother tongue as mentioned above, it may be worthwhile to offer specific additional learning opportunities to students who continue to struggle with oral

Annual Report Year 3 (01 October 2013-30 September 2014) 19

EdOata II Task Order 15 (DEP-AME)

20

comprehension ofNepali. Several in-school factors also demonstrated significant positive correlations with reading performance.

These included aspects of teachers' reading instructional practices, schools' practices that support reading remediation, parent-teacher association (PTA) support for reading, and the availability of teaching and learning materials. These fmdings suggest that certain aspects of school management, instruction, and school support from communities are important factors to address when rolling out an Early Grade Reading Program. For example, classrooms in which students were frequently asked to read aloud had a mean oral reading fluency score greater than the mean of students who were not asked to read aloud. In general, teachers who adhered to instructional practices that align more with recognized best practice and research on literacy instruction had students who were performing better. Getting such practices to become widespread is one fundamental challenge that a national reading program will need to confront. Another in-school indicator, teachers' having reference materials or teachers' guides, correlates with students having mean oral reading fluency scores higher than students whose teachers did not. This may reflect a more general relationship between availability of materials and student performance in reading. Further research could explore the types of materials that contribute most to improved reading outcomes. Nevertheless, it is safe to say that increasing the availability of instructional resources and reading material should be a cornerstone of a national reading program.

Schools that had staffs that recognized that attention must be focused on struggling students and that in general put more effort into supporting remediation also had higher mean reading fluency scores than those that did not. And when communities contributed to efforts to improve reading, the students in those schools performed better. These two fmdings suggest that building up greater awareness of the need to focus on assisting struggling students and mobilizing and supporting local initiatives related to reading improvement could be useful features of a national reading program. Another in-school factor concerns what teachers thought and believed about several aspects of the teaching and learning of reading. Teacher attitudes and beliefs were measured, and results revealed patterns of beliefs amongst teachers with regard to mother-tongue instruction, reading instructional practices, student ability, and student engagement. In general, most teacher attitudes and beliefs fell in the middle of the ranges of the continua used in this survey. This suggests that for most teachers, attitudes and beliefs reflected practices that will support reading instruction-support for mother tongue, active learning, and best practices for literacy acquisition. However, some teachers held attitudes and beliefs that run counter to what will be needed for a national reading program to succeed.

Research question 4 asked how well teachers understand a written explanation of a proposed early grade reading program. Results showed that a majority of teachers were able to read the passage and successfully answer most of the questions asked about it. This shows that most teachers had reading and comprehension abilities that would enable them to process information written at a secondary school level. Yet, a small minority of

Annual Repott Year 3 (01 Odober 2013- 30 September 2014)


teachers seemed to need help processing overly complex written information. In general however, the results pointed to the promise of designing written materials and professional development courses to introduce new instructional strategies for the teaching of reading, as teachers already are able to read well with comprehension. Also, materials in which teachers were asked to read and implement a lesson can be designed with confidence in teachers ' ability to comprehend the instructions and the lessons.

The final EMES report provided data on specific education management capacities and teaching practices that will affect the Ministry of Education's (MOE ' s) ability to implement and sustain a national EGRP at scale. In so doing, it examined the variety of system structures and attributes that lend themselves to EGRP implementation, identify existing gaps and capacities to be addressed, and highlight the strengths and opportunities that could be leveraged by the Ministry and its fmancial and technical partners.

The EMES-TOS provided a description of the current instructional practices and institutional and management capacity requirements to support a coherent national early grade reading program. Each of these assessments and studies conducted was intended to provide a useful baseline or benchmark by which the effectiveness of an upcoming national reading program. The three main deliverables included:

• EGRA report that contains analysis of teacher assessment and learner performance data, disaggregated by region and ecobelt.

• A brief anecdotal report on private schools ' performance viz EGRA and EMES/TOS.

• A separate report on the study testing print vs. electronic methods.

These reports presented data that addressed each of the research questions and used those data to reach summary conclusions about the extent to which the education system in Nepal is prepared to support implementation of a national EGRP. The Final Report was submitted to USAID following the Policy Dialogue Workshop in June. This concluded the activity.

Next Steps

• None, this activity is completed.

Philippines: Analytic Support Services for Early Grade Reading

Under DEP-AME, the 2014 Philippines EGRA study (PhilEd Data II) was conducted to expand the knowledge base for the Mother Tongue Based-Multilingual Education (MTBMLE) program by conducting a comparable exercise in three additional mother-tongue languages-Cebuano, Hiligaynon, and Maguindanaoan. Another representative set of data was collected for Ilokano, which provided additional information on whether quality of MT instruction is improving with implementation experience, and how much children improve their reading skills from Grade 1 to Grade 2. The 2014 study used the same EGRA instrument as 2013- with appropriate adaptions-to observe and measure reading

Annual Reporl Year 3 (01 October 2013 - 30 September 2014) 21


22

skills in five key areas: (1) Alphabetic principle; (2) Phonemic awareness; (3) Vocabulary; ( 4) Fluency; and ( 5) Con1prehension. The field work included a local survey research partner, TNS Global.

Finalization of the scope of work for the Philippines: Analytic Support Services for Early Grade Reading occurred in October and included activities to build on earlier work supported under the Philippines Ed Data task order: This included:

• Conducting a sample based EGRA in 4 mother tongue languages. Suggested languages included: Ilokano, Hiligaynon, Cebuano, and Maguindanao. Final determination of the languages was made in consultation with USAID/Philippines.

• Data were also collected and analyzed in the regions where the mother tongue assessments were conducted on the instructional practices and use of language in early grade classrooms (including information on teacher facility in different languages).

• DepEd capacity was built to use Tangerine® and to manage the data generated by Tangerine-based implementations ofEGRA.

This activity began at the start of the Fiscal Year immediately after fmal agreement with the Mission on the scope of work. RTI established a subcontract with Taylor Nelson Sofres (TNS), a local NGO, to support the selection and training of assessors and the management of all field data collection.

In November, RTI facilitated an instrument adaptation workshop during which DepED officials, local linguists and curriculum experts worked to develop the survey instruments for each of the four target languages: Ilokano, Hiligaynon, Cebuano, and Maguindanao. The teacher survey and classroom observation instruments were developed in conjunction with DepED colleagues. During December all the instruments were fmalized and rendered into electronic format using the Tangerine® open source software. All data collection was carried out using tablet computers. Survey teams included TNS staff and locally recruited teachers and other DepED officials. In December, TNS began identifying and recruiting assessors, and DepED designated the department staff that would participate in each region. The teacher survey and classroom observation instruments were developed in conjunction with DepED colleagues.

The results provided baseline data on Grade 3 students' reading achievement and language readiness under the previous (prior to adopting mother tongue-based instruction) national curriculum. PhilEd Data I also explored the relationship of these skills to selected factors in students' school and home environment. This evidence can be used to inform policy and the implementation of appropriate interventions in schools and classrooms across the country, as well as to monitor the effect ofMTB-MLE in years to come. The regional language EGRA for Ilokano was the first rigorous and large-scale assessment of student learning in one of the mother-tongue languages.



One of the key takeaways from the study is the existence of important inequities in achievement between boys and girls, and across regions. The challenges in the lowestperforming regions are well-known-instability, poverty, geography, lack of access to early childhood education, etc.-and many programs are in place to try to address these factors. However, understanding which factors are having the most influence and what specifically can be done most effectively and efficiently to address the achievement gaps will be critical going forward.

The most prevalent sign ofMTB-MLE being implemented in the four regions included in this study was the consistent use of the regional mother tongues as the languages of instruction in both reading and other lesson periods. All teachers observed were using the mother tongue for the vast majority of their instructional activities and interactions with students. Students were also observed to be engaged in the activities being conducted in mother tongue. Also, Filipino and English are being taught as expected, and children are acquiring beginning skills according to the progression expected by the curriculum. However teachers were not exhibiting the full set of skills needed to appropriately support literacy acquisition in the mother tongue during their reading lessons.

Many teachers indicated that they had not been sufficiently trained for mother-tongue instruction, and more than 70% stated that they did not have sufficient materials and resources for teaching reading. Majorities of teachers reported not having important materials like the mother-tongue teachers' guides in three of the four regions. A particular concern is whether the activity of reading practice is taking up a sufficient amount of time during reading lessons, or if teachers are disproportionally speaking and asking children to read or recite in chorus. There were also very different expectations of when children should be acquiring reading skills in different languages (or possibly just different expectations ofwhat "fluently" means).

For the vast majority of students in the regions included in this study, the language of instruction matched their home language. However there were students who did not speak the mother tongue of instruction at home, especially in Region 1 where as many as 12% of the students were in a school that was using a language of instruction that was not their mother tongue. Children 's self-reports of1anguage use in the home may be somewhat unreliable, and assessors noted in some circumstances children were unfamiliar with the terms 'language ' and so additional prompting needed to be done to make sure the children were answering accurately. This is an important fmding in itselfhow are children being prepared for instruction in multiple languages, and how are teachers bridging between the first and second (and third) languages if children are not exposed to the meta-language associated with code switching in the classroom, the community and the home? What are children learning about being multilingual?

Another somewhat accidental fmding was that teachers have been hearing about EGRA through a variety of different sources including K- 12 workshops, Basa Pilipinas, and different regional initiatives. In their responses it became clear that were different understandings about what EGRA is and what it is supposed to do. In some cases the



24

term "EGRA" is confused with the notion of teaching reading in the mother tongue rather than as a specific type of reading assessment tool.

As this was still early in the process of rolling out a complex reform, it is understandable that the resources, supports, and practices for MTB-MLE were not yet all in place. Notably, some important aspects ofMTB-MLE implementation show strong relationships with student performance in reading. For example, the 20o/o of students who often used their mother-tongue book for reading were 2. 8 times as likely as those who never used it to be in the top 25o/o of reading performance. Similarly, the 24% of students whose teachers often asked them to read in the mother tongue were more than three times as likely to be in the top 25%. This indicates that ifMTB-MLE implementation conditions could be better assured across a much broader range of schools, then it is very likely that reading performance will improve.

The uneven implementation that these data show helps explain the types of outcomes presented in this report. According to the MTB-MLE expectations, children should be learning to read in their mother tongues by the end of Grade 1. A large proportion of children learning in Maguindanaoan were scoring zero on all the EGRA subtests, except listening comprehension in Maguindanaoan. This indicates that many children were not acquiring even the building blocks of literacy in that language during Grade 1. Ilokano and Hiligaynon also had high proportions of children in Grade 1 with zero scores on several subtests.

Providing this support to DepEd reinforced their capacity for nation-wide use of the EMIS Toolbox to support generation of Basic Education Report Cards (BERCs). DEPAME completed implementation and concluded the activity with analysis of data from February 2014 data collection in four regions. The Final Report was drafted and submitted to USAID following a Policy Dialogue Workshop which was held in August. This concluded the activity.

Next Steps

• None, activity completed.

Tajikistan (GBAO)- Early Grade Reading Assessment

DEP-AME received a Concept Note in January from the USAID/CAR Mission requesting support for an EGRA to be conducted in the Gorno-Badakshan Autonomous Oblast (GBAO) region of Tajikistan. In March, a proposal was submitted to USAID and it was approved. Shortly thereafter, Dr. Marcia Davidson was identified to serve as the reading expert to accompany Dr. Hank Healey on the first field visit to launch the activity in GBAO in May 2014. Local partners were identified and dialogue, planning and subcontracting procedures began with M-Y ector as the local training and data-collection subcontractor. The ultimate goal is to launch an EGRA exercise and to obtain a first-hand understanding of the existing early grade reading (EGR) situation in GBAO.

Annual Report Year 3 (01 October 2013 - 30 September 2014)


Field implementation launched in May 2014 with a field visit by Drs. Hank Healey and Marcia Davidson to GBAO plan for the anticipated Fall EGRA administration. During this launch visit, the team met with key Ministry of Education (MOE), GBAO Department of Education (DOE), USAID, and relevant other stakeholders, including RTI local partner M-Vector.

The team identified and began to gather the information needed to conduct the adaptation workshop and EGRA exercise which was scheduled to occur in September/October 2014. This information included the primary-level reading curricula and curricula materials, existing EGRAs that have been (in 2011) or will be (via the Quality Reading Project in 2014) conducted in Tajikistan, school data, and general knowledge gained from conversations with key people in GBAO and Dushanbe. The team was impressed with the number of books and learning materials available in classrooms. Teachers seemed generally quite motivated and there was no indication of unmotivated students.

There was some indication (including teacher comments at presentation session) that children do not learn phonology and how that knowledge of letter sounds can be integral to a strong reading foundation. While higher level thinking skills are very much valued and strived for in the visited schools, many of the activities observed focused on review and examples of content previously learned. There was no evidence of procedural knowledge being taught as it might relate to problem solving or analyzing text.

In late August, the DEP-AME team was required to cancel the planned travel and data field collection set for September/October due to a change in Tajikistan/GBAO policy on the issuance of travel visas. In addition, the coming winter months required that the data collection be postponed until the spring when roads will again be passable. The planned adaptation workshop and EGRA exercise will be rescheduled at a later date, to be determined.

In GBAO, the team first met with the Deputy Director (DD) of the GBAO Department of Education (DOE) in Khorog. During this meeting, the following information was provided:

• The language situation in GBAO is one in which nine local languages, or mother tongues (MT), are spoken at home and that the language of instruction (LOI), Tajik, is virtually a foreign language for all students.

• The DD stated that multiple home languages are the greatest challenge in GBAO since the languages are quite different from one another and Tajik.

• Students are required to learn Russian and English as well as Persian script for Tajik.

• Some schools offer an early childhood development (ECD) and/or "Grade 0" program that have been created to help transition children from Mother Tongue to Tajik.

• Students not attending these schools are immersed in Tajik upon Grade 1 entry.



26

• The Open Society Foundation funds 103 preschool centers managed by the Institute for Professional Development (IDP).

• There is a lack of adequate texts for Grades 1-4, as anticipated.

• There is a need for family literacy programs so that parents can learn ways to support their children's early grade reading (EGR) learning.

• Currently, there are no after school support programs, yet summer camp (i.e. , summer school) is provided for 5,000 of the total 45,000 students.

• Overall enrolment has been declining for at least the last 10 years due in large part to a significant amount of out-migration. This has resulted in some very low pupil-teacher ratios-5-to-7 in one school as per the DD-as well as closed schools. When schools close, he queried aloud, "Where will children learn?''

• Despite these challenges, 83o/o of all Tajiks who study in graduate programs abroad are from GBAO.

It was noted for the DEP-AME team members that the DOE wanted all teachers to have more training and that they would like to channel more resources toward pre-primary and primary realizing that such an investment has a very high public return. At the conclusion of the meeting the team was provided with a detailed list of all the primary schools in GBAO inclusive of each school's Grade 1, 2, 3, and 4 enrolments. Following the meeting with the DD, the team conducted the formal EGRA Launch event, which took place within the facilities ofiPD. Those attending the launch included educators from IPD, teacher trainers, curriculum and curriculum materials specialists, assessment specialists, and M&E specialists



West Bank Early Grade Reading Assessment

DEP-AME continued implementation of the West Bank: Early Grade Reading Assessment (Grade 2) baseline this year. The activity was designed to strengthen local skills in the design, evaluation and management of education programs, and quality data capture and analysis to support them. This was the first baseline assessment of early grade reading skills representative of all Grade 2 students in MOE schools of the West Bank.

USAID support for this activity is intended to generate regional and country-specific education data, and analyses of those data, that can be used by the Ministry of Education and Higher Education (MOEHE) to prioritize education needs and future investments. The objective is to help inform policy decisions and planning by the Palestinian Authority (PA) for improved reading instruction and student learning outcomes to ; (1) infuse EGRA fmdings into the design and development ofMOEHE curricula and teaching resources for enhanced reading instruction in Grades 1-3; (2) establish a national baseline of Grade 2 reading skills to measure future progress in enhanced reading performance in MOEHE schools; and (3) strengthen the capacities and deepen the knowledge base ofMOEHE staff to implement EGRAs.

This Grade 2 baseline established the capacities and deepened the knowledge base of MOEHE staff to implement future EGRAs. During implementation, allEGRA assessors and assessor team leaders were MOEHE supervisors or candidate teachers and all planning, training, implementation, and dissemination was conducted in close collaboration with the Ministry's Assessment and Evaluation Department (AED) and EGRA Steering Committee. Central ministry and MOEHE districts actively supported the field implementation with school liaison, orientation, and enrollment lists for sample selection.

This began during a November Launch Week during which DEP-AME meetings were held with the USAID Mission and the Palestinian Ministry of Education. DEP-AME team members traveled to Ramallah to:

• Familiarize USAID/WBG Mission and the MOE with the objectives, design and implementation of the national EGRA baseline; identify counterpart officials and resource persons and their contacts; review and fmalize work plans and schedules; finalize the sample size and desired stratifications; and review and agree upon the roles, tasks, responsibilities, and deliverables of the parties.



28

• Review the EGRA F AQs and familiarize MOE and the WBG Mission with previous Arabic EGRAs and analytical reports.

• Review and confirm the purpose and intended use of results from the EGRA.

• Identify early grade literacy/language specialists and assessment specialists within MOE (i.e. future champions) as needed and communicate contact points.

• Identify local language and reading experts to provide technical and psychometric advice.

• Confirm EGRA personnel and MOE responsibility, and sufficient, appropriate personnel available to supply EGRA assessors and assessor team leaders.

• Finalize schedule with MOE and USAID/WBG for EGRA implementation.

• Clarify responsibilities, roles and timeframe for EGRA tool adaptation.

• Obtain Grade 2 textbooks for Arabic reading as resources for EGRA tool adaptation.

• EGRA Sample Frame and School Data with the MOE and decide the sample frame and desired strata for the analysis ofEGRA results.

• Clarify the process, MOE role and responsibilities in the selection of sample MOE schools and Grade 2 students for EGRA testing.

Adaptation of instruments to provide training for data collection in Nepal for the TOS and EMES (to be followed by the EGRA) in November and December. Following this, a decision was made on the a March time frame for the assessment and dissemination of results; selecting the final EGRA subtasks to be used; setting up a process for adapting the Arabic EGRA for Palestine from the instruments used in Jordan and Egypt; and preparing the database of schools to be used for creating the sampling frame and strata.

During the second quarter, DEP-AME conducted the Grade 2 EGRA itself, working closely with the Palestinian Ministry of Education, the USAID/West Bank/Gaza Mission, and subcontractor Infonex. (The policy dialogue workshop was then scheduled to conclude the activity in June but this was delayed due to hostilities.) The final week of field data collection and counterpart meetings concluded in April with a celebration event that convened all of the assessors, assessor team leaders (supervisors), field coordinators, US AID and MOE representatives, to mark the successful completion of this fieldwork. In fact, the USAID Mission asked DEP-AME to produce a documentary to show at a Policy Dialogue Workshop.

This EGRA was conducted with a representative sample of 150 Ministry of Education and Higher Education (MOEHE) primary schools, stratified by school gender and selected randomly from 16 districts in the West Bank. The 2,953 tested students were randomly selected from Grade 2 enrollment lists prior to each school visit. The results are representative ofMOEHE Grade 2 students and primary schools in West Bank. The fmdings of this national baseline were intended to inform the strategic design and development ofMOEHE enhancements to the curriculum, resources and teaching of



early grade reading. Implemented by MOEHE supervisors and candidate teachers, this EGRA strengthened ministry capacities and deepened its knowledge base of early grade reading and reading assessment. Key findings of this Grade 2 baseline included:

• Good foundation of pre-reading skills: The sample Grade 2 students demonstrated good proficiency in the basic pre-reading skills of letter sounds knowledge and nonword I invented word decoding. Average scores on these two subtasks are, by far, the highest of any Arabic EGRA in Grades 2 or 3; and the percentage of zero scores are the lowest. There is a strong foundation of reading proficiency in these pre-reading skills among MOEHE Grade 2 students. This was a strong and positive fmding.

• Students struggle to read connected text in passages: Most Grade 2 students were struggling to read familiar words in connected text (passages, stories), indicative of low reading fluency. Average scores on the oral reading fluency subtask (with diacritics) are low and the percentage of students scoring zero is high.

• Low reading comprehension: Most Grade 2 students do not comprehend what they are reading. Average scores on reading comprehension ( 60 seconds reading with diacritics) are low and the percentage of students with zero scores is high. The generally good results on the listening comprehension subtask, however, suggest that most Grade 2 children capably understand formal Arabic. Hence the low scores on reading comprehension in Grade 2 chiefly result from low reading fluency and undeveloped reading comprehension skills. Comprehension skills must be learned, practiced and mastered-they do not come automatically to students who mechanically read well.

The results will be used to inform policy decisions and planning by the Palestinian Authority (PA) for improved reading instruction and student learning outcomes in the early grade, while the longer term objective is enhanced teacher training and learning resources for improved reading proficiency by primary students.

Next Steps

• Complete review of the Final Report for USAID acceptance and fmalize a video documentary prepared to support future training.



DEP-AME activities focused this year on two analytic activities under Result 2. These included completion and translation of the Topical Analysis of Early Grade Reading Instruction in Arabic and development of related Early Grade Reading Briefs for use that the November 2013 All Children Learning Workshop in Rabat, Morocco, and the submission of an updated Topical Analysis of Scale up and Sustainability. In October, USAID and DEP-AME reviewed the Result 2 Trends Analyses fmal documents.



30

Agreement was reached that no further work would be conducted until updated analyses are conducted in 2015 or 2016. Throughout October and November communication also continued with the USAID Bureau for Economic Growth, Education, and Environment (E3) regarding the Morocco All Children Learning Workshop.

Topical Analysis: Early Grade Reading Instruction in Arabic

• In October, DEP-AME submitted a draft of the Topical Analysis: Early Grade Reading Instruction in Arabic to USAID for review. A subsequent revision was submitted on November 1 for review and transmittal to a translator in preparation for the All Children Learning Workshop I Rabat, Morocco. Further inputs were received in November from the US AID E3 bureau and the comments were incorporated into the paper. In early November, an updated budget was submitted to USAID for the Topical Analysis: Early Grade Reading Instruction in Arabic to incorporate additional inputs, including preparation of Country Briefs and travel of presenters to Rabat, Morocco to participate in the All Children Learning Workshop.

• The All Children Learning Workshop in the Middle East and North African region was held in Rabat, Morocco from December 2-5, 2013 in order to mobilize partners from governments, civil society, and development organizations together to share knowledge and best practices on how to dramatically improve the learning levels of children learning in the early grades across the region. DEPAME also researched and drafted Country Briefers on the status of early grade reading in Middle East/North Africa (MENA) countries, including Egypt, Iraq, Jordan, Lebanon, Morocco and Yemen. The Country Briefer profiles were developed for use with country delegations. DEP-AME also supported a roster of speakers for the Workshop, including Dr. Sylvia Linan-Thompson, Dr. Helen Boyle and Ms. Samah Al Ajjawi of Florida State University. Dr. Amber Gove also presented on the EGRA Barometer which is supported under Result 3.

• In November, a revised draft of the Topical Analysis: Early Grade Reading Instruction in Arabic was sent to USAID for translation, including a list of interviewees by country. Additional feedback was received and a reorganization of content was undertaken to incorporate the requested changes. In December, input received from USAID and incorporated into the fmal report. Subsequent revisions, including the eventual translation of the report into Arabic, concluded this activity by the fmal quarter of the year.

Next Steps

• None, this activity is completed.

Topical Analysis: Scale up and Sustainability

In the second quarter, DEP-AME submitted an updated draft of the Topical Analysis: Scale and Sustainability under Result 2. The paper offered guidance on how to scale up



and sustain efforts aimed at improving early grade reading (EGR) in developing countries around the world based on an extensive review of the scale-up and sustainability literature. This revealed that there are several fundamental factors of successful scale-up and sustainability:

• The overall effort unfolds within a potentially hostile political economic environment, therefore reform support measures must be taken to address it; the effort must be demand driven; people must be held accountable, and incentives, both positive and negative, matter;

• Measures must be taken to engender widespread and deep-rooted ownership;

• That which is going to scale and being sustained must be accommodated within a coherent systemic and institutional home, the effort must then be systemic;

• Trivial as it may sound, but often ignored, the effort must be doable over time, the host country must be able to carry out the tasks and assume the recurrent and capital costs required to keep it going;

• Much of the work of reform needs to be carried out by an entity whose primary if not sole purpose is to do this work-implementation support is necessary;

• Depending on the nature of the intervention being taken to scale and being sustained, it must, to some degree, be adaptable, flexible, and able to be accommodated across a variety of contexts;

• The effort needs to be seen as on-going-reform never really ends-and as such it must be learning- and evidence-driven.

The reported also noted that each fundamental factor has its respective set of operational requirements-the specific things that one must do in order for them to be realized. If, for example, the effort must be demand driven, then specific tasks must be undertaken to generate that demand. If people must be held accountable, then viable accountability mechanisms need to be put in place and fully operationalized to ensure that this happens. This is useful, as it suggests that scale-up and sustainability can be greatly abetted if implementers pay attention to these operational requirements, but it also imposes an obligation to do so. For this reason, listing and delineating them carefully is an important task.

Also, a number of these fundamental factors strongly suggested that what should be taken to scale and sustained within the education sector is the capacity to innovate as opposed to the innovation itself. Making this distinction is itself insightful. What works in one context may not work well in another. Therefore, it makes sense to scale-up and sustain the capacity to innovate.

Next Steps

• Pending feedback from USAID, revise the Topical Analysis: Scale and Sustainability if desired.



Result 3: Measurement Tools for Cross-Country Use


The EGRA Barometer is an interactive tool intended to offer wide range ofEGRA data from Asia and the Middle East and ultimately to be piloted in the Asia and Middle East. This year, DEP-AME began development of a proposal for Phase 2 to pilot the EGRA Barometer in the Asia/Middle East region and identified five objectives for further development next quarter:

• Share the Barometer. Let target audiences know about the tool.

• Teach target audiences how to use the Barometer to facilitate dialogue, inform policy and influence program planning.

• Learn how the tool is used in the field and identify its most useful features

• Add more countries to, and increase the functionality of, the Barometer

• Understand the resources required to add datasets resulting from assessments completed by other contractors (e.g., not under EdData II)

Ongoing feedback loops from US AID and input fron1 selected Mission staff continually improved its functionality this year, during which DEP-AME:

• Completed the Beta release of the EGRA Barometer for launch and submitted it to US AID.

• Developed a video to showcase the EGRA Barometer data content and use.

• Presented the EGRA Barometer on January 29 to the USAID Education Sector Council.

• Presented the EGRA Barometer with USAID on March 18 at the World Bank Community Discussion on Measurement & Data Issues in Early Grade Reading, which focused on updates on measurement and data issues in early grade reading.

• Presented the EGRA Barometer on March 31 at USAID's Innovation Lab launch.

• Began planning for Phase 2 of the EGRA Barometer for submission of a proposal to USAID in the next quarter.



During the first quarter, RTI worked on the design and development of the EGRA Barometer tool, which allows real time manipulation ofEGRA results, provide measures of average performance, show full distribution of student performances, demonstrate the relationship between levels of performance on various EGRA subtasks, and facilitate setting standards for student performance in reading using EGRA data.

In November, DEP-AME delivered the Beta version of the EGRA Barometer Tool with data for Egypt, Iraq, Jordan, Morocco, and Yemen for USAID review. Its three modules focused on using data to (1) understand existing levels of reading achievement, (2) generate projections for how reading interventions could impact learning outcomes on an increasing scale, and (3) measure the impact of reading programs.

During the second quarter, DEP-AME completed the Beta version of the EGRA Barometer, showcased it at several key meetings for USAID, and submitted a work plan and proposal for Phase 2 (which is now piloting the instrument for

EGRA Barometer Highlights:

• The number of users increased in June (80) and July (110) from May (41)

• The number of sessions increased in June (121) and July (152) from May (103)

• While the number of users and sessions increased, the Average session duration decreased: June (3:41), July (3:19) vs. May (4:01)

• The highest number of sessions each month is from the USA (81 in July)

• The second highest in July is Brazil (24), followed by Nepal (8)

• The second highest in June is Indonesia and Canada (9 each)

• To date, the highest number of sessions is from the USA (616), followed by Brazil (26), and the Philippines (19)

further development and field application). Presentations on the EGRA Barometer this quarter occurred on January 29 to the USAID Education Sector Council; presentation of the EGRA Barometer with USAID on March 18 at the World Bank Community Discussion on Measurement & Data Issues in Early Grade Reading; video demonstration at the USAID/Africa Bureau Africa Regional Education Workshop (AREW) on March 24, and presentation of the EGRA Barometer on March 31 at the USAID Agency's Innovation Lab launch.

The Barometer was also presented in Indonesia in August by Peggy Dubeck and Alastair Rodd; in the Philippines twice (in July and August) by Joe DeStefano and Sarah Pouezevara. It was also presented in Colombia in June by Carmen Strigel and Amber Gove.

Phase 2 key activities include:



Task 1: Dissemination and Outreach

This task addresses Pilot objectives 1, 2, and 3 (Share, Teach, and Learn). A robust and multi-faceted dissemination strategy is essential to realize the potential of the Barometer to facilitate policy dialogue and program planning. DEP-AME began undertaking activities, beginning with the development of:

• Develop and implement a dissemination plan.

• Leveraging social media, such as Facebook to engage as many stakeholders as possible.

• Establishing a network of trusted dissemination partners to maximize Barometer reach.

• Conducting up to four (4) presentations in Washington, DC (or other venues) for key audiences as requested by US AID.

• Capturing profiles (vignettes) of how countries use the Barometer and creating links to these profiles within the Barometer so that other users can see examples ofhow the tool is being used. These profiles (vignettes) will be sent out as electronic newsletter (E-News).

• Gathering user testimonials for posting on the Barometer's home page.

• Preparing quarterly E-News and disseminating them. Training. Conducting in-country training and capacity building workshops around using the Barometer to facilitate policy dialogue. Workshop participants include US AID education staff and other counterparts or stakeholders designated by the identified USAID Mission.

• Website Analytics. Real-time monitoring of user activity has been underway to provide an important vehicle for tracking progress and improving the effectiveness of the dissemination strategy. Monitoring of website activity (for example, number of users, the pages they viewed, and so on) was collected using Google Analytics and tracked, as well as the number of times the Take a Tour video is played with data from Vimeo. This information has been supplemented for US AID by summaries from the user account signup portion of the website, and the responses to the online user survey.

Next Steps

• Monitor data and prepare a monthly activity summary for USAID.

• Report on dissemination efforts and outcomes.

• Prepare a fmal report on the effectiveness of the dissemination efforts and lessons learned.

34 Annual Report Year 3 (01 Odober 2013- 30 September 2014)


Task 2: Data

This task addressed Pilot objectives 4, and 5 (Add and Understand). While the first data task began incorporating additional EGRA datasets collected by RTI under EdData II , this second involves determining the level of effort associated with incorporating additional early grade reading assessments from other agencies contracting with USAID. This began during the later quarters of the fiscal year. When completed, the end products will allow US AID to assess the advantages, both in terms of cost and time to dissemination, of requiring contractors to use a set of uniform standards for measures for EGRA studies and delivering data files in a format optimized for easy transfer to the Barometer. Activities begun under this task include:

• New RTI Assessment Data. EGRA's from Nepal, Philippines, Indonesia, and Egypt were added to the Barometer. Incorporating these new data required reviewing distributions, examining associations among subtasks, creating analytic variables for the web tool, and comparing results to any existing reports about the particular country's data. The scope and number of assessment measures and the number of grades and languages in each assessment have varied by country, which has required modifying the Stata programs that produce the data for the web application.

• EGRA 's Conducted by Other Organizations. USAID has additional early grade reading assessments designed and collected by other organizations. Although there has been agreement among implementers to a common framework, efforts to achieve such standardization are in their infancy. DEP-AME began work this year to design a framework for data integration into the Barometer and began investigating the level of effort required to add two non-RTI datasets to the Barometer. While the work required is similar to those performed for adding a new R TI EGRA dataset, since the data are from another organization these activities will involve greater complexity. DEP-AME efforts to address these complexities confirms the need for even greater data cleaning preparation prior to transmitting the information to RTI.

• Data Specification Document and Instructions for Data Providers. In addition to including sample datasets in the Barometer, DEP-AME produced a draft data specification document for use by organizations contracting with US AID to conduct EGRAs. The document provides clear definitions of each subtask to be incorporated into the EGRA Barometer; the format for each of these measures, including variable names, labels, and types; and a list of, and structure for, additional information needed for the technical and methodological appendices included in the Barometer.

Task 3: Application Development

This task focuses on the part of Pilot objective 4 for increasing the functionality of the Barometer. Application development involved a number of extensions to the analytic and



presentation capabilities of the tool, allowing broader and deeper examination of early grade reading data by US AID staff and host country partners. DEP-AME planning focused on the following critical areas:

• Benchmark Projection Methodology. The EGR Barometer used data from the Egypt Girls' Improved Learning Outcomes (GILO) study, a grade 2 reading intervention, to model projected changes in the percentage of students meeting oral reading fluency targets (set by the user) through 2020. As additional EGRA data become available, DEP-AME used the opportunity to develop alternative benchmark projections and to build the datasets. New data for existing countries allowed DEP-AME to begin to explore alternative approaches for calculating growth, and to examine such options for improving and making more appropriate the projection methodology used in the benchmarking portion of the Barometer.

• Trend Analyses. As new waves of data for existing studies became available or new studies conducted in countries that have already completed an EGRA, DEPAME began to analyze change over time and provided graphics and tabular displays highlighting those areas where progress had been made and others where it lags behind. This involved adding another feature to the Barometer that allows the user to compare the results of assessments conducted at two points in time in the same country.

• Side-By-Side Comparisons. Currently, performance on the various EGRA subtasks is presented separately by grade and by language. In many cases, users may benefit from being able to compare results for different languages or grades on the same graph. For example, the distributions of scores in oral reading fluency for grades 2 and 3 could be displayed on the same graph, potentially illustrating how scores shift from one year to the next and providing a concise view of a year's growth in the particular skill area. Similarly, side-by-side comparison of EGRA outcomes in the different languages in which a student was assessed (like in the Philippines) or among different language groups (where different students are assessed in different languages) may be of particular interest to US AID and host country staff This has required adding considerable functionality to the existing results feature. DEP-AME has developed features dedicated to making compansons.

• Stand-alone Versions of the Barometer. As new datasets are added to the Barometer, DEP-AME worked to develop a stand-alone versions that does not require an Internet connection and which can post them for download.

• More Reading Resources. In response to feedback already received, DEP-AME expanded the Resources section to create a richer collection of information that can be used as a reference and as a tool in advancing the progress of all children in learning to read.

• Content Management System. The capacity of nontechnical staff to modify the Barometer text (not EGRA data) is being developed for use as a customized

36 Annual Report Year 3 (01 October 2013-30 September 2014)


Content Management System (CMS). As the Barometer is revised, it will be updated to ensure US AID staff retain this ability.

• Tool Documentation. DEP-AME prepared detailed technical documentation to ensure the Barometer can be managed by another vendor or USAID technical staff in the future. The documentation will be updated whenever major changes are made to the Barometer.

• Other Features. DEP-AME also began to respond to requests for additional functionality by users and/or USAID. RTI will work with USAID to develop a scope of work for such new features if and when they are deemed necessary to add to the application.

Task 4: Site Maintenance and User Support

This task underlies all the Pilot objectives to ensure that the Barometer is always available and functioning as expected, regardless of changes in web browser versions. Throughout the year DEP-AME monitored site maintenance and user support.

Task 5: Tablet Application

This task addressed Pilot objectives 1, 2, and 3 (Share, Teach, and Learn). To ensure the Barometer is available for a wide range of users, DEP-AME will develop a tablet version. The tablet application will mirror the functionality of the downloadable version of the website. It will be redesigned so that the user interface can accommodate the common screen resolutions of tablets, write necessary iOS- and android-specific code, conduct comprehensive quality control testing, work with Apple through application review and approval, and launch the applications. Further work on this will continue. The application will target the most common tablet hardware and operating systems, including: iOS : iPad Air, iPad 2, and iPad Mini with Retina Display, Android: 7-inch tablets (similar to Nexus 7), and 10-inch tablets (similar to Galaxy Tab 10.1).

Task 6: Other

Additional areas of work were mentioned in the early rounds of feedback when the Barometer was being demonstrated and tried out. The first concerned whether it would be possible to add more contextual data/information about each country. Some suggested adding information on education expenditures or other education sector information. Adding such data would involve identifying reliable sources and creating standardized ways to important and display such information. The second issue raised during the initial demonstrations of the Barometer concerned making it available in languages other than English. Thus, DEP-AME will prepare the materials necessary to obtain cost estimates for creating an Arabic and a Russian version of the Barometer for the coming year.

Next Steps

Monitor use and refme the instrument and populate new country data throughout the pilot period.



MOROCCO: 2011, Arabic

ThiS tool faolttates the use of Early Grade Readmg Assessment

(EGRA) data to:

• Analyze ex1stmg levels of readmg ,1eh1evement and the factors that are associated w1th different levels of oerformance;

• Set reasonable standards for readmg performance. gsven the oasehne d1stribution of scores on various EGRA subtasks; and

• Consider scenanos for h~ety traJectones of trnprovernent m that

performance.

RESOURCES

• Readt'1£ DPVeloprnent

• About E:GRA

• About the Data

Questions? Jnfo@nameofo•g org {hn;.. oenO·ng)

l\";~tJKrr•""l "i~·~*"'l'·"f{l~. nw u~F~~l' w),'1m1 ""'t>l.~i\~

~~' Homf! "' '\ Reporu ;.;. ResQurces t, AbotJt ' 1 ~. " ! F , ,t ~ t

• BETA VERSION » Morocco

Regions Included in Study

(h~u.., Ov~nl•it~•

Oov.~.:t

M~bKn lf'f'iS<ft NH.lOul

Not mc..Juded m Hudy

Grand Casoblanta

Eljadoda

Doult.!<ala

GROSS ENROLLMENT RATIO

Total enrollment 1n a educat1on level. regardless of age. expressed as a percentage of the population of :he OffiCial age tor lhat level Can exceed lOO% due to Inclusion or over-aged and under-aged

students bec.3use of early or late schoof entrance and grade retention.

58% 115% 70% 2012 Pre-primary Primary s~condary Data Year

.:J Go

Example of EGRA Barometer Refinement



MOROCCO: 2011, Arabic Benchmarks for Oral Reading Fluency

a Select gradr .:J SHOW

D Choose Number of Words Per Minute (WPM)

• Most recent assessment levels at

14%

24o/o

2o/o

llliD Moroccan grade 2 students

read1ng at or ao0\!1! 35 WPM

Average read1ng c:omprehtol"ls on

for students at thts oral read1ng nuency level

Esnmated probability that a

student with this orat read•ng

ftuency scor-e w1ll bt aole to comprehend 80~ or morP of

passage

« BETA VERSION • Morocco .:JGo

0 P"'JKIIOnJ •r1'1h0111m on 'I fot' 1 He 2 "•lard~' o' th• ar•dc s..tecutf

FOf CO~.tt'ltt')' IUrHmtnts W thout pad• 2._ JttojKt:lonJ. mil~ be baud Oft other lGI.A WNt:fl wtlh I'..S.l tlata

Intervention Egypt GILO Gr~de 2 .:J 100 GIL0Impt011oment

- Partral GILO tmpn>vom<t'lt (SOO..) 90 • Base ca .. (no 1ntorventoon)

80

70

i 60

~ so ~ ~ 40 Air

30

20

10

2011 2014 2015 2016 2017 2018 2019 2020

• Starting figure ts from the most current survey. AH other data are est1mates.

Quesuons for Discussion Morocco~ Kmg Mol1dmmed VI aeclate'd educatiOn"

pnonry S«ond only to Morocco's terfltOn.JI mtegnty. A clearer

understandmg of how children are lcoammg tusk mdrhemaCIC'S

and r~ddtng slc!lls m rhe early grJdes IS an essNWal first step m

The •part.Jol GILO'"' est1mate shown below assumes that

cond•t•ons for a nataonal •mplementauon of the GllO program

W1llltkely not ach~ the same results as a controlled tnal

improvmg student pt>riori'Tk1nce From Sl Jen P~rtormanc e m What m gh be dont to tnctease the- hke11hood that more of the d I ga•ns aSSOCiated With GILO ca~ ~ captured natJOnally?

The figure above- shCYorS three e-sumates for the •mprovernem m Are there othe, factors you can thtnk of that m!£ht accelerate the percernage of students meeung the selected benchmat1t progres$ towards a<htevtng Morocco·s nattooal benchmark

These esnmates are based 011 the G1r!s· Improved Ute racy goa~?

Outcomes (GILO)study. l""' Mo•• (PDF, 940 K8) M •!hOdo ogy

f r Pro

Example of EGRA Barometer Refinement

Annual Report Year 3 (01 Odober 2013 - 30 September 2014) 39


..

111 - . ,_. - ·-·

r •• . ~- . . . -.- ~ . - -~

-

. .. :1

I I , ~ I ,

il 11 r :. I ,

~ I , ~ I , I I t-


Coogle AIKI ytics

Audience OveJView

Overview

• Ses5ioos

Sessions

176

All Sessions 1CODJ'!t

"\_,.-....,...~-.........,--

Pages 1 Sessioo

3_56 ·~.._A__rA~~

% New SessiOOs

36_36o/o """""' ,-/~~· .r----...

2.. fr

3_ c

4. fr-fr

Users Pageviews

85 626 ...... \./"""'" ~A.--~_,.. ... \ .... \ ~___........_ ...r--- __,

JWg. Sessicn lba1ioo BotJIICe R~

00:03:56 56.25o/o '-"'.. /\_)\ .E--... ,..._ _...! ,.,j•v--1''../"V ~-~



Early Gm:le Reti!Ola r - h ~ -- Go to 1his report MWI!GS~OIIhl

Apr1 ,2014 -Apr J0,2014

• Rebni!!g Visiter • New ViSitnr

172 Q7.73%

2 I 1.14%

1 I {1.57%

1 I o.57%

41


Barometer Data Usage Summary-May 2014

Google A ~a ytics

Audience Overview

All Sessions 100.00~

Sessions

103 -~--- ............

Pages. J Session

4 .. 20 "-......,.._..___ ..... . /\,

% New Sessions

29 .. 13°/o ....... /'/'1 ,--.. /'-......

2.. ~-gc>

/""'

f-......

,-"'\

r.tayS IMIIY 15

Users Pageviews

41 433 _ ...... _,,./'·, ... J'--...._ /'-..

_.,.........._ -- _../\____,~

Avg. Session Duration Bounce Rate

00:04:01 60 .. 19% "\, ......---... ,~~ ,../\ l"- -".Jv-" ~

E.artr G111111e Ree<Diw ~~r - tt ~.. Go to this report M\\lebSitr-01111!

May 1, 2014- May 31 , 2014

• • • •

• Returning VisitDr • New Vrsh

~". /"....

A /v\

Sessions 1!11. Sessions

100

3 I 2..91%

42 Annual Repott Year 3 (01 October 2013 - 30 September 2014)

Barometer Data Usage Summary-June 2014

Googl~ A'la ytics

Audience Overview

Al l Sessions - 1J . c~...,

• Sessions

20

J .. .

Sessions

121 1'- /\._J\f\..r-..,-.J'

Page!> f Session

3.74 !'---.... .,--... ../''v -..f'....--..l'w-

% New Sessions

55.37o/o r-v .... ,(-1 \r'-J··.l·\-/~

1. 8'1-'J'S

3_ p;-br

4. e5-€S

Users

80 /'--'\ /'--.!~'\/' ,....,.,._rJ

Avg. Ses:siOIIl· Duration

00:03:41 ~~\ /\,

Pagelliews

453

~~ + Add Se{Jnent .._ .

/'--... ,......._ .. /'.J\ ...-/\I

Bounce Rale

55.37o/o r\1\. ;--.. r-'".N \ /'

Annual Report Year 3 (01 Odober 2013 - 30 September 2014)


Ellrtl' Gfade R"elfho e.amn:.e-tJ!r- h ~- - Go to this report 1'.1 W"l> Sib! O nlll

Jun 1, 2014 - Jun 30, 2014

• New Visitor • Rehlming V"ISi!Dr

Sessions .,. Sessions

113

5 I 4.13%

2 I u 5%

1 I o..83%

93.39%

43


Result 4: USAID Staff Knowledge Strengthened


During the first quarter DEP-AME continued preparations for face-to-face Evaluation for Education Projects Workshop, originally planned for February 2014 in Bangkok, Thailand at the USAID Regional Training Center. In October, the proposed timeline and update was submitted to USAID including proposed evaluation experts to be considered as workshop presenters. Dr. Ben Castleman of the University ofVirginia was selected as the consultant to design and provide the training. A complete planned schedule was prepared and submitted to USAID. An application for the USAID/Bangkok Regional Training Center was also submitted to secure the date and venue. The training date for this first classroom training was confrrmed for February 24-26, 2014.

In January, confrrmation was received of the postponement of the planned February training at the USAID Regional Training Center. An Annotated Agenda was submitted at this time in January for USAID review and feedback. Recommendations from USAID on the importance of non-experimental methods, evaluation in program design, and results dissemination were received and reviewed, including these:

• Summarize the current and available reading research to clarify what we know works, and to identify gaps in research and evaluation.

• Expand coverage on non-experimental methods, including pair matching and value-added modeling, and on how to incorporate evaluation into project design.

• Develop an overview and summary of how to review and interpret evaluations.

• Assess communication approaches for diverse audiences on how to communicate technical info to a largely non-technical audience in a clear and compelling way.

Based upon this input, research activities continued with Dr. Ben Castleman although a new date had not yet been confrrmed. During the third quarter a November 2014 timeframe was confrrmed, during which Dr. Castleman would no longer be able to serve as the course trainer. In response, RTI identified a replacement instructor Dr. Melissa Chiapetta of Social Impact, who was quickly contracted to provide the training. Meetings were held throughout the latter part of this reporting period to refme and complete the training agenda and to meet with USAID and Dr. Chiapetta, and with Allegro to complete the training design. Throughout the fourth quarter the training materials were refmed and presentations prepared for the face- to-face training, confrrmed for November 17-19, 2014.

Toward the end of the Fiscal Year the course content was being further refmed, with the goal that the participants would be able to:

• Describe the importance of evaluations for education programs, projects, and activities

• Differentiate between qualitative and quantitative data collection techniques and describe value of each



• Differentiate between performance and impact evaluations and understand when each is appropriate

• Differentiate between experimental, quasi-experimental, and non-experimental evaluation designs as well as strengths and limitations of each

• Manage an evaluation from design and procurement to dissemination of results

• Write an effective SOW for an evaluation and critically review evaluation deliverables

The course designed was premised on USAID 2011's Evaluation Policy and the ADS and was designed to address these core considerations:

Accountability

• Did the project/activity deliver the inputs and outputs it was contracted to? Why or why not?

• Were funds used efficiently?

Learning

• Did the pilot project/activity prove effective at achieving the outcomes and impacts it targeted? Why or why not?

• Which aspects of the project/activity were most effective or cost effective?

• In which contexts would a similar project/activity work well in the future?

• How might we improve the sustainability of projects/activities?

The table below presents an excerpt from the training presentation, which will be covered more fully in the next reporting period.

Next Steps

Complete the training course design, materials, and face-to-face instruction.



Coverage Monitoring Question Examples Evaluation Question Examples

Inputs

(Activities, Resources, etc.) How many participants were invited to the How effective were those inputs? Did we target the right inputs, or were there others training? How was the program resources we should have provided or activities we should have delivered? How well implemented? Was implementation in was the program implemented? For outreach, did we use the best avenues and accordance with design and specifications? methods we could have? How well did we access hard-to-reach and vulnerable

Outputs

(Products, Services, Deliverables, Reach)

Outcomes

How many people or communities were reached or served? Were the targeted numbers reached?

(Usually the medium-term What has changed since (and possibly as goals of the project; things that a result of) program implementation? How happen to people and much have outcomes changed relative to communities) targets?

Impacts

populations? Did we reach those with the greatest need? Who missed out, and was that fair?

How adequate was program reach? Did we reach enough people? Did we reach the right people?

How substantial and valuable were the outcomes? How well did they meet the most important needs and help realize the most important aspirations? Were they not just statistically significant , but educationally, socially, economically, and practically significant? Did they make a real difference in people's lives? Were the outcomes worth achieving given the effort and investment put into obtaining them?

(The longer-term, higher-level Did impact-level measures or indices To what extent did the project contribute towards changes in impact-level measures or results of the project; things change over the same period of time as the higher-order results? that happen to sectors, project (cause is not evident)? economies, countries)



Overarching DEP-AME Next Steps

While DEP-AME has had a highly productive year and enjoyed robust activity, some implementation has proceeded more slowly than originally anticipated. This is due in large part to the fact that activity plans and schedules must align with US AID Missions ' schedules and priorities. And, the close work with in-country partners, often results in extensive delays due to scheduling, security, or simply the need for host country policy dialogue to progress. As an example, this year delays were encountered in Bangladesh, Nepal, Tajikistan/GBAO, and in development of the Evaluation Training Course. The ongoing activities that are slated to continue under Result 1 and to be completed in this coming fiscal year include the following:

• Impact Evaluation Training Course

• Tajikistan Gorno-Badakhshan Autonomous Oblast (GBAO) Early Grade Reading Assessment.

• Early Grade Reading Assessment (EGRA) Barometer

• West Bank EGRA

• Philippines Reading

Other activities also remain within DEP-AME's scope of work which have not yet been implemented, including development or refmement of evaluation tools, and implementation surveys and assessments in response to Missions' data needs. Given the need to work closely with missions, it can take several months to schedule activities and they in tum are working closely with their ministerial counterparts for country-buy-in. Once activities begin they routinely take from 6-9 months.

Under Result 2, this includes a second trends analysis (needed to measure the impact of USAID's regional efforts) .

Under Result 3, this includes the EGRA Barometer.

Under Result 4, this includes completion of the existing Impact Evaluation training and development of an additional training course.



Annex A: Financial Summary

RTIInternationai- Yearly Financial Statement, Task 15 (Year 11) Contractor/Recipient: RTIInternational Data for Educational Programming in Asia and Middle East (AME)

Award No. : Prime Contract AID-OAA-BC-11-00001 Performance Period: September 29, 2011 November 30, 2014 Yearly Financial Report as of September 30, 2014

A B c D E F=C+D+E Original or

Revised Total Total Total Total Results Estimated Obligated Amt. Expended as Expended Outstanding Expended to

Description Cost to Date of (March '14) (Apr-June '14) Commitments Date Data for Education Programming

[REDACTED] in Asia and the Middle East (DEP-AME)

Annual Report Year 3 (01 Odober 2013 - 30 September 2014)

G = B-F H = A-F

Balancing Balance Remaining of Remaining of

Obligated Total Est. Cost

A-1

summary annual progress reportpdf.usaid.gov/pdf_docs/pbaaf528.pdf · summary annual progress report...

Documents