education innovation fund...host classroom is used by the tutor based in islamabad to connect...

Education Innovation Fund

Project Title: Ilm on Wheels

Grant Award Number: RFA2012-IN01-FOG03

Quarterly Progress Report July-September 2012

10th Oct 2012

Submitted to:

Ilm Ideas Programme

Submitted by:

TeleTaleem (Pvt.) Limited

Plot 291, Street 3, I-9/3, Islamabad

Phone: +92 51 111 784 784 • Email: [email protected] Website: www.teletaleem.com

TeleTaleem – Ilm on Wheels Quarterly Progress Report ii | P a g e

TABLE OF CONTENTS

ACRONYMS ...................................................................................................................................................... III

EXECUTIVE SUMMARY .................................................................................................................................. 1

INTRODUCTION ............................................................................................................................................... 3

GEOGRAPHIC COVERAGE ............................................................................................................................. 4

ACTIVITIES AND ACCOMPLISHMENTS .................................................................................................... 5

QUARTERLY INDICATORS UPDATE .......................................... ERROR! BOOKMARK NOT DEFINED.

FINANCIAL SUMMARY ................................................................................................................................. 12

TRAININGS AND MEETINGS ...................................................................................................................... 13

TRAININGS ...................................................................................................................................................................13 MEETINGS ....................................................................................................................................................................13

NEXT QUARTER WORKPLAN .................................................................................................................... 14

ANNEXES .......................................................................................................................................................... 15

TeleTaleem – Ilm on Wheels Quarterly Progress Report iii | P a g e

ACRONYMS

ADO Assistant District Officer E&SE Elementary and Secondary Education EDO Executive District Officer EMIS Educational Management Information System ESRU Education Sector Reform Unit ICT Information and Communication Technologies IRT Item Response Theory KPK Khyber Pakhtunkhwa M&E Monitoring & Evaluation MOU Memorandum of Understanding NEAS National Education Assessment System PEAS Punjab Education Assessment System SLO Student Learning Outcome T2 TeleTaleem

TeleTaleem – Ilm on Wheels Quarterly Progress Report 1 | P a g e

EXECUTIVE SUMMARY

The Ilm on Wheels project is targeting enhanced learning achievements in Mathematics for student groups in Grade- IV and V, by connecting them to the widely available, but inaccessible, online learning content and tools. The goal of the project is to leverage Advanced Learning Technologies to attain broad based access and improved quality of education in public primary schools. The project is currently being implemented in 10 student groups (5 Grade IV classes and 5 Grade V classes in the schools of Balakot in Mansehra district. Implementation is being carried out in close coordination with local education authorities. Since the project inception, TeleTaleem has tried to engage with all relevant stakeholders at various levels. At the highest level, project details were shared with the Minister of Education KPK, Secretary Elementary & Secondary Education, Director Education Sector Reform Unit and other key officials at the provincial level. The project received extremely healthy feedback from the provincial education administration, culminating in signing of a formal MOU between TeleTaleem and E&SE Department for the execution of the project. At the district level, interactions were held with EDO E&SE Mansehra, who helped identify Balakot as the target region for this intervention. ADO Balakot was designated as a focal person to coordinate with TeleTaleem from the Govt. side. The project is based on the use of a mobile satellite-enabled Van to provide ICT enabled learning services like teacher training, student learning and assessments, supported by an advanced learning environment. The mobile Van, a.k.a. School Garee, delivers exciting and engaging learning practices and content currently not available to students. The daily teaching learning activities cover four major components, split over in-class and after-class activity. In class, curriculum-tied educational videos are played back, followed by solving practice questions, moderated by a live tutor connected remotely via School Garee. Post class session, students are handed out tablet PCs to conduct personalized practice sessions, leading to instantaneous generation of a formative assessment report. This report is shared and discussed with the math teacher, by project’s Academic Coordinator, to identify corrective actions (class-wise and student-wise). A summative assessment on topics covered during previous sessions is also conducted. The session ends with the students using the tablet playing educational games, loosely tied to the subject/curriculum, for reinforcing the learning process and motivational level. Content has been developed using Khan Academy’s videos as the base material. The videos were mapped to the national curriculum, unit-wise and SLO-wise. The videos were also adapted for local use by replacing English with Urdu text in the existing videos. A large number of missing videos (approx 72) were created from scratch. On the assessment front, efforts have been made at multiple levels. SLO-tied question bank has been generated which is being used in self-paced student assessment. Practice questions for in-class sessions have also been developed. Major large scale efforts were done in formulating appropriate standardized pre and post-intervention assessment. Services of a consultant from NEAS were engaged to spearhead this effort. Mapping of competencies and cognitive skills was carried out in accordance with the national curriculum. A large number of very seasoned teachers were engaged for coming up with a large pool of question items, which were then put through sample testing in select schools in vicinity of Balakot. The results were analyzed based


on IRT to short list the appropriate quality and number of questions for the pre and post assessments. Pre-Assessment was subsequently conducted in all control and treatment classrooms. The results are being analyzed by a team of professionals and will soon be published in the Baseline Assessment Report, attached in Annex-1. We are in the 4th week of training cycle and so far the response from all key stakeholders has been quite encouraging. This includes the local ADO, male and female teachers and students. There have been multiple instances where local teachers have offered their homes to be used for running the class when inclement weather and overflowing streams have prevented the School Garee from reaching the school premises.

Figure 1: Class taking place inside Teacher's Home


INTRODUCTION

The Ilm on Wheels project is targeting student groups in Grade- IV and V, for enhanced learning achievements in Mathematics, by connecting them to the widely available, but inaccessible, online learning content and appropriate teaching-learning practices. The goal of the project is to leverage Advanced Learning Technologies to attain broad based access and improved quality of education in public primary schools.

The primary point of access to/from these schools is through an innovative, fully self-contained

satellite-enabled mobile van, nicknamed School Garee. The van equipped with tablet PCs,

multimedia and power sourcing equipment is being used to provide connectivity to learners

and teachers in individual and class/group settings. The unavailability of core infrastructure,

support network and supporting budget favors the transactional use of portable ICT infrastructure

over multiple schools.

Using this mobile platform coupled with abundant online resources, the project aims to improve student learning outcomes by enhancing the teaching and learning process and competencies. The project is currently being implemented in 10 classrooms (treatment group) in the schools of Balakot in Mansehra district. Direct Beneficiaries:

Students (Grades IV, V): 300

Indirect Beneficiaries: Students (Grades I, II, III): 530

Teachers: 28

Parents & Local Communities

The project also intends to conduct an assessment of this approach in a realistic and manageable context in Pakistan. A pre and post intervention assessment for both treatment and control groups will help evaluate outcomes of the intervention. Purposive sampling methodology has been adopted for selection of treatment and control groups to establish the difference brought about in learning achievements of students specifically due to TeleTaleem intervention. For pre-assessment 100% of the targeted schools and learners have been assessed based on a standardized test. An identical number of additional schools in adjoining locations are being used as the control group. Assessment results will be published in the baseline report.


GEOGRAPHIC COVERAGE

Treatment Schools Control Schools


ACTIVITIES AND ACCOMPLISHMENTS

PROJECT INITIATION Project Offices for the duration of the Project Project office was established within the TeleTaleem head office located in Islamabad. A field office was established by renting space at Pine Track Hotel near Balakot. The field office serves as the home base for School Garee, and also houses the driver and field technical support staff. Equipment For Host Class Rooms at T2 Premises A host classroom was created and furnished within the project office in Islamabad. The room has been equipped with all necessary equipment required for conducting daily class sessions. Host Classroom is used by the Tutor based in Islamabad to connect remotely with schools. Hiring of Core Project Staff The core project staff included Academic Coordinator, Tutor, M&E Expert, Content Specialist, Backend Technical Support Engineer, Field Technical Support Staff and van driver. All of these staff members have been hired. The content specialist, backend technical support engineer and Field Technical Support Staff were hired from TeleTaleem/Comcept’s existing team, while remaining were hired externally. Activities for Academic coordinator were split in two distinct sets; with two different persons serving different time periods. Hiring of Project Management Team The project management team includes TeleTaleem’s CEO, Project Director and Admin & Finance Executives. These team members are contributing partial time to the project. Orientation of Project Team The orientation of project team was conducted at various stages. Project brief was shared with all team members followed by a face to face orientation session involving relevant team members. For staff requiring field exposure, on-site trips were conducted and meetings with stakeholders arranged for better understanding of project requirements, in local settings. Development of project management framework A detailed project activity plan was developed using Microsoft Project, which included activities, time lines, roles and responsibilities of different team members. Project review meetings are held weekly to discuss and update activities wherever required. More recently, daily progress reporting has also been initiated. PROGRAM DESIGN & FINALIZATION Consultation with relevant stakeholders Since the project inception, TeleTaleem has tried to engage with all relevant stakeholders at various levels. At the highest level, project details were shared with the Minister of Education KPK, Secretary Elementary & Secondary Education, Director Education Sector Reform Unit and other key officials at the provincial level. The project received positive feedback from the provincial education administration, culminating in signing of a formal MOU between TeleTaleem and E&SE Department for the execution of the project. TeleTaleem would specially like to acknowledge the tireless efforts Mr. Riaz Bahar Akhunzada, Director ESRU, who went the extra mile to support TeleTaleem to ensure that the project received all the requisite govt. approvals to start on time.


At the district level, interactions were held with EDO E&SE Mansehra. It was with his help that we were able to select Balakot as the target region for this intervention. ADO Balakot was designated as a focal person to coordinate with TeleTaleem from the Govt. side. He has also proved to be extremely accessible, responsive and helpful. His undaunting support has enabled the project team to gracefully launch the initiative without any major impediments. School Selection Process & Criteria EMIS data for elementary and secondary education of Khyber Pakhtunkhwa was used in the identification of the primary schools in the tehsils of Mansehra and Balakot. The basic initial criterion in the selection of the primary schools of the district was the enrolments numbering more than 20 for girls and 40 for boys in the target grades (grade IV and V). District Mansehra carries a mix of summer and winter zone schools, i.e. winter zone school holidays are scheduled in winter season instead of the summer. Winter zone schools were not considered because 6 months of schooling had already transpired in comparison to 1 month for the summer zone schools. Hence our search within Mansehra zeroed in on Tehsils and UCs housing summer zone schools. Discussion with EDO Mansehra helped identify Tehsil Balakot as the focal area, including 5 UCs with summer zone schools, i.e. Balakot, Garhi Habibullah, Talhatta, Karnol and Shohal. These UCs are co-located and have decent enough road network to support Van access. A scoping survey of schools in the shortlisted UCs was carried out for collecting detailed information from the schools. This scoping survey (attached in Annex-2) was designed to identify and collect information on all public schools in the target area and to help in devising the sampling frame, the target and control groups. Scoping survey was also necessitated because of the gaps detected in the authenticity of the stated data, including EMIS. For example, the data logged in EMIS was at least a year old and had numerous omissions in terms of missing schools and incorrect entries. In addition some finer details only became apparent, once the scoping survey was done. All told, the scoping survey helped put a lot of corrective measures in the project planning early on. The data collection team trained on the questionnaire carried out the scoping survey in the premises of the ADO’s office in Balakot where a total of 48 head teachers were present at that time. Physical visits to 24 of these schools for Part II of the survey resulted in previously unidentified schools which were listed and scoped out to prepare the list of available schools. The representatives of the schools were required to bring their enrolment sheets as well as results of their respective classes for the past three years. Due to various factors, majority of the teachers could not bring the required documents and were later contacted to ensure completeness of the information. The data collected through the scoping survey was then digitized and processed to prepare the dataset for final selection. Enrolment and accessibility were the primary criteria for eligibility of a school. 22 of the 48 schools were removed from consideration due to low (actual) enrolment and inaccessibility. The remaining 26 schools were then segregated into the respective grades (IV and V) for both girls and boys primary schools. In all, 9 schools for girls and 15 schools for boys were identified as potential candidates and prepared for statistical analysis for pairing. Class enrolment, teacher qualifications, years of experience in teaching, and the past grades in Maths for the students of these schools were then analyzed through non-parametric tests to determine the factor that would allow pairing of similar classes in to the treatment and control groups. It was imperative that the target and control groups be identical in characteristics to establish the difference brought about in the learning achievements of students in the target group compared to the control group. Of the 26 schools, pairs of 10 (20 classes) were finalized for the intervention. Table below presents the final list of target and control groups.


Treatment Control

Type Name Students Grade Type Name Students Grade

1 Boys GPS Ghari Habib Ullah - A 42 4 Boys GPS Ghari Habib Ullah - B 35 4

2 Boys GPS Shoal Najaf Khan 45 4 Boys GPS Talhatta 48 4

3 Boys GPS Jagir 39 4 Boys GPS Sugdhar 20 4

4 Girls GGPS Ghari Habibullah 31 4 Girls GGPS Bagal Char 40 4

5 Girls GGPS Hassari 20 4 Girls GGPS Balakot 22 4

6 Boys GPS Jagir 20 5 Boys GPS Zameeri 22 5

7 Boys GPS Ghari Habib Ullah - B 30 5 Boys GPS Ghari Habib Ullah - A 35 5

8 Boys GPS Talhatta 32 5 Boys GPS Banphora 45 5

9 Girls GGPS Ghari Habibullah 31 5 Girls GGPS Narrah 50 5

10 Girls GGPS Hassari 25 5 Girls GGPS Balakot 15 5

Boys GPS Banphora 40 4

Finalization of Program Design This has been a major activity consuming a significant amount of time and resources of the key project team. Services of seasoned experts were also engaged, on transactional basis, for fine-tuning the pedagogy, content development, assessment methodology and analysis of results’ data. The daily session activity has been finalized to cover four major components, split over in-class and after-class activity. In class, curriculum-tied educational videos are played back, followed by solving practice questions, moderated by a live tutor connected remotely via School Garee. Post class session, students are handed out tablet PCs to conduct personalized practice sessions, leading to instantaneous generation of a formative assessment report. This report is shared and discussed with the math teacher, by the (project assigned) Academic Coordinator, to identify corrective actions (class-wise and student-wise). A summative assessment on topics covered during previous sessions is also conducted. The session ends with the students using the tablet playing educational games, loosely tied to the subject/curriculum, for reinforcing the learning process and motivational level. An in-depth session planning exercise was carried out to map curriculum with local text books and subsequently coming up with detailed session plans, outlining specific units, topics and SLOs to be covered on any particular day. Based on these session plans, teacher guides were prepared for each of the sessions which are shared with local teachers in advance. This helps them introduce topics to their students based on the new content, before School Garee visits that school. The topics and SLOs are further reinforced when the School Garee visits the school for teaching session. A sample teacher guide is attached in Annex-3. Content has been developed using Khan Academy’s videos as the base material. The videos were mapped to the national curriculum, unit-wise and SLO-wise. The videos were also adapted for local use by replacing English with Urdu text in the existing videos. A large number of missing videos (approx 72) were created in totality. Detailed session activity plans and teaching guides have also been developed to facilitate local teachers. On the assessment front, efforts have been made at multiple levels. SLO-tied question bank has been generated which is being used in self-paced student assessment. Practice questions for in-class sessions have also been developed. Major large scale efforts were done in formulating


appropriate pre and post-intervention assessment. Service of a consultant from NEAS were hired to spearhead this effort. He, in-turn engaged a large number of very experienced teachers coming up with a large pool of question items, which were then put through sample testing in select schools in Balakot. The results were processed through an appropriate statistical tool to short list the number of quality questions finalized for the pre and post assessment. Development of Pre & Post Assessment Tools The assessment tools have been developed based on national curriculum 2006, and comprising of multiple choices questions. Question items were developed according to the distribution of competencies and cognitive skills as defined in the national curriculum.

Test Specification Table for Grade IV Mathematics Curriculum 2006 for Pilot Test

Competencies Cognitive Skills

Number & operations (60%)

Measurement & Geometry (35%)

Information Handling (5%)

Total

Conceptual understanding (25%)

15 9 01 25

Procedural Knowledge (60%)

36 21 03 60

Problem Solving (15%)

09 05 01 15

Total 60 35 05 100

Test Specification Table for Grade V Mathematics Curriculum 2006 for Pilot Test

Competencies Cognitive Skills

Number & operations (52%)

Measurement & Geometry (40%)

Information Handling (8%)

Total

Conceptual understanding (25%)

12 9 2 23

Procedural Knowledge (60%)

28 22 4 54

Problem Solving (15%)

7 5 1 13

Total 47 36 7 90 For the development of items in these competencies, services of six skilled and trained item writers were hired. Assignment packages were prepared for these item writers based on SLOs defined in the national curriculum. The cognitive skills were already identified for them, which provided base for the development of items. Detail of the assignment packages for different writers for grade 4 and grade5 is given in the Annex-4 and Annex 5. After the completion of the assignment package, according to the above table of specifications, pilot tests were designed for grade 4 and 5. Four booklets were assembled for the pilot test for each grade. Almost four items were administered in the field for each learning outcome given in the curriculum. It is because the survival rate of retention of best fit items on the basis are very low.


400 items were piloted for grade 4 and 352 items were piloted for grade 5 for the retention of statistically best fit items. These pilot tests were conducted in both girls and boys schools. Around 600 students appeared in the tests. Since the tests covered full curriculum, and the objective of the test was to identify the best statistically fit and reliable items, as such Grade 5 students were tested on Grade 4 items and Grade 6 students were tested on Grade 5 items. A comprehensive Test Administrator (TA) manual was also prepared containing all techniques which are essentials for the conduction of standardized tests. This manual provided basic guide line for the new test administrators for the conduction of assessment activity (attached as Annex 6). Tests were administered at the same time in all schools by an external group of test administrators, who were provided basic training for the purpose. Local school teachers were not allowed inside the premises for the duration of the test. All test papers were collected and transported back to Islamabad on the same day. Data entry was completed in record time by professionally trained data entry operators under the supervision of TeleTaleem’s team. All the entered data was physically verified by a separate quality assurance team. Services of Dr. Muhammad Azeem, Assessment Expert & Coordinator-I for District Wide Large Scale Assessment associated with Punjab Education Assessment System (PEAS) were hired for conducting IRT based analysis on the results data. The analysis framework is attached as Annex 7. While we were able to get a large number of statistically fit items for Grade 4, the case for Grade 5 was totally different. The analysis resulted in only getting 35 fit items. Therefore, in order to assemble well representative assessment tool, the items were again reviewed and administered in the field. Re-piloting was done with around 200 boys and girls students. Data was entered and IRT based analysis was performed using the same criteria as before. After the analysis of re-pilot test, we were finally in the position to construct well representative Pre-Test booklet for grade 5. Development of M&E framework The primary objective of the internal monitoring function is to ensure that the achievement of project goals is undertaken with the highest quality, minimum risk and cost effectively through an enabling approach. The intents embedded in this objective are:

1. Provide assistance to stakeholders in the analysis of development progress, and; 2. Furnish an early warning mechanism which allows for the timely identification of

trouble spots and rectification of problems. The focus will remain on the basic elements of a complete monitoring system and thus the following structure will be used:

1. Monitoring of the timeliness of inputs and processes; 2. Monitoring of outputs/results: 3. Monitoring of assumptions and frame-conditions: 4. Documentation of lessons learnt and best practices: ensures that all positive or negative

results of the project are appropriately documented and disseminated in a timely manner for improvements in future activities.

Detailed M&E framework is attached as Annex-8


PROGRAM DISSEMINATION / AWARENESS CAMPAIGN Stakeholder Awareness Multiple stakeholder awareness sessions were held before and after the start of the program. All these sessions were organized with support from local ADO office, which facilitated arrangement of these sessions in Boys Middle School Shohal Najaf Khan. Head Teachers from 60+ primary schools were invited for a high level introduction about the program and its intended objectives. After the short listing of schools, more in-depth sessions were held with head teachers belonging to prospective control and treatment schools. The pedagogy was explained to head teachers and their feedback was sought, in order to incorporate any suggestions.

Orientation of Math Teachers Orientation of mathematics teachers was conducted during the month of September 2012, just after the final selection of control and treatment schools. Basic orientation covered both control and treatment schools, however in-depth training sessions were conducted for treatment school teachers. Mock in-class sessions were conducted and teachers were made to sit in the class along with students to observe the teaching-learning process. Teachers were encouraged to try out the teaching tools and material available in the class, in order to get used to ICT enabled advanced learning environment. TeleTaleem’s Senior Resource Person visited these sessions and interacted with teachers. Follow-up interactive sessions were conducted using School Garee as well.

Figure 2: Stakeholder Interactions (Head Teachers & Teachers)


STUDENTS' LEARNING ACTIVITIES Guided access to curriculum specific Maths related educational content Students’ learning activities were formally started in mid-September after reopening of schools from summer break. The daily session activity covers four major components, split over in-class and after-class activity. In class, curriculum-tied educational videos are played back, followed by solving practice questions, moderated by a live tutor connected remotely via School Garee. Post class session, students are handed out tablet PCs to conduct personalized practice sessions, leading to instantaneous generation of a formative assessment report.

This report is shared and discussed with the math teacher, by the (project assigned) Academic Coordinator, to identify corrective actions (class-wise and student-wise). A summative assessment on topics covered during previous sessions is also conducted. The session ends with the students using the tablet playing educational games, loosely tied to the subject/curriculum, for reinforcing the learning process and motivational level. Open & self-paced learning via Laptops available in T2 School Van Students are encouraged to access learning material and content at the end of the formal daily session. This includes educational games, curriculum specific videos to revise/reinforce concepts etc. Assessment of students Pre-Assessment has already been conducted. The methodology has already been explained in the earlier sections. In addition to the pre-assessment, periodic formative and summative assessments are also being carried out regularly. The results of these assessments are discussed with local teachers, so that they can take corrective actions. MONITORING & EVALUATION The project has an independent M&E consultant reporting directly to the CEO. Based on his findings, an internal M&E report has been compiled is attached as Annex-9 .

Figure 3: In-Class Sessions

Figure 4: Pre-Assessment in one of the schools


FINANCIAL SUMMARY

Description of line items

Total

Grant

Amount

Funds

released

to date

Funds

expended

to date

Funds

expended

during the

reporting

quarter

Percentage

of fund

utilised

Personnel 2,200,000 922,000 922,000 42%

Office Costs 170,000 36,380 36,380 21%

Activities 2,040,000 215,348 215,348 11%

Total 4,410,000 1,173,728 1,173,728 27%


TRAININGS AND MEETINGS

TRAININGS

SN Training Title Date Participants Outcome/results

achieved Male Female Total

1 Mock Training Session 15 Sept 2012

7 7 Teachers were given an overview of how the daily training sessions will be conducted.

2 Mock Training Session 17 Sept 2012

5 5 Teachers were given an overview of how the daily training sessions will be conducted.

MEETINGS

Meeting with Date Purpose Outcome/results achieved

Secretary E&SE, Peshawar 28 June 2012

Project Introduction Secretary Education was apprised about the intervention

Director ESRU, Peshawar 11 July 2012 Follow up on MOU formalization

Secretary E&SE, Peshawar 12 July 2012 Project Introduction to the new Secretary E&SE

Secretary Education was apprised about the intervention

EDO E&SE, Mansehra 20 July 2012 Project Introduction EDO committed his support to the project

ADO Balakot 31 July 2012 Project Introduction ADO committed his support to the project

Head Teachers of 60+ Schools at ADO Office Balakot

31 July 2012 Project Introduction and Stakeholder orientation

Project introduction conducted. Survey forms filled

ADO Balakot, Senior Head Teachers

6 Aug 2012 Stakeholder Meeting Gathered useful information about the needs of the teachers and present situation of teacher training in the area


NEXT QUARTER WORKPLAN

Act. No.

Activity / Task Description Expected Result(s) Responsibility

Schedule of Activities

Oct-2012 Nov -2012 Dec -2012

W1 W1 W2 W3 W4 W2 W3 W4 W1 W2 W3 W4

Activity-1 Students Learning Activities

Task-1 Guided access to curriculum specific Maths related educational content

Ongoing Learning Activity

Task-2 Open & self-paced learning via Laptops available in T2 School Van

Ongoing Learning Activity

Activity-2 Monitoring & Evaluation

Task-1 Periodic Monitoring (online/onsite) M&E Reports M&E Expert

Task-2

Task-3

Activity-3

Task-1

Task-2

Task-3

Activity-4

Task-1

Task-2

Task-3


ANNEXES

Annex-1: Baseline assessment report Annex-2: Scoping survey Annex-3: Sample teacher guide Annex-4: Assignment package for item writers, Grade 4 Annex-5: Assignment package for item writers, Grade 5 Annex-6: Test administration manual Annex-7: IRT analysis framework Annex-8: Internal M&E framework Annex-9: Internal M&E report – Sept 2012

education innovation fund...host classroom is used by the tutor based in islamabad to connect...

Documents