evaluation of dbe1 district interventions. application software quality and future development...
TRANSCRIPT
Decentralized Basic Education 1: Management and Governance
Evaluation of DBE1 District
Interventions
The DBE1 Experience: 2005 - 2011
December 2011
This report is one of a series of special reports produced by Research Triangle Institute (RTI), Implementing Partner for the USAID-funded Improved Quality of Decentralized Basic Education (IQDBE) program in Indonesia
Evaluation of DBE1 District Interventions
iii
Table of Contents
List of Tables ...................................................................................................................... iv List of Figures .................................................................................................................... iv Acknowledgements ............................................................................................................ v Abbreviations and Acronyms ............................................................................................ vi Executive Summary ............................................................................................................ 1 Recommendations .............................................................................................................. 5 Ringkasan Eksekutif ........................................................................................................... 7 Rekomendasi..................................................................................................................... 12 1. Introduction to the Evaluation ..................................................................................... 13
1.1. Background ................................................................................................................13 1.2. Objectives of the Evaluation .....................................................................................14
1.3. Methodology ..............................................................................................................15
1.4. Constraints in Undertaking the Evaluation ................................................................16
2. Tools for Educational Management & Governance .................................................... 17 2.1. Introduction ...............................................................................................................17 2.2. Clarification and Description of the Tools ................................................................17 2.3. Application Software Quality and Future Development ...........................................20
3. Evaluation of the Tools ................................................................................................ 21 3.1. Introduction ...............................................................................................................21
3.2. Do the Tools Work? ..................................................................................................21 3.3. Do the Tools Achieve the Stated Objectives? ...........................................................22
3.4. What is the Early Evidence of Outcomes? ................................................................26 3.5. Case Study: Asset Information Management System (SIMA) ..................................29 3.6. Conclusion .................................................................................................................32
4. Data, Data Analysis and Implementation .................................................................... 33 4.1. The Quality of Data in Indonesian Education ...........................................................33
4.2. Data Analysis .............................................................................................................33
5. Service Providers ......................................................................................................... 35 5.1. Key Questions ...........................................................................................................35
5.2. Service Provision .......................................................................................................35 5.3. The Need for External Service Provision ..................................................................37
5.4. Evaluation of Service Providers ................................................................................38 5.5. The Level of Interest and Need for the Tools and Support Services .........................38
5.6. The Experience of Service Providers with the Tools ................................................39 5.7. Networked Support Systems .....................................................................................40 5.8. Conclusions about Service Providers ........................................................................41
6. Sustainability ................................................................................................................ 43 6.1. Introduction ...............................................................................................................43
6.2. Threats to Sustainability ............................................................................................43
6.3. Addressing the Threats to Sustainability ...................................................................44
6.4. Models of Sustainability ............................................................................................45
7. Conclusion: The Development of District Management and Governance................ 47 7.1. The Role of Principles of Development ....................................................................47 7.2. DBE1 Principles for Technical Assistance ................................................................47 7.3. Lessons from Research ..............................................................................................48
Appendix 1: Evaluation Study Plan ................................................................................. 52 Appendix 2: Good Practice Examples in the Implementation of Tools ......................... 55 Appendix 3: Alternative Policies and Costs .................................................................... 58
Evaluation of DBE1 District Interventions
iv
Appendix 4: List of Institutions and Resource Persons Met ......................................... 59
List of Tables
Table 1: Evaluation of Tools in Evaluation Study Districts .............................................23 Table 2: Evaluation of Selected Tools Implemented by DBE1 ........................................25 Table 3: DBE1 Implementation of Tools .........................................................................26 Table 4: Administrative Outcomes from Using SIMA in Subang District ......................31 Table 5: Good Practice Case Study 1 – SIMA: Subang District ......................................55
Table 6: Good Practice Case Study 2 – SIMA & SIMPTK: Purworejo District ..............56
List of Figures
Figure 1: Analytical Steps ................................................................................................34 Figure 2: DBE Transition Strategy ...................................................................................45 Figure 3: Transition Strategy for Later Implementation of Tools (Year 4-5) .................46
Figure 4: DBE1 Assistance for District Management and Governance ...........................48
Evaluation of DBE1 District Interventions
v
Acknowledgements
We acknowledge with warm appreciation the considerable assistance given to us by the
numerous government officials, school principals and teachers in the districts and provinces
we visited and for the information they so readily gave us. We are also grateful for the
support given to us both in the field and in the Jakarta and provincial offices by the helpful
staff of DBE1. A full list of the people we acknowledge is provided in Appendix 4.
Evaluation of DBE1 District Interventions
vi
Abbreviations and Acronyms
AKPK Analisa Keuangan Pendidikan Kabupaten/Kota (District Education
Financial Analysis)
AusAID Australian Agency for International Development
BEC-TF Basic Education Capacity Trust Fund
BOS Bantuan Operasional Sekolah (School Operational Fund)
BOSP Biaya Operasional Satuan Pendidikan [School Unit Cost]
BPK Badan Pemeriksa Keuangan (Financial Audit Office)
DBE Decentralized Basic Education
DEO District Education Office (Dinas Pendidikan Kabupaten/Kota)
DPRD Dewan Perwakilan Rakyat Daerah (District Parliament)
Dinas Pendidikan
Kabupaten/Kota
Education Office of Local Government (District Level)
Dinas Pendidikan
Provinsi
Education Office of Provincial Government
Dewan Pendidikan Education Board
EMIS Education Management Information System
LPMP Lembaga Penjaminan Mutu Pendidikan (Provincial Education Quality
Assurance Institution)
M&E Monitoring and Evaluation
NAD Nanggroe Aceh Darussalam (Province of Aceh)
PBPSAP Penghitungan Biaya Pencapaian Standar dan Akses Pendidikan
(Calculation of Costs to Achieve Minimum Service Standards and
Universal Access)
Renstra Rencana Strategis (Strategic Plan)
SBM School Based Management (in Bahasa Indonesia, MBS – Manajemen
Berbasis Sekolah)
Evaluation of DBE1 District Interventions
vii
SDN Sekolah Dasar Negeri (state primary school)
SIMA Sistem Informasi Manajemen Aset (Education Asset Management
Information System)
SIMDA Sistem Informasi Manajemen Daerah (District Management
Information System)
SIMPK Sistem Informasi Manajemen Pendidikan Kabupaten/Kota (District
Information Management System)
SIMPTK Sistem Informasi Manajemen Pendidik dan Tenaga Kependidikan
(Personnel Management Information System)
SIPPK Sistem Informasi Manajemen Pendidikan Kabupaten/Kota (District
Education Management Information System)
SLB Sekolah Luar Biasa (Special School)
SMA Sekolah Menengah Atas (Senior Secondary School)
SMP Sekolah Menengah Pertama (Junior Secondary School)
SPM Standar Pelayanan Minimal (Minimum Service Standard)
TRIMS Tool for Reporting and Information Management by School
UNICEF United Nations Children‟s Fund
USAID United States Agency for International Development
Evaluation of DBE1 District Interventions
1
Executive Summary
Background
Objective of the Evaluation
This evaluation for the Decentralized Basic Education program component More Effective
Decentralized Education Management and Governance, known as DBE1, is to contribute to
maximizing the acceptance, widespread use and the implementation of the DBE1 results. The
focus of the study is on the acceptance, use and impact of tools to support the use of data in
the management and governance of education, particularly at district level.
Methodology
Conducted in August - November 2011, the evaluation team used four basic methods to
obtain information from five provinces, twelve districts and nine schools:
Reviewing DBE1 project documents
Observations in schools, district and provincial education offices
Collection of locally produced evidence of the use of the tools
Targeted questions and interviews.
The tools are in a process of development and some developments have been occurring very
recently. This means that districts‟ field experience with some tools is limited. In order to
give districts as much experience as possible with the tools, the evaluation was delayed until
late 2011.
Development of the Tools
DBE1 assists district governments to produce development plans, finance plans and budgets
to support these plans. Over the six years of the project from 2005, DBE1 and its partners
developed a set of budgeting and planning tools. The development of the tools has been
through a sequence of four, interconnected „phases‟. These phases and tools are:
Phase 1: Tools for District Planning and Budgeting (SIPPK and AKPK)
Phase 2: A Tool to Calculate School Operating Costs (BOSP)
Phase 3: Tools to Manage Human Resources (SIMPTK) and Assets (SIMA)
Phase 4: A Tool to Calculate Costs to Meet Minimum Service Standards and
Universal Access (PBPSAP).
Findings
Overall strong, positive evidence of acceptance and use of the tools
This evaluation has concluded that the tools are very acceptable to users and are being used in
productive ways to improve the management and governance of basic education in districts
as intended in the project design. The early evidence in support of these positive outcomes
comes from many different and mutually confirming sources.
There is a strong sense among users in districts of the great utility of the tools in relation to
present needs and circumstances. This reflects well on the good quality of the application
software prepared by DBE1.
Evaluation of DBE1 District Interventions
2
In addition to providing participating districts with the means to provide data to support
improvements in the quality of their planning and policy processes, there is evidence that the
use of the various application software tools is having an additional impact in revealing the
poor quality of data that is currently available, leading in turn, to the recognition of the
importance of good quality data and to the adoption of strategies by users to improve the
quality of data collections and data management. This move in the direction of achieving
better quality data may ultimately be one of the most important and lasting outcomes.
Rapid adoption and wide dissemination
Much of DBE1‟s work in implementing the tools in provinces, districts and schools has
occurred only recently – sometimes within the past few months – yet adoption of the
technical procedures is clearly very rapid and so too is the most important next step in the
analysis and use of new information in the formulation of policies and application strategies.
The rapid adoption reflects well on the bottom-up design and implementation strategy
followed by DBE1. Moreover, the wide dissemination of the tools to non-DBE districts is
further testimony to their value.
Integration
DBE1‟s implementation strategy for this set of management tools illustrates the important
outcomes made possible when the three levels of educational management and governance in
Indonesia at central, provincial and district levels are considered in holistic and integrated
ways rather than separately.
Do the tools work?
The question is answered from evidence available from several complementary perspectives.
First, yes, the tools do work in the technical sense of taking the available data and
processing it to produce useful outputs that are used for analysis, planning, policy
development, implementation action and monitoring.
Second, districts and provinces that use the tools find that they work in the ways listed
above and some have refined the tools to met specific local needs. Evidence of
helpfulness is in their use, and successful dissemination to other provinces and
districts often without direct DBE1 support.
As of November 2011, the DBE1 tools have been implemented in 117 districts in 13
provinces. Of these, 78 districts are supported directly with USAID funding; others
have used non-project resources to implement the tools in 69 non-DBE districts.
Governments at the national, provincial and district levels have used results of the
technical assistance from DBE1 to inform education policy, planning and budgeting.
Why do the tools work? – A ‘principles-based’ approach
The tools work in the ways intended because they meet local needs for data based
management, reflect government policy, have been developed collaboratively with districts
and have not been imposed top-down. The tools work particularly well where local
administrative leadership and competent technical support are present.
Evaluation of DBE1 District Interventions
3
From its project implementation experience and from its evaluation studies, DBE1 has
developed a sound understanding of the principles underpinning successful change. The
implementation of the tools evaluated in this study is a rare case illustrating change in
Indonesian education where a technical development has been undertaken within a clear
principles-based context that reflects a solid understanding of change processes and lessons
learned from past experience in Indonesian education.
Attention to analysis and use of data
DBE1 has given thorough attention to the analysis and use of data outputs produced by the
various tools. This attention is reflected in these ways:
In the manuals supporting the software tools, some of which provide explicit criteria
against which data can be judged
In the training materials such as DBE1 PowerPoint presentations located in district
education offices revealing that steps in analysis have been an important part of local
capacity development activities.
Well positioned
DBE1‟s technical and professional expertise in district management has been well positioned
to meet the evolving needs in districts and provinces for better quality data-based
management. Local demand for the tools has been reinforced by central government demands
to address such matters as external government audits and the introduction of the concept of
Minimum Service Standards.
Improved District Management Resources
Districts visited gladly produced evidence of their work with the tools in the form of
spreadsheets and finished plans and reports. These outputs from the use of the tools indicate a
better quality input to local planning and management of education than has been evident in
the past.
Improved data quality
A major outcome of DBE1‟s district level interventions with the tools and policy
development has been its impact on the quality of data. Beginning with existing data sets,
local users who then use this data can see the issues emerging from its quality and are then
better informed, and motivated, to ensure better data collections and use of that data for
management and governance purposes. This outcome could well be one of the most
important and enduring outcomes of all.
Immediate Challenges: Sustainability, Service Providers and Dissemination by Other
Donors
Matters that will arise in the near future for attention concern the sustainability of the
software in terms of its maintenance and updating when DBE1 has closed (that is, from the
beginning of 2012 onwards). If districts update their own software without some coordinated
oversight, this will lead inevitably to the growth over time of very different, and possibly
incompatible approaches. If updating is not done to meet changing needs and policies, it is
unfortunately the case that the tools will become obsolete very rapidly.
Evaluation of DBE1 District Interventions
4
This is an early time to draw firm conclusions about the service provider concept as it is now
implemented at district level. It is a relatively new concept. Based on the evaluation the
following major conclusions can be made about service providers:
The concept appears to have considerable merit as a sustainability strategy and
warrants further development and close monitoring.
There is a need for advocacy of the concept at district level to overcome the hesitancy
identified in this evaluation.
The concept will not succeed unless it is backed up by a commitment to high levels of
performance in customer service by certified individuals and institutions.
The matter of sustaining the quality of professional performance standards set by
DBE1 will have to be addressed in the future. Negative comments about services
provided by other donors and government agencies are an indicator that this matter is
important to ensure capacity development and that support succeeds as intended.
Service providers need to sell „packages‟ of integrated services, including monitoring
and troubleshooting, and not just one-off services such as document production.
Service providers need to be assisted in the future in selling their services and linking
these services to other projects that have a service provider objective, such as
USAID‟s Kinerja project, until they become self sustaining.
It was possible to determine that the tools are being taken up and disseminated by currently
on-going projects supported by AusAID, the World Bank and UNICEF. No evaluative data
were obtained to assess their experience so far, partly because of the constraints of resources
to do so and partly because their experience with the tools at the time of the evaluation was
limited. A challenge for the near future is to monitor the implementation experience of these
donors and to ensure the continuing harmonized developed and dissemination of the tools.
Evidence-based and evidence producing
The development and implementation of tools has been firmly based on the evidence from
studies in education in Indonesia and lessons learned from DBE1‟s own implementation
experience. In turn, DBE1‟s evaluation studies and its articulation of a set of principles for
providing technical assistance have produced evidence of „what works‟ and further lessons
learned that should be used in subsequent development activities.
Conclusion and Key Lessons Learned
The DBE1 experience from developing and implementing the tools is neatly summarized in
this quotation from the literature on policy reform in developing countries: Instead of
identifying ideal solutions up-front and top-down, policy implementers need to iteratively
develop second or third-best answers that collaboratively agencies and stakeholders can agree
upon1.
1 Brinkerhoff, D. and Crosby, B. (2002). Managing policy reform; concepts and tools for decision-making in
developing and transitioning countries. Bloomfield, CT: Kumarian Press. p.6.
Evaluation of DBE1 District Interventions
5
DBE1‟s solutions are not judged to be „second best‟. However, the key concepts in this
conclusion that describe DBE1‟s approach are „iteratively‟ and „collaboratively‟. These key
concepts are also illustrated by the evolution of DBE1‟s technical approaches over six years
with feedback from users and the progressive development of software in consultation with
users as well as with policy-makers at all levels of government.
The key lesson from the study is that the set of „principles‟ articulated by DBE1 for providing
technical assistance have been affirmed. In brief, the principles are:
Focus on developing products that local governments are required to implement.
Products and procedures must be based on government regulations.
Utilize technology commonly in use or available in the districts.
One-off training is ineffective for developing quality products – it must be ongoing.
All interventions include inputs to promote local participation, transparency and
accountability.
All tools and procedures incorporate facilitation for policy alternatives based on data
analysis.
All tools and procedures use currently available data.
The quality of products is determined by the quality of data; local stakeholders
determine the quality of data sufficient for their purposes.
Recommendations
The following recommendations are directed principally to USAID and to USAID-supported
development programs that may begin in the near future. The recommendations are also
relevant to other development partners and to the Government of Indonesia.
Recommendation 1: In any future development of tools, the confusion created in their
naming should be carefully reviewed with the purpose of simplifying the communication of
their names and purposes (Section 2.1).
Recommendation 2: In future activities that seek to develop the DBE1 tools and their further
use, major attention be given to the sustainability of the tools, their continuing technical and
policy support, and to the challenges of going to scale through implementation in larger
numbers of districts and provinces and dealing with larger-scale management issues (Section
2.3).
Recommendation 3: A review and synthesis study should be undertaken of DBE1’s research,
successful implementation strategies, and lessons learned, with the intention of producing
clear and simple guidelines for implementing change and development in Indonesian
education. The review should also consider published research, the documented experience
of other recent and successful development activities supported by USAID, other donors and
the Government of Indonesia (Section 3.4).
Evaluation of DBE1 District Interventions
6
Recommendation 4: When future development of the DBE1 tools is to be undertaken, it is
recommended that further and continuing attention be given to the intellectual processes of
the analysis and judgment of data in education, to the impacts of corruption, and to the use of
data (Section 4.2).
Recommendation 5: When subsequent development activities adopt the service provider
model, (a) the operation and acceptance of this approach to sustainable development be
carefully evaluated to ensure that quality, integrated support services (including networked
services) can be advocated and delivered in ways that meet clients’ needs in a timely,
professional and cost-effective manner; (b) service provider capacity building needs to begin
as early as possible as it requires significant levels of continuing training and mentoring; (c)
service providers be recruited from among the pool of knowledgeable and experienced
former DBE1 national consultants (Section 5.8).
Recommendation 6: To strengthen DBE1’s approaches to sustainability, a thorough risk
analysis should be undertaken of the tools and of the institutional and organization
environments in which the tools are used. The results of that analysis should be used to
develop evidence-based strategies to further strengthen the approaches to sustainability
currently being implemented. As the professional staffing stability of the district education
office is critical in sustainability, this key organization should be guided to develop strategies
for sustainability and the institutionalization of tools in further development activities
(Section 6.3).
Evaluation of DBE1 District Interventions
7
Ringkasan Eksekutif
Latar Belakang Tujuan Evaluasi
Evaluasi untuk komponen program Pendidikan Dasar yang Terdesentralisasi yaitu
Desentralisasi Manajemen dan Tata Kelola Pendidikan yang Lebih Efektif, yang dikenal
sebagai DBE1, adalah untuk memberikan kontribusi untuk memaksimalkan penerimaan,
penggunaan secara luas dan pelaksanaan hasil DBE1. Fokus kajian adalah pada penerimaan,
penggunaan, dan dampak dari alat-alat (tools) untuk mendukung penggunaan data dalam
manajemen dan tata kelola pendidikan, khususnya di tingkat kabupaten/kota.
Metodologi
Dilakukan pada Agustus-November 2011, tim evaluasi menggunakan empat metode dasar
untuk mendapatkan informasi dari lima provinsi, dua belas kabupaten/kota dan sembilan
sekolah:
Mengkaji dokumen proyek DBE1
Pengamatan di sekolah, kantor Dinas Pendidikan kabupaten/kota dan provinsi
Mengumpulkan bukti penggunaan alat-alat yang diproduksi di daerah
Pertanyaan terstruktur dan wawancara.
Alat-alat masih dalam proses pengembangan dan beberapa pengembangan baru dilakukan
akhir-akhir ini. Ini berarti bahwa pengalaman kabupaten/kota dalam menggunakan beberapa
alat masih sangat terbatas. Untuk memberikan kesempatan kepada kabupaten/kota untuk
memperoleh pengalaman sebanyak mungkin dengan alat-alat, evaluasi ini ditunda sampai
akhir 2011.
Pengembangan Alat-alat
DBE1 membantu pemerintah kabupaten/kota untuk menghasilkan rencana pembangunan,
rencana keuangan dan kebutuhan anggaran untuk mendukung rencana ini. Selama enam
tahun proyek mulai dari tahun 2005, DBE1 dan mitranya mengembangkan seperangkat alat
perencanaan dan pennyusunan anggaran. Pengembangan alat-alat telah melalui empat
tahapan, yang saling berhubungan. Tahapan dan alat yang dikembangkan adalah:
Tahap 1: Alat untuk Perencanaan Kabupaten dan Penganggaran (SIPPK dan AKPK)
Tahap 2: Alat untuk Menghitung Biaya Operasional Sekolah (BOSP)
Tahap 3: Alat untuk Mengelola Sumber Daya Manusia (SIMPTK) dan Aset (SIMA)
Tahap 4: Alat untuk Menghitung Biaya untuk Memenuhi Standar Pelayanan
Minimum dan Akses Pendidikan (PBPSAP).
Temuan Secara keseluruhan bukti yang kuat dan positif dalam penerimaan dan penggunaan
alat
Evaluasi ini menyimpulkan bahwa alat-alat sangat dapat diterima oleh pengguna dan sedang
digunakan secara produktif untuk meningkatkan manajemen dan tata kelola pendidikan dasar
Evaluation of DBE1 District Interventions
8
di kabupaten/kota sebagaimana dimaksud dalam desain proyek. Bukti awal untuk mendukung
hasil-hasil positif ini berasal dari beberapa sumber yang saling mengkonfirmasikan.
Ada perasaan kuat di antara pengguna di kabupaten/kota bahwa penggunaan alat-alat sangat
erat kaitannya dengan keadaan dan kebutuhan saat ini. Hal ini tercermin pada kualitas yang
baik dari aplikasi perangkat lunak yang disiapkan oleh DBE1.
Selain menyediakan sarana bagi kabupaten/kota mitra untuk menyediakan data dalam
mendukung perbaikan kualitas dalam proses perencanaan dan kebijakan, ada bukti bahwa
penggunaan berbagai aplikasi perangkat lunak memiliki dampak tambahan dalam
mengungkap buruknya kualitas data yang saat ini tersedia, yang menuju pada, kesadaran akan
pentingnya kualitas data yang baik dan penerapan strategi oleh pengguna untuk
meningkatkan kualitas pengumpulan data dan manajemen data. Pergerakan ke arah
pencapaian kualitas data yang lebih baik pada akhirnya dapat menjadi salah satu hasil yang
paling penting dan berkelanjutan.
Adopsi yang cepat dan penyebaran yang luas
Banyak kegiatan DBE1 dalam mengimplementasikan alat-alat di provinsi, kabupaten/kota
dan sekolah telah terjadi baru-baru ini - kadang-kadang dalam beberapa bulan terakhir -
namun adopsi prosedur teknis ini jelas sangat cepat dan demikian juga langkah berikutnya
yang paling penting dalam analisis dan penggunaan informasi baru untuk perumusan
kebijakan dan strategi aplikasi. Adopsi yang cepat mencerminkan dengan baik adanya desain
bottom-up dan strategi pelaksanaan yang dilakukan oleh DBE1. Selain itu, penyebaran luas
dari alat-alat ke kabupaten/kota non-DBE merupakan bukti lebih lanjut dari nilai alat-alat
tersebut.
Integrasi
Strategi implementasi DBE1 untuk perangkat alat manajemen menggambarkan hasil penting
yang dapat diperoleh apabila tiga tingkat manajemen dan tata kelola pendidikan di Indonesia
di tingkat pusat, provinsi dan kabupaten dipertimbangkan secara menyeluruh dan terpadu
daripada masing-masing secara terpisah.
Apakah alat-alat bekerja dengan baik?
Pertanyaan ini dijawab dengan bukti-bukti yang tersedia dari beberapa perspektif yang saling
melengkapi.
Pertama, ya, alat dapat digunakan dalam arti teknis yaitu mengambil data yang
tersedia dan mengolahnya untuk menghasilkan output yang berguna untuk analisis,
perencanaan, pengembangan kebijakan, pelaksanaan aksi dan pemantauan.
Kedua, kabupaten/kota dan provinsi yang menggunakan alat-alat tersebut bekerja
dengan cara yang disebut di atas dan ada beberapa yang telah menyempurnakan alat
untuk memenuhi kebutuhan daerah yang spesifik. Bukti dari kegunaan alat adalah
penggunaannya dan diseminasi ke provinsi dan kabupaten/kota lain tanpa dukungan
langsung dari DBE1.
Sampai November 2011, alat-alat DBE1 telah digunakan oleh 117 kabupaten/kota di
13 provinsi. Dari jumlah tersebut, 78 kabupaten/kota didukung langsung dengan
pendanaan USAID; 69 kabupaten/kota non-DBE yang lain menggunakan sumber
daya non-proyek untuk menggunakan alat-alat tersebut.
Evaluation of DBE1 District Interventions
9
Pemerintah di tingkat nasional, provinsi dan kabupaten/kota telah menggunakan hasil
bantuan teknis dari DBE1 untuk menginformasikan kebijakan pendidikan,
perencanaan dan penganggaran.
Mengapa alat bermanfaat? – Pendekatan ‘berbasis prinsip’
Alat-alat digunakan dengan cara yang dimaksudkan karena mereka memenuhi kebutuhan
daerah untuk manajemen berbasis data, mencerminkan kebijakan pemerintah, telah
dikembangkan bersama-sama dengan kabupaten/kota dan tidak dipaksakan dari atas. Alat-
alat dimanfaatkan dengan sangat baik di mana terdapat kepemimpinan administratif daerah
yang baik dan dukungan teknis yang kompeten.
Dari pengalaman pelaksanaan proyek dan dari studi-studi evaluasi, DBE1 telah mendapatkan
pemahaman yang baik tentang prinsip-prinsip yang mendukung keberhasilan suatu
perubahan. Penggunaan alat-alat yang dievaluasi dalam kajian ini merupakan kasus yang
jarang terjadi yang menggambarkan perubahan dalam pendidikan di Indonesia di mana
pengembangan teknis telah dilakukan dalam konteks berbasis prinsip yang jelas dan
mencerminkan pemahaman yang mantap tentang proses perubahan dan pelajaran yang
dipelajari dari pengalaman masa lalu dalam pendidikan di Indonesia.
Perhatian terhadap analisis dan penggunaan data
DBE1 memberikan perhatian penuh pada analisis dan penggunaan output data yang
dihasilkan bermacam-macam alat. Perhatian itu dapat dilihat pada:
Dalam pedoman yang mendukung alat-alat perangkat lunak (software tools), beberapa
diantaranya memberikan kriteria eksplisit untuk penilaian data
Bahan pelatihan seperti presentasi Powerpoint DBE1 yang terdapat di kantor dinas
pendidikan kabupaten/kota dan menunjukkan langkah-langkah analisis merupakan
bagian penting dari kegiatan pengembangan kapasitas daerah.
Penempatan yang baik
Keahlian teknis dan profesional DBE1 dalam manajemen kabupaten/kota ditempatkan
dengan baik untuk memenuhi kebutuhan yang berkembang di kabupaten/kota dan provinsi
untuk kualitas manajemen berbasis data yang lebih baik. Kebutuhan akan alat-alat di daerah
telah diperkuat oleh permintaan pemerintah pusat untuk hal-hal seperti audit eksternal
pemerintah dan pengenalan konsep Standar Pelayanan Minimal.
Peningkatan Sumber Daya Manajemen Kabupaten/Kota
Kabupaten/kota yang dikunjungi dengan senang hati memperlihatkan bukti yang dihasilkan
dari pekerjaan mereka dengan alat-alat dalam bentuk spreadsheet, serta rencana dan laporan.
Output dari penggunaan alat-alat menunjukkan kualitas yang lebih baik untuk masukan
perencanaan daerah dan manajemen pendidikan dibandingkan dengan yang terdapat di masa
lalu.
Evaluation of DBE1 District Interventions
10
Peningkatan kualitas data
Sebuah hasil utama dari intervensi DBE1 di tingkat kabupaten/kota dengan alat-alat dan
pengembangan kebijakan adalah dampaknya terhadap kualitas data. Dimulai dengan
perangkat data yang ada, pengguna di daerah yang kemudian menggunakan data ini dapat
melihat isu-isu yang muncul dari kualitas dan kemudian lebih terinformasi, dan termotivasi,
untuk memastikan pengumpulan data yang lebih baik dan penggunaan data untuk keperluan
manajemen dan tata kelola. Hasil ini bisa jadi menjadi salah satu hasil paling penting dan
berkelanjutan dari semua.
Tantangan Sekarang: Keberlanjutan, Penyedia Layanan dan Diseminasi oleh Donor
Lainnya
Hal-hal yang akan timbul dalam waktu dekat berkaitan dengan keberlanjutan dari perangkat
lunak adalah dalam hal pemeliharaan dan pembaharuan ketika DBE1 telah selesai (yaitu, dari
awal tahun 2012 dan seterusnya). Apabila kabupaten/kota melakukan pemutakhiran (update)
perangkat lunak sendiri tanpa pengawasan terkoordinasi, ini akan mengarah pada
berkembangnya pendekatan yang sangat berbeda setelah beberapa waktu, dan mungkin tidak
kompatibel. Jika pemutakhiran tidak dilakukan untuk memenuhi perubahan kebutuhan dan
kebijakan, maka alat akan menjadi usang sangat cepat.
Masih terlalu awal untuk menarik kesimpulan tegas tentang konsep penyedia layanan seperti
yang sekarang diterapkan di tingkat kabupaten/kota. Ini adalah konsep yang relatif baru.
Berdasarkan evaluasi, kesimpulan utama berikut dapat diambil tentang penyedia layanan:
Konsep ini tampaknya memiliki manfaat yang cukup besar sebagai strategi
keberlanjutan dan menjamin pengembangan lebih lanjut dan pemantauan ketat.
Ada kebutuhan untuk memperkenalkan konsep di tingkat kabupaten/kota untuk
mengatasi keraguan yang teridentifikasi dalam evaluasi ini.
Konsep ini tidak akan berhasil kecuali didukung oleh komitmen untuk tingkat kinerja
yang tinggi dalam pelayanan pelanggan oleh perorangan dan lembaga tersertifikasi.
Mempertahankan kualitas standar kinerja profesional yang ditetapkan oleh DBE1
harus ditangani di masa depan. Komentar negatif tentang layanan yang disediakan
oleh donor lain dan instansi pemerintah merupakan indikator bahwa masalah ini
penting untuk memastikan pengembangan kapasitas dan bahwa dukungan dilanjutkan
seperti direncanakan.
Penyedia layanan perlu menjual 'paket' layanan terpadu, termasuk pemantauan dan
pemecahan masalah, dan bukan hanya layanan sekali selesai (one-off services) seperti
produksi dokumen.
Penyedia layanan harus didukung di masa depan dalam menjual jasa mereka dan
menghubungkan layanan mereka pada proyek-proyek lain yang memiliki tujuan
penyedia layanan, seperti proyek Kinerja USAID, sampai mereka mampu berdiri diri.
Dapat diketahui bahwa alat-alat ini diambil dan sedang disebarluaskankan saat ini oleh
proyek-proyek yang sedang berlangsung dan didukung oleh AusAID, Bank Dunia dan
UNICEF. Tidak ada data evaluasi yang diperoleh untuk menilai pengalaman mereka sejauh
ini, sebagian karena kendala sumber daya untuk melakukannya dan sebagian karena
pengalaman mereka dengan alat-alat pada saat evaluasi masih terbatas. Sebuah tantangan
untuk waktu dekat adalah untuk memantau pengalaman pelaksanaan para donor dan untuk
memastikan terusnya harmonisasi pengembangan dan diseminasi dari alat-alat.
Evaluation of DBE1 District Interventions
11
Berbasis bukti dan menghasilkan bukti
Pengembangan dan implementasi alat-alat telah dilakukan berdasarkan bukti dari kajian
dalam pendidikan di Indonesia dan pelajaran dari pengalaman implementasi DBE1 sendiri.
Pada gilirannya, kajian evaluasi DBE1 dan artikulasinya dari satu perangkat prinsip untuk
memberikan bantuan teknis telah menghasilkan bukti 'apa yang dapat dilakukan' (what
works) dan pelajaran yang dipelajari (lessons learned) yang harus digunakan dalam kegiatan
pembangunan selanjutnya.
Kesimpulan dan Pelajaran Penting yang Dipelajari
Pengalaman DBE1 dalam mengembangkan dan menerapkan alat-alat dirangkum dengan baik
dalam kutipan yang diambil dari literatur tentang reformasi kebijakan di negara berkembang:
Daripada mengidentifikasi solusi yang ideal pada awal dan dari atas-ke bawah,
pelaksana kebijakan perlu secara iteratif (iteratively, berulang-ulang dan makin lama
makin baik) mengembangkan jawaban kedua atau ketiga terbaik yang dapat
disepakati secara bersama antara lembaga dan pemangku kepentingan2.
Solusi DBE1 tidak dinilai sebagai 'terbaik kedua'. Namun, konsep-konsep penting dalam
kesimpulan ini yang menggambarkan pendekatan DBE1 adalah 'iteratif' dan 'bersama-sama'.
Konsep-konsep penting ini juga digambarkan oleh evolusi pendekatan teknis DBE1 selama
enam tahun dengan umpan balik dari pengguna dan pengembangan perangkat lunak progresif
dalam konsultasi dengan pengguna serta dengan pembuat kebijakan di semua tingkat
pemerintahan.
Pelajaran utama dari kajian ini adalah bahwa himpunan 'prinsip' yang diartikulasikan oleh
DBE1 untuk menyediakan bantuan teknis telah ditegaskan. Secara singkat, prinsip-prinsipnya
adalah:
Fokus pada pengembangan produk yang harus dilaksanakan oleh pemerintah daerah.
Produk dan prosedur harus berdasarkan peraturan pemerintah.
Gunakan teknologi yang biasa digunakan atau tersedia di kabupaten/kota.
Pelatihan sekali selesai (one-off) tidak efektif untuk mengembangkan produk
berkualitas – harus terus menerus.
Semua intervensi mencakup masukan untuk mempromosikan partisipasi daerah,
transparansi dan akuntabilitas.
Semua alat dan prosedur menggabungkan fasilitasi untuk kebijakan alternatif
berdasarkan analisis data.
Semua alat dan prosedur menggunakan data mutakhir yang tersedia.
Kualitas produk ditentukan oleh kualitas data; pemangku kepentingan daerah
menentukan kualitas data yang sesuai dengan keperluan mereka.
2 Brinkerhoff, D. and Crosby, B. (2002). Managing policy reform; concepts and tools for decision-making in
developing and transitioning countries. Bloomfield, CT: Kumarian Press. p.6.
Evaluation of DBE1 District Interventions
12
Rekomendasi Rekomendasi berikut terutama ditujukan kepada USAID dan program pembangunan yang
didukung USAID yang mungkin dimulai dalam waktu dekat. Rekomendasi ini juga relevan
untuk mitra pembangunan yang lain dan Pemerintah Indonesia.
Rekomendasi 1: Dalam setiap pengembangan alat-alat di masa mendatang, nama alat
hendaknya dikaji dengan baik dengan tujuan untuk menyederhanakan komunikasi tentang
nama dan tujuan alat-alat itu sehingga tidak menimbulkan kebingungan (Bagian 2.1).
Rekomendasi 2: Dalam kegiatan masa depan yang akan mengembangkan alat-alat DBE1
dan penggunaannya lebih lanjut, perhatian utama harus diberikan pada keberlanjutan alat-
alat, dukungan teknis dan politis terus menerus,dan tantangan untuk memperluas
penggunaannya di lebih banyak kabupaten/kota dan provinsi dan menangani masalah
manajemen dalam lingkup yang lebih besar (Bagian 2.3).
Rekomendasi 3: Sebuah tinjauan dan studi sintesis perlu dilakaukan pada kajian-kajian
DBE1, strategi pelaksanaan yang berhasil, dan pelajaran yang dipelajari, dengan tujuan
untuk menyusun pedoman yang jelas dan singkat untuk melaksanakan perubahan dan
pembangunan pada pendidikan di Indonesia. Tinjauannya juga harus mempertimbangkan
hasil penelitian yang dipublikasi, pengalaman terdokumentasi dari kegiatan pembangunan
yang berhasil akhir-akhir ini yang didukung USAID, donor lain dan Pemerintah Indonesia
(Bagian 3.4).
Rekomendasi 4: Apabila pengembangan alat-alat DBE1 dilakukan lebih lanjut, disarankan
agar melanjutkan dan memberikan perhatian secara terus menerus pada proses intelektual
dari analisis dan penilaian data pendidikan, pada dampak dari korupsi dan penggunaan data
(Bagian 4.2).
Rekomendasi 5: Apabila kegiatan pembangunan selanjutnya mengadopsi model penyedia
layanan, (a) pelaksanaan dan penerimaan dari pendekatan ini untuk pembangunan
berkelanjutan agar dikaji secara hati-hati untuk menjamin kualitas, dukungan terintegrasi
(termasuk layanan berjejaring) dapat disampaikan dengan cara-cara yang sesuai dengan
kebutuhan pelanggan yaitu dengan cara yang tepat, professional, dan biaya efektif ; (b)
pengembangan kapasitas pemberi layanan harus dimulai sesegera mungkin karena
membutuhkan tingkat pelatihan yang signifikan dan pemantauan secara terus menerus; (c)
pemberi layanan direkrut dari kelompok mantan konsultan nasional DBE1 yang memahami
dan berpengalaman (Bagian 5.8).
Rekomendasi 6: Untuk memperkuat pendekatan DBE1 pada keberlanjutan, sebuah analisis
resiko yang rinci harus dilakukan tentang alat-alat dan lingkungan institusi dan organisasi di
mana alat-alat tersebut digunakan. Hasil analisis tersebut hendaknya digunakan untuk
mengembangkan strategi berbasis bukti (evidence-based) untuk lebih memperkuat
pendekatan pada keberlanjutan yang sekarang dilaksanakan. Karena stabilitas staf
professional di dinas pendidikan kabupaten/kota penting untuk keberlanjutan, organisasi
utama ini harus dibimbing untuk mengembangkan strategi untuk keberlanjutan dan
institusionalisasi alat-alat dalam kegiatan pembangunan berikutnya (Bagian 6.3).
Evaluation of DBE1 District Interventions
13
1. Introduction to the Evaluation
1.1. Background
The Decentralized Basic Education program component More Effective Decentralized
Education Management and Governance, known as DBE1, is a project implemented by
Research Triangle Institute under an agreement between the United States Agency for
International Development (USAID) and the Indonesian Ministry for People‟s Welfare. The
project commenced April 2005 and will close in December 2011.
The objective of DBE1 is to develop more effective decentralized education management and
governance. At District level, DBE1 provides technical assistance to 50 target Districts, plus
many dissemination districts, to strengthen management, planning, budgeting and
governance. In support of this assistance to districts, DBE1 has developed and disseminated a
comprehensive set of application software financial and management „tools‟ to support the
management and governance of basic education at school level and district levels. The focus
of this study is at district level where the tools have been developed and used. The application
software tools are as follows:
Financial Tools:
District Education Finance Analysis (AKPK3)
School Unit Cost Analysis (BOSP)
Calculation of Costs to Achieve Minimum Service Standards and Universal Access
(PBPSAP)
Data and Planning Tools:
District Education Planning Information System (SIPPK)
Education Asset Management Information System (SIMA)
Personnel Management Information System (SIM-PTK)
Integrated District Management Information System (SIMPK), which incorporates the
former SIPPK and SIM-PTK
District Strategic Planning Manual (Renstra).
Personnel from universities and other institutions are being trained as „service providers‟ in
the use of the tools. The overall aim is to develop a means by which districts can be assisted
to improve their management, planning, budgeting and policy development on the basis of
improving standards of data quality and on the careful analysis of that data.
The tools are used to turn raw data into a useful output. Districts that have used the tools are
finding the outputs useful because the outputs directly inform strategic planning, budgeting,
policy formation and decision-making. Schools using the asset management and preventive
maintenace tools (SIMA) similarly report the usefulness of data for planning and
management. All the districts studied in this evaluation (Subang in West Java; Purworejo in
Central Java; Sampang, Kota Mojokerto and Tuban in East Java; Kota Tebing Tinggi and
3 Explanation of these abbreviations of the Indonesian terms is provided in Chapter 2.
Evaluation of DBE1 District Interventions
14
Kota Tanjung Balai in North Sumatera; and Aceh Besar and Kota Banda Aceh in Nanggroe
Aceh Darussalam) report that the tools are useful.
Through this process of using data, there are other useful outcomes too. One of these is that
the collection and use of data clarifies quality weaknesses in the data and this, in turn,
provides a stimulus to improving data quality and better policy outcomes and actions. DBE1
expressed this to the evaluation team as a „principle of technical assistance‟. The principle is
that increased use of data will generate a greater demand for data that, in turn, generates a
supply of better quality data. This outcome could well be one of the most important and
enduring outcomes of all. DBE1‟s experience in this area should be shared widely and should
be looked upon as a pilot for future interventions.
Other policy outcomes are possible after data analysis and a report are produced. In fairness,
DBE1 cannot guarantee the outcome of this last activity because of its local political nature.
These policies are intended to make a difference to the efficiency and effectiveness of the
governance and management of the district education system, and ultimately of schools and
classrooms, in providing quality education to children.
1.2. Objectives of the Evaluation
The broad objective of this evaluation study is to contribute to maximizing the acceptance,
widespread use and the implementation of the DBE1 results, mainly at the district level. The
focus of the study is on the acceptance, use and impact of the tools and not on the
characteristics of the tools themselves (although feedback from users on technical matters is
included where this has relevance to the use of a tool).
To address the objective of the assignment, an evaluation of the tools‟ acceptance by users
and the use of the tools have been undertaken. The key issues and questions addressed in the
evaluation are as follows:
To clarify and describe the tools
Do the tools work? Do they achieve the stated objectives of translating policy into
implementation action in districts? If so, why and how? If not, why not?
Is there an appreciation of the need to work with an external service provider? Do the
districts, working with service providers, have the capacity to use the tools
effectively? Are there potential service providers available with the capacity to meet
that need/demand? What is the initial experience of potential service providers with
the tools?
Are the tools and the implementation strategies sustainable?
What is the level of interest and need/demand for the tools and support services?
What is the experience with the dissemination, institutionalization and
adaptation/evolution of the tools? This includes the experience of other donors (World
Bank, UNICEF, AusAID).
Evaluation of DBE1 District Interventions
15
What is the early evidence of outcomes? (The study may consider early evidence of
outcomes – at District level and beyond at school level and national level – should
these exist. Outcomes for non-organizational phenomena should also be considered
such as on the quality of data and on people/participants in implementation. Is there
early evidence of outcomes and, if so, what are factors that appear to support these
outcomes?)
Important questions for the future are the “why” and “how” questions embedded in the
following: “Do the tools work? Do they achieve the stated objectives of translating policy
into implementation action in Districts? If so, why and how? If not, why not?”
The answers to these questions are important as they contribute to a deeper understanding of
the processes of change in the management of Indonesian education. The answers therefore
provide critically important guidance to future change and development strategies that may
be implemented by the Government of Indonesia and its development partners. Chapter 0
addresses many of these important questions.
This Report focuses on the use of the tools. The Report includes lessons learned and
recommendations of relevance to current and future development assistance programs and
partners, and for potential local service providers where these may exist.
1.3. Methodology
The evaluation was conducted discontinuously in August and again in October/November
2011 by a small independent team of one national and one international consultant. Fieldwork
was led by the independent consultants but guided by DBE1 project staff. Two DBE1 staff
directly assisted in the August fieldwork visits.
The evaluation team used four basic methods to obtain information for its findings, including:
Reviewing DBE1 project documents and studies in educational development
Observation of DBE-assisted districts and schools, training sessions, government
officials, principals, teachers and other educators at work with the tools
Collection of locally produced evidence and examples of the use of the tools
Targeted questions and interviews with individuals knowledgeable about DBE1 and
the various tools. These individuals include DBE1 staff and consultants in Jakarta and
in districts as well as project beneficiaries in schools, districts and provinces.
Checklists for discussions about the tools were developed and used in most locations where
attendance by appropriate district staff made this possible. A total of 12 districts and nine
schools was visited in five provinces – East Java, Central Java, West Java, North Sumatra and
Nanggroe Aceh Darussalam (NAD). The schools visited were in three school districts,
Subang and Cimahi in West Java and Purworejo in Central Java. We visited schools known
to be using a specific tool and this was limited to the asset management tool, SIMA.
Evaluation of DBE1 District Interventions
16
1.4. Constraints in Undertaking the Evaluation
The evaluation was constrained by an unusual set of circumstances. The tools are in a process
of constant development and some developments have been occurring very recently indeed.
This means that districts‟ field experience with the latest tools is limited. In order to give
districts as much experience as possible with the tools, the evaluation was deliberately
delayed until late 2011. By November, DBE1 was well advanced in closing the project,
scheduled for December 2011, and several key staff were in the process of departing from
DBE1, having completed their contracts. November also proved to a challenging time to
undertake the evaluation, as it is a busy month for district education offices when they are
working hard to finish off all activities for the 2011 year.
Given the scale of the DBE1 project, there were insufficient resources for the evaluation team
to undertake visits to a representative sample of districts that may have used the tools. The
evaluation is therefore restricted to a relatively few sites but this restriction did not prevent
key evaluation questions to be answered with a good degree of confidence.
This proved to be a complex evaluation because of the number and nature of tools available
to districts. It was unclear to the evaluation team as to exactly what tools had been provided
in some districts and districts themselves were confused on occasions among the different
DBE1 tools.
Finally, most districts were expecting the evaluation team, and some were very well prepared
for our visit with PowerPoint presentations and examples of relevant documents. Others were
less forthcoming and key people did not attend planned visits on some occasions. This has led
to some gaps in our data.
We have taken all these constraints into account in evaluating the data and forming our
conclusions.
Evaluation of DBE1 District Interventions
17
2. Tools for Educational Management & Governance
2.1. Introduction
Key matters in the complex history of the development of the various tools are briefly
described here to provide a context for the evaluation.
We noted during the evaluation that there is some confusion about the history, purposes and
current status of the tools. This matter is not helped by the numerous acronyms and
contractions that have become a feature of development and educational discussion in
Indonesia more generally and of these tools specifically. The challenge to outsiders is made
doubly difficult when acronyms look similar, use the same letters, but refer to very different
things. These examples illustrate the issue:
SIMPTK and SIMPK: Sistem Informasi Manajemen Pendidik dan Tenaga
Kependidikan (Personnel Management Information System); Sistem Informasi
Manajmen Pendidikan Kabupaten/Kota) (District Education Management
Information System)
BOS and BOSP: Bantuan Operasional Sekolah (School Operational Grant); Biaya
Operasional Satuan Pendidikan (School Operational Cost)
SPM (Standar Pelayanan Minimal or Minimum Service Standard) and SBM (school
based management). This particular example illustrates another difficulty when
abbreviations are drawn inconsistently from both Indonesian and English.
At best, these complications do not help implementation, change and dissemination. At worst,
they are an irritation and a source of confusion, particularly for those stakeholders new to
DBE1‟s work and otherwise fundamentally well disposed to seek to understand. The
confusion created by these complications did lead to data gathering problems in the field and
consequential gaps in data as a direct result. And we acknowledge here that there may still be
errors in our reporting that arise from our own continuing misunderstandings.
However, the present evaluation is neither a linguistic nor an historical analysis. What
matters more, in our view, is the credit that DBE1 is to be given for the fact that the tools
have been developed, are being used effectively, and that they reflect effective principles of
change and development in Indonesian education.
Recommendation 1: In any future development of tools, the confusion created in their
naming should be carefully reviewed with the purpose of simplifying the communication of
their names and purposes.
2.2. Clarification and Description of the Tools
DBE1 assists district governments to produce education sector development plans and
finance plans and budgets to support these plans. Initially, DBE1 tried to use currently
available tools for planning and budgeting, but these were either not available or considered
unsatisfactory. Therefore, over the six years of the project from 2005, DBE1 and its partners
developed a new set of budgeting and planning tools. These tools continue to evolve and to
be improved. Work on this continues in late 2011 to match local needs and changing policy
circumstances.
Evaluation of DBE1 District Interventions
18
The development of the tools has been through a sequence of four, interconnected „phases‟.
These phases are described below4.
Phase 1: Tools for District Planning and Budgeting – SIPPK and AKPK
The first set of tools developed in 2006, comprising application software and manuals, was to
base district development plans and budgets on the available data and on government
regulations to produce five-year plans called Renstra (Rencana Strategis or Strategic Plan).
Supported by manuals to guide trainers and to provide technical assistance to districts, this set
of tools consisted of:
Software for a district database called District Planning and Information Support
System (Sistem Informasi Perencanaan Pendidikan Kabupaten/Kota or SIPPK).
SIPPK contained basic profile information for each school and madrasah in the
district. SIPPK used Microsoft Access software. SIPPK is no longer to be used and
has been replaced by a new District Information Management System (Sistem
Informasi Manajmen Pendidikan Kabupaten/Kota or SIMPK.
Software for an analysis of annual budgets using district, province and national data
sources called District Education Financial Analysis (Analisa Keuangan Pendidikan
Kabupaten/Kota or AKPK).
Phase 2: A Tool to Calculate School Operating Costs – BOSP
In 2007, at the request of the Bupati of Sidoarjo District in East Java, DBE1 developed a
Microsoft Excel based tool called Calculating School Operations Costs (Penghitungan Biaya
Operasional Satuan Pendidikan or BOSP).
BOSP uses standards for school operational costs that were established by the Board of
National Education Standards. It allows district governments to adjust the unit costs
according to local needs and resources.
Since 2005, the national government has been transferring school operation funds to schools
through a program known as Bantuan Operasional Sekolah (BOS). The BOSP tool enables
local officials to analyze the difference between their operations calculations and the amount
schools receive from the BOS program. In almost every case the analysis showed a large gap
between national government funding and operational needs of schools. As a result of BOSP
applications, districts and provinces have budgeted local funds to help close the gap between
school income and expenditures for operations.
In late 2010 DBE1 presented results of school operations calculations to a meeting of high-
level government officials and other stakeholders. The Deputy Minister of National
Education and Secretary to the Vice President asked DBE1 to enlarge the scope of BOSP to
enable local governments to also calculate costs for achieving the recently introduced
Minimum Service Standards and to reach school access targets. This new tool is called
Calculation of Costs to Achieve Minimum Service Standards and Universal Access –
Penghitungan Biaya Pencapaian Standar dan Akses Pendidikan or PBPSAP.
4 Material in this section draws on a paper titled DBE1 Tools and Procedures to Support Local Government
Education Management and Governance, November 2011, prepared by DBE1.
Evaluation of DBE1 District Interventions
19
PBPSAP, developed in the first half of 2011, is described below. Part of the earlier BOSP
tool has now been integrated into the PBPSAP with modifications.
Phase 3: Tools to Manage Human Resources (SIMPTK) and Educational Assets (SIMA)
The Personnel Management System or Sistem Informasi Manajemen Pendidik dan Tenaga
Kependidikan (SIMPTK) took three years from 2007 to develop, test and implement.
SIMPTK is a Microsoft Excel based tool.
District governments have enthusiastically endorsed the tool and several have implemented
personnel hiring policies based on the data analysis from this tool. In 2011 this personnel
management system – SIMPTK – was incorporated in the final DBE management and
governance tool described below in Phase 4, PBPSAP.
The Assets Information Management System or Sistem Informasi Manajemen Aset (SIMA)
was developed and tested in 2010 and implemented by six district governments and one
provincial government by 2011. SIMA enables district governments to track and inventory
assets through this Excel-based software application.
SIMA provides assets data required for audits by the national auditing agencies. Many
districts have been receiving audit disclaimers because they have not been able to accurately
inventory assets most of which are in schools. The system also enables districts and schools
to better plan and schedule maintenance and repair school buildings and equipment. A
preventive maintenance training manual accompanies the package.
Phase 4: A Tool to Calculate Costs to Meet Minimum Service Standards and Universal
Access (PBPSAP)
From late 2010, DBE1 developed a new tool called Calculation of Costs to Achieve
Minimum Service Standards and Universal Access (Penghitungan Biaya Pencapaian Standar
dan Akses Pendidikan or PBPSAP. By December 2011 the new tool is expected to be
implemented in over 50 districts.
As noted above, the (then) Deputy Minister of National Education and Secretary to the Vice
President asked DBE1 to expand its school operations costing tool (BOSP) to enable local
governments to calculate costs to meet the new Minimum Service Standards and to achieve
national basic education access targets. In 2010 the Ministry of National Education had
approved the new Minimum Service Standards and had calculated the costs to meet the
standards aggregated at the national level. However, this calculation did not take into account
the costs if access targets were met. Further, the calculation did not disaggregate costs at the
district level. DBE1 developed the new tool to address these needs.
PBPSAP uses data readily available through the Ministry‟s Educational Management
Information System (EMIS) that requires district education offices to gather, compile and
analyze data from all schools in the district annually. DBE1 developed an Excel-based
software application called District Information Management System (Sistem Informasi
Manajemen Pendidikan Kabupaten/Kota) (SIMPK) which uses data analysis for planning,
costing and policy development purposes. SIMPK replaces the earlier District Planning and
Information Support System (Sistem Informasi Perencanaan Pendidikan Kabupaten/Kota or
SIPPK), originally developed to support medium term planning (see Phase 1 above) and the
Personnel Management System (SIMPTK) (see Phase 3 above).
Evaluation of DBE1 District Interventions
20
SIMPK provides data analysis for costing to meet Minimum Service Standards and access
targets. The original tool for calculating school operations costs (BOSP) (see Phase 2) was
modified and incorporated into the PBPSAP tool.
PBPSAP enables districts to calculate costs to meet standards and achieve access targets over
a period of five years. More importantly, the package also provides analysis and guidance for
local governments to determine policy alternatives to meet standards and targets most
efficiently.
2.3. Application Software Quality and Future Development
No detailed judgments are made here about the quality of the application software that has
been produced. This was beyond the scope of the present evaluation. There is, however, a
strong sense among users in districts of its great utility in relation to present needs and
circumstances. This reflects well on the good quality of the application software and we have
received only very limited evidence from schools and districts of early implementation
difficulties.
In addition to providing participating districts with the means to provide data to support
improvements in the quality of their planning and policy processes, there is evidence that the
use of the various application software tools is having an additional impact in revealing the
poor quality of data that is currently available, leading, in turn, to the adoption of strategies to
improve the quality of data collections and data management.
Matters that will arise in the future concern the sustainability of the software in terms of its
maintenance and updating when DBE1 has closed (that is, from the beginning of 2012
onwards). Not all districts currently have technical staff that will be able to undertake these
tasks, and where they do exist, these local human resource assets are fragile and vulnerable to
staff movements at any time. If districts do update there own software without some
coordinated oversight, this will lead inevitably to the growth over time of very different, and
possibly incompatible approaches.
Another concern is the matter of going to scale. There is little question that using locally
available and well-understood software such as Microsoft Excel was a good strategy to make
a start with local, data based management. But industry experience is that this will lead to
difficulties in the future when larger-scale demands are made from provincial and national
levels and software integration is required. In addition, in the absence of a coordinated
application software and technical maintenance plan reflecting new laws and regulations,
there will inevitably be a decline in the utility of the software and supporting manuals as
these become out-of-date and incompatible across larger administrative units such as
provinces.
Recommendation 2: In future activities that seek to develop the DBE1 tools and their further
use, major attention be given to the sustainability of the tools, their continuing technical and
policy support, and to the challenges of going to scale through implementation in larger
numbers of districts and provinces and dealing with larger-scale management issues.
Evaluation of DBE1 District Interventions
21
3. Evaluation of the Tools
3.1. Introduction
In this Chapter, the three key questions about the financial, data and planning tools are
addressed5. In summary, these three questions are:
Do the tools work?
Do the tools achieve the stated objectives of translating policy into implementation
action in Districts?
What is the early evidence of outcomes?
Answers to the questions are provided from three sources, first by reviewing the feedback
about the tools generally from district-level users; secondly, by examining one tool, SIMA, in
depth as a case study, third, by studying two good practice cases (that are presented in
Appendix 2). Conclusions structured according to the three key questions are then drawn
from the evidence.
During the evaluation, one provincial education office, twelve districts in five provinces, and
nine schools were visited to gather information about users experiences with using the tools.
Reviewing documentary evidence provided by DBE1 supported the fieldwork.
Tables 1 and 2 summarize the responses in relation to the main questions asked at district
level.
3.2. Do the Tools Work?
The answer to this key question is an unqualified „yes‟. This is true for each tool that was
evaluated in the field and true in each district that was using one or more of the tools
together. Feedback was consistently positive: the tools are useful for the range of purposes
for which they were intended including data analysis, planning and policy development.
Further evidence that the tools work is in the responses to other questions summarized in
Table 1 and 2 and in the more specific good practice case studies presented in Appendix 2.
Appendix 3 presents a simple case where accurate data about teacher characteristics was able
to save substantial sums of money to one district.
Can they use the tools?
Yes, those that have been trained can use all tools in all districts visited. There is evidence
that some replacement staff who did not receive training directly from DBE1 are struggling to
use the tools.
Are they going to continue to use the tools?
Yes, all districts visited are intending to continue using the tools. Some districts have
institutionalized these commitments by allocating budgets for this purpose, initiating internal
staff development activities to support continuity, and undertaken their own upgrading and
adaptation of some tools themselves.
5 The full set of questions addressed in the evaluation are in the scope of work in Appendix 1.
Evaluation of DBE1 District Interventions
22
The major threats to continuity of use lie in the loss of key staff, a lack of budget support, and
unclear strategies for securing continuing technical support after DBE1 has closed.
A rather perverse source of evidence that the tools do work lies in the persistent rumors that
unscrupulous third parties have copied the software and manuals and are providing these and
training services for sale at very high prices to schools and districts.
3.3. Do the Tools Achieve the Stated Objectives?
The tools have the capacity to support the stated objectives of the development of policies
and the subsequent translation of those policies into implementation action. No negative or
critical feedback was received about the tools‟ capacity to support these actions. Of course,
whether these steps are actually taken depends very much on the will and skill of users to do
so and a supportive local administrative and political environment.
The value of the tools for their intended purpose is highlighted by their widespread
acceptance and use in DBE1 districts, their dissemination to non-partner districts (of which
there were 28 to the end of June 2011)6 and take-up by The World Bank, AusAID and
UNICEF in their projects. The good practice case studies in Appendix 2 provide specific
examples of how the tools actually work at district level.
The current (November, 2011) intensity of continuing development and dissemination of the
tools only one month out from project closure further attests to their value to users.
Examples of where the tools have had an impact on policy and implementation are briefly
described in section 3.4 below.
6 DBE1, Quarterly Report No.25, July 2011, p. 39
Evaluation of DBE1 District Interventions
23
Table 1: Evaluation of Tools in Evaluation Study Districts
District (n = 12)
How useful are the tools?
Can they use the tools?
If no, do they still need help to use
them?
Are they going to continue to use
the tools?
If help needed help, do they want to hire a service provider?
Are the tools institutionalized in the education
office? West Java
Karawang Useful Yes. Loss of key staff an issue
Not now, but in future
Yes: planned and budgeted
Uncertain Uncertain; few staff use tools and staff losses
Kota Bogor Very useful for policy
Yes. All staff in planning section.
Not now, but in future
Yes: planned and budgeted
Yes Yes, all staff in planning section.
Lebak Very useful for planning
Yes. Loss of key staff an issue
Yes No; staff losses and no replacements
Subang Very useful Yes. Not now, but in future
Yes Uncertain Yes.
Central Java
Purworejo Very useful Yes No; self-reliant Yes No Yes. Also have clear internal capacity development strategy
Nangroe Aceh Darussalam
Aceh Besar Useful Yes Yes, needed for SIMA
Yes Yes; trained providers are in Provincial Dinas Pendidikan
Yes, special educational data group has been formed.
Kota Banda Aceh Useful Yes, but need help Yes Yes, but not budgeted
Uncertain No
North Sumatra
Kota Tebing Tinggi Very useful Yes; SIPPK modified for local needs
Yes, for further tool development
Yes. Updates planned
Yes, accessible local provider is LPMP
Yes
Kota Tanjung Balai Useful Yes. Loss of key staff an issue
Yes Yes Uncertain No. Data is held by individuals which is not sustainable
Evaluation of DBE1 District Interventions
24
District (n = 12)
How useful are the tools?
Can they use the tools?
If no, do they still need help to use
them?
Are they going to continue to use
the tools?
If help needed help, do they want to hire a service provider?
Are the tools institutionalized in the education
office? East Java
Sampang Useful Yes, but need help Yes, to upgrade software
Yes Uncertain – did know about providers
Yes. Further training of staff necessary
Kota Mojokerto Useful Yes Yes, but further training needed
Yes. Budgeted for 2012 Yes
Tuban Very useful Yes Yes Uncertain Yes. Further training of staff necessary
Summary All districts find the tools useful
All districts can use the tools. Main threats to continuing use are loss of staff and uncertainty about support.
Most districts will still need help for software development
All are going to continue use the tools. Major threats are lack of budget and need for support
Four want a service provider and seven are uncertain or negative
Eight say yes. Threats to institutionalization are staff losses and weak planning to support the use of tools
Evaluation of DBE1 District Interventions
25
Table 2: Evaluation of Selected Tools Implemented by DBE1
District How useful are
the tools? Can they use the
tools?
If no, do they still need help to use
them?
Are they going to continue to use
the tools?
If help needed help, do they want to hire a service provider?
Are the tools institutionalized in the education
office? BOSP
Sampang (East Java) Useful Yes Not now, but in future will need help to upgrade software
Yes Uncertain; do not know about service providers
Yes
Kota Mojokerto (East Java)
Useful Yes Yes: planned and budgeted
Yes, budgeted for 2012 Yes
Tuban (East Java) Very useful for policy making
Yes Yes Yes Yes
SIMA
Purworejo (Central Java)
Very useful Yes No; self-reliant Yes No Yes. Also have clear internal capacity development strategy
Subang (West Java) Very useful Yes. Not now, but help may be needed in future
Yes Uncertain Yes.
SIPPK
Kota Tanjung Balai (North Sumatra)
Useful No. Original trained person has moved; new staff cannot use the data generated
Yes Yes No. Data is still kept by individuals
Kota Tebing Tinggi (North Sumatra)
Very useful Yes. Uncertain. Have undertaken their own modifications.
Yes Yes
SIMPTK
Kota Tebing Tinggi (North Sumatra)
Very useful Yes Yes Yes Yes Yes
Kota Tanjung Balai (North Sumatra)
Very useful Yes Yes Yes No
Evaluation of DBE1 District Interventions
26
3.4. What is the Early Evidence of Outcomes?
The following „snapshot‟ of outcomes is brief but it reports the evidence that the tools are
leading to important management outcomes at each level of the education system. The
evidence is drawn from field visits and published reports from DBE1. More detail is provided
in the SIMA case study below and the two good practice studies in Appendix 2.
Implementation and dissemination in districts
Implementation and dissemination to districts has been substantial: as of end 2011, DBE1
district level tools have been implemented in 117 districts in 13 provinces7. The extent of
dissemination to non-partner districts can be assessed against the DBE1 target district8 total
of 70 districts in Table 3. In addition, the World Bank, AusAID and UNICEF are known to
be disseminating the tools. Regrettably, due to the constraints of the present study, it proved
difficult to obtain comprehensive data on their experiences with the tools.
Tools have been successfully used in schools, districts and at provincial levels. Table 3 shows
data on the extent of implementation of the tools, including dissemination.
Table 3: DBE1 Implementation of Tools
Tools
Cumulative Implementation in
Districts to 30 September 2011
(Including Dissemination) N = Districts
AKPK 70
BOSP 93
Updating BOSP 21
PBPSAP 26
SIMPTK 7
SIMA 9
Renstra 57
Source: DBE1. Annual Report, October 2010 – September 2011, p.2. Jakarta, DBE1, October 2011. (Draft)
Outcomes at school, district, provincial and national levels
School outcomes: School visits in Purworejo and Subang districts have demonstrated the
clear value of SIMA in asset management. These outcomes have been noted in a range of
schools from comparatively poor primary schools in unsatisfactory condition to a senior high
school seeking to meet international-standard school accreditation. As training was only
carried out recently in February 2011, achievements and outcomes so far are quite
remarkable. These brief cases from Subang illustrate outcomes in a range of different types of
schools:
7 Source: DBE1 Project Data Management System (PDMS).
8 DBE1 defines „Target District‟ as a district where most of the costs for implementing an intervention are borne
by the project. „Dissemination District‟ refers to districts that have implemented at least one DBE1 tool without
out DBE1 funding.
Evaluation of DBE1 District Interventions
27
SDN Manasuka Subang, has now accurately identified its assets. The poor condition
of these assets has alerted the District Education Office to the genuine needs of this
school for budget support and maintenance.
SMP Negeri 1 Subang has entered all its infrastructure and equipment data into
SIMA. This makes the infrastructure and equipment data up-to-date. In terms of asset
policy, the tool provides accurate information for further actions. The Provincial
Financial and Development Supervisory Agency of West Java carried out an asset
spot check in August 11, 2011. The result was that Agency is happy with the standard
of asset management in this school.
SMA Negeri 1 Subang is now able to produce assets reports to support school
management. These reports had been submitted to the District Education Office and
used by the Regional Inspectorate for asset spot checks with good results. Reports
were also submitted for accreditation. The result was that the school was awarded
more points towards the international school standard.
District outcomes: The Personnel
Management System (SIMPTK) has
been implemented in Purworejo since
early 2011. The results of this work
have been recognized in the local
media (see Box and link to full article
at: KRjogja.com).
District government found the results
of the use of SIMPTK to be valuable
and requested DBE1 to assist in the
preparation of district regulations to
support the implementation of
recommendations arising from the
analysis of data from SIMPTK. The
results focus on regrouping small
schools and the management of the
allocation of teachers to schools by
relocation or assigning teachers to
more than one school to ensure they
meet the required 24 hours teaching
load. SIMPTK is leading to quality
information that, for the first time,
enables this district to make better
quality policy decisions based on
accurate data and good analysis.
Impressive business outcomes from the use of SIMA are shown in Table 4: Administrative
Outcomes from Using SIMA in Subang District.
Equitable Teaching Hours: 116 Teachers
in Purworejo transferred
(KRjogja.com, 28 September, 2011)
The Purworejo district has determined the
need to transfer 116 junior and senior
high school teachers to meet national
government policy of teachers teaching
24 hours a week.
Transfers were implemented after the
education office mapped the distribution
of 2,038 junior high teachers, 715 high
school and 895 vocational teachers in the
state and Islamic school systems.
Before the transfers, a number of junior
and senior high schools still lacked an
appropriate number of teachers and other
schools had not met the required number
of teaching hours.
The education office is also planning the
implementation of the mapping of
Purworejo’s 5,227 teachers for primary
schools.
Evaluation of DBE1 District Interventions
28
Provincial outcomes 1 – East Java: The school unit cost analysis, BOSP, has been the main
district tool replicated with non-DBE1 funding. East Java is a case where strong support has
been provided by the provincial government. Recognizing the importance of data analysis for
planning, the East Java provincial government allocated Rp.205 million for replication to the
end of June 2011.
Provincial outcomes 2 – Central Java: Evidence of the dissemination of BOSP to non-
DBE1 districts is gaining public recognition and evidence has been provided in recent press
coverage. In Solo, Central Java, press articles, report how helpful the BOSP tool has been at
all levels of school education in developing a more complete and accurate understanding of
financial needs. An immediate outcome of the use of BOSP has been to plan for an increase
in budget in the coming period to address identified issues of underfunding. Relevant press
articles can be found in Jolosemar, 31 October, 2011 and Solopos, 31 October, 2011.
Provincial outcomes 3 – North Sumatra: In June and July 2011, DBE1 trained staff from
the Provincial Education Quality Assurance Office (LPMP) and the Provincial Education
Office in the use of the Integrated District Management Information System (SIMPK). The
LPMP has requested all districts to start using SIMPK to gather information about the
education sector and is intending to use this data as the basis for preparing the 2011 North
Sumatra education sector profile.
National outcomes: In late 2010, the then Deputy Minister of National Education and
Secretary to the Vice President asked DBE1 to enlarge the scope of BOSP to enable local
governments to calculate costs for achieving the Minimum Service Standards and to reach
school access targets. A new tool, called Penghitungan Biaya Pencapaian Standar dan Akses
Pendidikan (PBPSAP) or Calculation of Costs to Achieve Minimum Service Standards and
Universal Access, has been developed for this purpose and implemented in 24 districts by
July 2011 with a target of 50 districts completed in the final quarter.
Impact on data quality
A major impediment to good planning and governance in Indonesian education is the
generally poor quality of data that is available from government collections for these
purposes. In schools, districts and provinces using the DBE1 tools there is reported evidence
that the tools are demonstrating to users the consequences of poor data quality leading, in
turn, to the adoption of strategies to improve the quality of data collections, data management
and data use. The matter of data is considered to be so important in good management and
governance that a full Chapter has been devoted to this topic (Chapter 4).
Impact on participants in implementation
The enthusiasm and commitment among the many users we have encountered from schools,
districts and provincial offices we have visited are indicative of a positive impact that the
tools are having on individuals. We noted the indication of commitment to the tools through
the extended time voluntarily given to us by such senior leaders as the Head of the West Java
Provincial Education Office and the Head of District Education Office in Purworejo. School
principals, teacher-operators, and district staff gave similar non-verbal messages of
commitment. What factors appear to support this outcome? As reported to us, they are that
the tools are easy to use, simplify work and make it more productive and enjoyable and lead
to observable and immediate outcomes that benefit the organization from better
Evaluation of DBE1 District Interventions
29
understanding, to better plans and better standards of resourcing. Another factor is the
satisfaction from achieving external audit standards.
Impact on understanding change and development strategies
The DBE1 reports we have studied in this evaluation contain much useful contextual
information about successful implementation strategies, impacts on education system
stakeholders and lessons learned from these management tools. In aggregate, this information
is vital in planning and implementing future development activities. However, much of it is
dispersed through the DBE1 reports and risks being lost unless the information is gathered
and synthesized to enable ready access to it in future project implementation. Such a
synthesis could potentially have been a potent source of influence to help in addressing the
strategic issues documented in relation to the recent cascade training of principals and school
committee members throughout Indonesia9.
Nevertheless, the lessons learned are clearly having an impact on strategies for disseminating
the tools. Moreover, these lessons have been synthesized into a statement of principles that is
discussed in Chapter 7.
Recommendation 3: A review and synthesis study should be undertaken of DBE1’s research,
successful implementation strategies, and lessons learned, with the intention of producing
clear and simple guidelines for implementing change and development in Indonesian
education. The review should also consider published research, the documented experience
of other recent and successful development activities supported by USAID, other donors and
the Government of Indonesia.
3.5. Case Study: Asset Information Management System (SIMA)
Background
The implementation outcomes of SIMA illustrate the way in which this tool can facilitate
effective management of assets in ways that involves schools, districts and the province.
The Asset Information
Management System developed by
DBE1 is known as SIMA - Sistem
Informasi Manajemen Aset. The
lack of data in the past about
assets, such as buildings, partly
explains why the local press
regularly reports news about
damaged schools, collapsing
school roofs, and associated injury
to children and teachers (see
adjacent box).
9 DBE1, Quarterly Report No.25, July 2011, p. 38.
101,000 SD Classrooms In Dire Conditions
Posted on August 8, 2011 by bectrustfund
Kompas, Page 1
Minister of National Education Mohammad
Nuh said the government will allocate Rp2
trillion in the 2012 state budget and the
2011 revised state budget for renovating
schools in dire need of repair. It is hoped
that there will not be any more damaged SD
classrooms by 2014, he said.
Evaluation of DBE1 District Interventions
30
Until now, district educational administrators have not had accurate information about the
condition of these assets. SIMA has helped to provide this information in a form that assists
with this management challenge.
Two things complicate understanding of the asset situation in schools: the large number of
schools in many districts, and administrators, principals and teachers lacking the technical
knowledge to be able to make informed judgments about the conditions in schools.
„Assets‟ are land, equipment, machinery, roads, buildings, installations, irrigation systems,
and other fixed assets. SIMA consists of two components:
a computerized system to inventory these assets – SIMA
a training program for school and district level administrators on preventive
maintenance to accompany SIMA.
School-level implementation of SIMA
Asset data is prepared at school level. The task of doing this is usually the responsibility of
the principal and an operator – either the principal or a teacher. Completed data sets are then
sent to the district education office where they are used for planning and policy work.
In order to evaluate whether SIMA works at school-level, nine state schools were visited. We
were also interested in the way these technologies were received and being used in schools,
especially in poor rural areas. The schools visited were:
four rural primary schools in Purworejo, Central Java
four schools: two primary, one junior secondary and one senior secondary school in
urban areas in Subang, West Java
one special school in Cimahi, West Java.
In all of these schools, the following was noted:
Functional computers available with SIMA software readily accessible and school
data loaded. In all schools there was at least one competent operator, often an
enthusiastic teacher, who had responsibility for maintaining the asset inventory. No
major issues with the acceptance of the technology were identified.
A clearly well informed and enthusiastic response to SIMA in all schools.
Detailed feedback on operating issues, software performance and continuing support
needs were expressed. This feedback did not detract from the overall perception of the
good quality of SIMA and its usefulness to the schools.
There is clear evidence that the tool does work, and is well regarded, in the beneficiary
schools studied. Given the broad range of schools and locations studied in this present
evaluation, it is likely that SIMA will work in all appropriate schools where there is the
motivation to use it and the basic skills and computer resources to do so.
Evaluation of DBE1 District Interventions
31
District and provincial level implementation of SIMA
District asset data is derived from data entered at school level. Completed data sets are
aggregated at district level and used for planning and policy work. Provinces also use this
data in their work. In order to evaluate whether SIMA works at district-level, two district
education offices were visited to focus only on SIMA. The districts visited for this purpose
were:
Purworejo, Central Java
Subang, West Java.
In Subang, the education office disseminated SIMA to a total of 973 schools in March 2011.
Furthermore, the District Asset and Finance Management Office also asked to be trained by
DBE1 and the Education Office. Subang District provided this simple table to illustrate the
positive outcomes of using SIMA since its adoption this year in terms of time required to
administer assets, coverage of schools, and the quality of outcomes.
Table 4: Administrative Outcomes from Using SIMA in Subang District
Management Criteria Former Manual Process Using SIMA Time required to administer asset system
Approximately one year Approximately six weeks
Number of schools reporting assets
283 SDN/TKN, 31 SMPN, 14 SMAN/SMKN
853 SDN/TKN, 57 SMPN, 19 SMAN/SMKN
Other outcomes from processes used
Poor quality data Incomplete data sets Poor and slow return rates Intensive and repetitive annual work load in schools and in district
Better quality data Data more complete 100% returns from schools Reduces time for schools and education office in asset management
Source: Data provided by the District Education Office, Subang, West Java
From the visits to districts, the following key points were noted:
SIMA enables the verification of school asset reports so that districts and their schools
can properly plan for preventive maintenance. Verification is further provided by
regular external audit by the National Financial Audit Agency – Badan Pemeriksa
Keuangan (BPK).
The concept of verification through these external audits has been an important factor
in supporting the dissemination and the use of SIMA. In interviews with government
officials in District Education Office in Subang and in the Provincial Education Office
for West Java, it was made very clear that the need to meet the requirements of audit,
and the recognition of achieving audit standards, has been most important in
strengthening the acceptance and use of SIMA.
In addition to providing participating districts with the data to support improvements
in the quality of their planning and policy processes, there is good evidence that it is
having the outcome of demonstrating the often poor quality of data that is currently
available, leading in turn, to the adoption of strategies to improve the quality of data
collections and data management.
Evaluation of DBE1 District Interventions
32
The introduction of DBE1 tools in Purworejo has proved to be so successful that the
Central Java Provincial Education Office decided to use World Bank BEC-TF project
funds in eleven districts to support SIMA dissemination. The Head of the Education
Office of Purworejo, his staff and school principals are acting as facilitators to support
the World Bank project.
Dissemination of SIMA in West Java has been significant. Since 2010, DBE1 has
been providing assistance to the West Java Provincial Education Office to implement
SIMA in its schools for students with special needs (Sekolah Luar Biasa or SLB).
After DBE1 assisted in the implementation of SIMA in 15 special schools, the West
Java Provincial Education Office continued the process with dissemination to 14 other
schools. As a result of these activities, the Education Office was able to gain a
complete picture of the condition of assets in these schools and present the
information in a public consultation session to stakeholders for further policy
development and actions including support for schools in preventive maintenance.
3.6. Conclusion
In this chapter, three key questions about the financial, data and planning tools have been
addressed. These three questions are:
Do the tools work?
Do the tools achieve the stated objectives of translating policy into implementation
action in Districts?
What is the early evidence of outcomes?
There is compelling evidence that the tools work in the schools, districts and the one province
reported here, that they help achieve their intended objectives and that the evidence on
outcomes is positive.
Why these positive results have been achieved has been analyzed in terms of what we know
about change and development in Indonesian education and from the substantial field
experience that DBE1 has accumulated in both schools and districts during the past six years.
The results of that analysis are presented in Chapter 7.
Evaluation of DBE1 District Interventions
33
4. Data, Data Analysis and Implementation
4.1. The Quality of Data in Indonesian Education
The quality of educational data has been a matter of concern to DBE1. Experience with
districts shows that much of the data they have, and are meant to be reporting to the Ministry
of Education and Culture, is inaccurate, out of date, and often not available at all.
Good quality policy decisions that lead to good actions become very difficult in this
environment. Decision makers are left to use poor quality data or to make informed guesses
about the real situation. Worse, it is impossible to accurately monitor progress on achieving
policy goals and to make well-informed judgments about adjustments to both policy and
practices. To address the challenges presented to policy makers, many organizational units in
the education system operate their own independent data management systems in order to
address the shortcomings of the central data collections.
In this challenging context, the DBE1 data management tools fill an important need. The
implementation of these tools has demonstrated how directly involving stakeholders in the
collection, analysis and use of data reveals weaknesses that can be successfully addressed in a
bottom-up processes focused on newly understood needs and purposes.
A study by RTI International for DBE1 in 2007 10
found that data supply and validity was
low. The study further concluded that those who supply the data for national data collections
did not use it for their own purposes and therefore had little motivation to provide up-to-date
and accurate data. The study proposed the theory that increased use of data will result in
greater demand for data and hence a better supply of timely and valid data. This finding has
been a guiding principle in the development of all DBE1 planning and management tools and
its validity has been confirmed through field experience.
4.2. Data Analysis
DBE1 has given thorough attention to the analysis and use of data outputs produced by the
various tools. This attention is reflected in these ways:
The manuals supporting the software tools provide a number of constructive
procedures for transparent data analysis. Specifically, these include internal
consultations within the education office and external consultations with local
government organizations, stakeholders such as schools, and representatives from
community organizations, the media and non-government organizations.
Training materials such as DBE1 PowerPoint presentations located in district
education offices reveal that steps in analysis have been an important part of local
capacity development activities.
10
DBE1 Special Report. EMIS Assessment. Jakarta, June 2007.
Evaluation of DBE1 District Interventions
34
Recommendation 4: When future development of the DBE1 tools is to be undertaken, it is
recommended that further and continuing attention be given to the intellectual processes of
the analysis and judgment of data in education11
, to the impacts of corruption, and to the use
of data as illustrated in Figure 1.
Figure 1: Analytical Steps
(Source: Penghitungan Biaya Pencapaian Standar dan Akses (PBPSAP). DBE1, Jakarta, p.5 (Edisi Juli, 2011)
It is generally true in the domains of assessment and evaluation that people tend to be rather
better at, and focus more, on data collection and processing rather than on the skills of
judgment and the use of data for planning, policy development and decision making.
The analytical processes documented in the DBE1 training materials go some way to help
address the serious challenges to data integrity and quality planning and policy presented by
corruption in government systems. All of the care and work in developing tools and planning
procedures is for nothing when the goals are corrupted and not focused on achieving high
integrity outcomes for all stakeholders. Perhaps in future development of the tools and
implementation more attention needs to focus on this challenge.
11
Educational literature on teaching analytical and thinking skills is not easy to locate. A good source of recent
literature is provided in this journal article: Weiping Hu, et al. (2011) Effects of a „learn to think‟ intervention
programme on primary school students. British Journal of Educational Psychology, 81, 531-557.
Evaluation of DBE1 District Interventions
35
5. Service Providers
5.1. Key Questions
The evaluation design required attention to the following questions in relation to service
providers:
Is there an appreciation of the need to work with an external service provider?
Do the districts, working with service providers, have the capacity to use the tools
effectively?
What is the level of interest and need/demand for support services?
Are there potential service providers available with the capacity to meet that
need/demand?
What is the initial experience of potential service providers with the tools?
These questions were explored with users of the tools in DBE1 partner districts and schools,
with staff of the Provincial Education Office in West Java, and with DBE1 staff. An earlier
DBE1 study, The DBE1 Service Provider Program: An Evaluation (DBE1, Jakarta, March
2011) was also consulted for evidence.
5.2. Service Provision
On-going service provision is a key element in DBE1‟s sustainability strategy. It is intended
that service providers can take over the DBE1 programs, manuals and training modules and
implement them in districts without further project support when DBE1 has concluded. The
main agents for this dissemination and sustainability strategy at district level have been a
group of universities and government agencies such as the provincial educational quality
improvement agency or LPMP. In addition, other projects, including those projects funded by
the World Bank, UNICEF and AusAID have begun disseminating district level interventions
and providing support in their districts.
The March 2011 DBE1 study implies a level of dependence on DBE1 at this relatively early
stage of the tools development and implementation cycle. This is neither a surprising nor a
bad phenomenon at this time and is a form of evidence of the recognition of the actual utility
of the tools. For example, the study notes in the lessons learned section that: “…service
provider personnel require further training and field experience in order to become fully
confident as independent service providers in the delivery of these programs”12
. This
comment also raises the very old question expressed as: „who will train the trainers?‟ or „by
what mechanism will this model of service provision be sustainable in itself?‟ To some
extent, this matter has begun to be addressed through DBE1‟s certification of service
providers program but we feel that more will be needed to ensure the sustainability of high
quality service provision into the future13
.
12
DBE1. The DBE1 Service Provider Program: An Evaluation. March, 2011. 13
DBE1. The Certification of Service Providers for DBE1 Programs. September, 2011.
Evaluation of DBE1 District Interventions
36
In the context of the tools of interest in this evaluation, service provision can be conceived as
providing any one or more of the following services to districts:
Systems analysis
Software development
Training in software operation and in analysis and use of outputs
Policy monitoring and review
Updating software and training to reflect policy
Continuing and timely support and advice, particularly in the use of data and in the
preparation of various planning documents and policies
Monitoring, evaluation and feedback.
In the past, DBE1 has been providing each of these services to provinces and districts as an
integrated „package‟ in constructive collaborations with key staff at district and province
level. Effective service provision that can replace DBE1 following project closure will
require an approach modeled on this integrated approach if it is to achieve the high standards
already set by DBE1.
Integration of support services is essential. It
has proved to be a successful approach by
DBE1 in which each type of service supports,
and is supported by, each of the others.
Whether effective services can be provided
using this model is a challenge and needs to
be tested over more time than is now
available to DBE1.
Service provision is challenging because the
provider will need to develop and maintain a
team of specialists to undertake coordinated
services. Former DBE1 specialists can be
recruited into these teams and can generally
provide services with little further training.
Developing and maintaining a team of
specialists is likely to be expensive and
require each team member to keep up-to-date if they are to remain relevant to the needs of
districts. Whether potential district clients will be able to afford such services is a question
that needs to be tested in future if such services are to be provided on a commercial or cost-
recovery basis. In this context, it may be that the only sustainable model of service provision
would be through a government institution that can draw on appropriate expertise and
provide services free or on a subsidized basis if that institution can absorb at least some of the
costs.
The need for integration of services
is illustrated by considering the
reported experience in Purworejo
district where an external service
provider team introduced a
university-developed software
package. But then that team failed
to provide continuing and timely
support and advice and was unable
to keep up with critically important
policy changes.
The service providers understood
little of this because the team had
not undertaken any monitoring or
sought feedback on their services.
Evaluation of DBE1 District Interventions
37
5.3. The Need for External Service Provision
There is a clear need for service providers into the immediate future. District data presented
in Table 1 shows that most districts visited believe they will need help with the tools.
Conversely, Table 1 also shows that most districts express uncertainty about wanting to hire a
service provider. The reasons for this attitude need to be explored and addressed more fully in
subsequent development activities.
For the district and provincial staff met during this evaluation, „services‟ to them so far have
been provided by DBE1. These services included all or some of the matters listed above in
paragraph 5.2: Service Provision. There is unqualified appreciation of the high standard of
these services in the field and of the approach and helpful attitude of DBE1 staff in providing
them.
An observation of the quality of these DBE1 services made by the Head of the District
Education Office and his staff in Purworejo is worth reporting here as it illustrates local
appreciation of how quality service provision has an impact on local attitudes to the tools and
eventual outcomes. Compared to some other donors operating in Purworejo, DBE1 training
and support has been provided in a timely and professional manner. For example, relevant
and practical training extended over the whole of the planned period. In other words, a three-
days workshop from 0800 on Monday to 1600 on Wednesday began at 0800 on Monday and
concluded at the planned time on Wednesday. Unfavorable comparisons were made with the
chaotic approach to similar training provided by another donor to this district.
District staff made another key point about service provision and this is the provider‟s
capacity to react quickly to changing circumstances and policies. This capacity is something
DBE1 has had, both in terms of technical operational concerns, in terms government policy
changes, and providing an integrated support service spanning all of the district‟s needs and
concerns. Considerable doubt was expressed about the capacity of others to provide timely
and up-to-date support in this way.
Therefore, based on the experience of working with DBE1 as the „service provider‟, the
answer to the question: “Do the districts, working with service providers, have the capacity to
use the tools effectively?” is an unqualified „yes‟. But in relation to other service providers,
there is uncertain evidence at this time. Some districts providing SIMA services to schools
have proved their competence to do so, but there is a question mark over their physical
capacity to extend services all schools in their districts. It is a problem of going to scale.
Do the districts, working with service providers, have the capacity to use the tools
effectively? The evidence in Tables 1 and 2 suggests that districts are able to use the tools
effectively on their own. In addition, the DBE1 Service Provider Program: An Evaluation
suggests that the answer to the question is „yes‟.
The present evaluation team did not visit sufficient districts who had worked with service
providers other than with DBE1. Therefore, this question cannot be answered directly. The
West Java Provincial Education Office has been providing services to special schools (SLB)
in the province and this support has been effective as evidenced by the observed use of the
SIMA tool. District education office staff in both Subang and Purworejo have trained their
schools in SIMA and this support has been successful. Districts clearly have the capacity to
use the tools observed effectively as do the schools visited that have benefited from the
Evaluation of DBE1 District Interventions
38
services provided to them. The real challenge for the education offices is to scale-up their
support to all schools.
Finally, it seems to us that there are two different scenarios in which such services might be
provided:
Orientation and initial capacity development through training and coaching for
districts using the tools for the very first time
More advanced support for districts that have established a good level of competence
in using the tools and require more specialized support.
It is likely that serving this latter group will be the most challenging.
5.4. Evaluation of Service Providers
The reported experience with some university-level service providers is not encouraging.
This relates to three reported issues: status, credibility, availability and service content.
The perception of a status distance between a university lecturer and a district education
office official can be a barrier to an effective working relationship. District staff tend to feel
more comfortable with DBE1 trainers and District Facilitators. This perception of university
lecturers is reported in some districts as being compounded by knowing that the lecturer,
while having good theoretical knowledge, will often lack the credibility that comes from the
practical knowledge that comes from the routine use of the tools in a real working situation.
Complaints were made of a tendency by university lecturers to be unavailable at times when
needed by districts due to the demands of their other duties. It is not known if this is a valid
complaint or one that mainly reflects the way in which districts have been „spoiled‟ by the
readiness of DBE1 to provide support, perhaps as a result of a telephone call. Nevertheless, if
a service is to be provided at all, it is reasonable to expect that service to be available when it
is needed. This issue was also identified in the DBE1 study of service providers and was
discussed as the difficulty in „harmonizing agendas‟14
.
It may be misleading to generalize too much from the information presented here. There is
some evidence that there is variation between provinces in relation to service providers. In
both West Java and East Java, where Universitas Pendidikan Indonesia Bandung and
Universitas Negeri Malang respectively provide services to their province, there were
expressions of uncertainty at district level as to who to contact for support. This raises further
questions about accessibility of services. In North Sumatra, training and support was
provided by the LPMP for district staff but no follow-up support was given.
5.5. The Level of Interest and Need for the Tools and Support Services
In all districts visited, there is a high level of interest in the tools and a strongly felt need for
them (Table 1 and 2). Clear and consistent evidence of interest and need was found across all
institutions and was evidenced by the observable enthusiasm of users and by the immediacy
of being able to demonstrate the tools in action and to show hard or soft copy evidence of
outputs for analysis and planning.
14
DBE1. The DBE1 Service Provider Program: An Evaluation. 2011.
Evaluation of DBE1 District Interventions
39
The interest partly reflects a strongly felt need for these tools and partly an intrinsic interest in
tools that generate useful and revealing data. Moreover, the tools have a direct benefit in
helping to improve data quality so that administrators at all levels, and for the first time, are
beginning to get a more accurate picture of the state of education within their jurisdiction.
At school level with SIMA, principals and teachers realize and accept the need to have proper
control of their assets so that these can be better managed and developed. At district and
provincial level there are two kinds of needs. One need is to have accurate knowledge of
conditions for planning and financial management purposes and the other is to meet the
requirements of external audit by the Financial Audit Agency – Badan Pemeriksa Keuangan
(BPK) and to avoid the negative consequences of a „disclaimer‟ that may bring great shame
on the receiving organization. Among all the forces for development and change this one
emerged as a particularly powerful one. Conversely, there was no strongly expressed interest
or need for external service provision. However, this view needs to be considered in the
context of DBE1‟s present levels of well-regarded support. When this support ceases, what
then?
Two scenarios seem possible. First, in the short-term, where the policy environment remains
relatively stable and districts and provinces are able to continue to use the current versions of
tools without any external support. In most districts visited they seem very well able to do
this. Second, in a changing policy environment, provinces and districts will find a need for
technical and professional support but, most importantly, technical changes will need to be
coordinated and harmonized so that a multitude of incompatible technical responses do not
proliferate. Could service providers do this? This coordination is best done by government
that can exert some control and authority over continuing developments. Other service
providers such as universities and NGOs are unlikely to be able to achieve this kind of
coordination as they themselves will need to be coordinated which is improbable in a
competitive service environment.
5.6. The Experience of Service Providers with the Tools
DBE1 has identified a range of potential service providers that include:
District facilitators who are individuals trained by DBE1; the majority of these are
school supervisors working under district education offices or, in some cases, the
Ministry of Religious Affairs.
Institutions that are independent of the schools and education systems to which they
provide a service; these are universities, non-government organizations and
independent consulting agencies.
Institutions from within the education system; these include, for example, government
agencies such as LPMP and the West Java provincial education office that currently
provides a service in support of SIMA in districts.
In addition, this evaluation revealed an active network of schools in Purworejo district that
have agreed to provide support to each other. This networking concept could be considered
for further development, although natural systems such as these may be damaged by
bureaucratic control.
Evaluation of DBE1 District Interventions
40
Only one source evidence was available during the evaluation to address the matter of service
providers‟ experiences – the experiences of provincial and district government officials
working with SIMA. These officials have provided initial training and support directly to the
schools for which they are responsible.
According to officials that have provided training support to schools in West Java and in
Subang and Purworejo Districts, these activities have proceeded smoothly. The principals,
teachers and support staff interviewed during the evaluation, who had participated in the
training provided, verified that this is true. They expressed satisfaction with that training in
terms of its utility in understanding concepts, and in practical operating skills. Further
evidence of success in providing this service is in the outcomes of that training – the quality
of data entry for SIMA and the production of good quality outputs for transmission upwards
to the district education office through the sub-district office.
In the case of schools and the „service provider‟ education office to whom they report, the
roles of service provider, supervisor and client are confused and may make schools
disinclined to seek help from those to whom they may report.
In fact, it is beyond the physical capacity of existing staff of district offices to provide on-
going support to all schools at all times considering the very large numbers of schools in most
Districts. Schools in Purworejo have already addressed this difficulty creatively by working
together and supporting each other in local, informal networks of users. When sustainability
and support is being considered in future development activities, it is recommended that this
local network model be examined more closely as a potential strategy for continuing school
support.
We were also asked the question: “Are there potential service providers available with the
capacity to meet needs and demands?” We were unable to explore this question within the
constraints of the evaluation, but believe a positive answer has already been established as
indicated in reports produced by DBE1. The DBE1 Service Provider Program, An Evaluation
(DBE1 Special Report, March 2011) is supportive of this conclusion. So too is the more
recent report The Certification of Service Providers for DBE1Programs (DBE1 Special
Report, September, 2011).
5.7. Networked Support Systems
One of DBE1‟s concerns is to support the continuing development and dissemination of tools
and strategies. Can this be done through informal networks of teachers and administrators?
The answer is „yes, it can be done - and it is being done now‟. These are good indicators of
the power of local commitment and responsibility within districts that needs to be understood
and sensitively supported to ensure continuation.
Centralized, top down strategies for change are likely to be less effective than decentralized,
bottom up strategies. Three reasons justify this assertion. The demonstrated track record of
top-down, large-scale education projects in Indonesia is poor. As scale increases, the capacity
to manage an activity and to serve beneficiaries in a coherently professional way suffers.
What appears to be more efficient and effective is the capacity of local people to prioritize
their development needs. The processes of tool development encouraged by DBE1 have
Evaluation of DBE1 District Interventions
41
clearly demonstrated the success of this approach. This is evidence that decentralized, bottom
up strategies do work.
Much development support is occurring through the multiplicity of networks that exist in the
education sector. This is a less tidy, localized and flexible approach than is customary with
the normal approaches adopted in development assistance. Dramatic shifts in communication
technology demand that attention be given to this practice of bringing about change and
development through the power of groups working collaboratively and through the power of
networked computers. Primary schools in Purworejo that have supported each other in their
work with SIMA illustrate the potential of such networks. The cluster concept can also be
applied where small clusters of mutually supporting districts can assist each other and
strengthen sustainability. The processes through which these networks operate deserve close
analysis and dissemination through good practice networks such as those supported by the
World Bank.
5.8. Conclusions about Service Providers
This is an early time to draw firm conclusions about the service provider concept as it is now
implemented at district level. It is a relatively new concept having and so experience is
limited overall and non-existent in the case of some of the newer tools. Based on the
evaluation we have undertaken, we believe the following conclusions can be made.
The concept appears to have considerable merit as a sustainability strategy and
warrants further development and close monitoring. The recommended post-
implementation study of service provider impact by USAID in the years following the
close of DBE1 is strongly supported (DBE1 Service Provider Program, An
Evaluation (DBE1 Special Report, March 2011)).
DBE1‟s commitment to strengthening the concept through certification is evidence of
informed commitment to a worthwhile concept.
There is a need for advocacy of the concept at district level and for service providers
to actively sell integrated packages of services to overcome the hesitancy identified in
this evaluation. The concept will not succeed unless it is backed up by a commitment
to high levels of professional performance in service provision and customer service
by certified individuals and institutions. Initially, former DBE1 consultants can
provide such high standard services.
In addition to providing certification, the matter of professional performance will have
to be addressed in the future. The frequency of positive feedback about the standards
set by DBE1 and negative comments about services provided by other donors and
government agencies is a strong indicator that this matter is important and urgent to
ensure capacity development and support succeeds as intended.
Evaluation of DBE1 District Interventions
42
Recommendation 5: When subsequent development activities adopt the service provider
model, (a) the operation and acceptance of this approach to sustainable development be
carefully evaluated to ensure that quality, integrated support services (including networked
services) can be advocated and delivered in ways that meet clients’ needs in a timely,
professional and cost-effective manner; (b) service provider capacity building needs to begin
as early as possible as it requires significant levels of continuing training and mentoring; (c)
service providers be recruited from among the pool of knowledgeable and experienced
former DBE1 national consultants.
Evaluation of DBE1 District Interventions
43
6. Sustainability
6.1. Introduction
Are the tools and the implementation strategies sustainable? The sustainability of a tool refers
to its continuing use, and to the continuation of the benefits from using the tool, into the
future. Sustainability also demands that the tools and their related implementation strategies
are affordable to users. At provincial, district and school levels there is a strong view that the
tools and strategies are sustainable when judged against these criteria. The tools meet the
needs of beneficiaries, and they are affordable, relevant, up to date and practical. As
encouraging as this feedback may be, this is a „static‟ view largely representing the current
circumstances. Unfortunately, there are threats to the sustainability of the tools as we look to
the future.
6.2. Threats to Sustainability
The main threats to the sustainability of the tools come from the organizational and
institutional environments in which the tools are used. Strategies for managing these threats
are identified below. Threats to sustainability are not clearly related to the tools themselves or
from the ways they were designed, implemented and supported. The evidence suggests that
DBE1 has addressed these matters as well as it could have done. However, a thorough risk
analysis would strengthen understanding of the threats to sustainability and help prepare
evidence-based strategies to address them.
Threats in the institutional environment
The institutional environment is one level of analysis for examining potential threats. This is
the „big picture‟ environment in which tools are used. The institutional environment of the
district education office, where the tools are mostly implemented, is its relationship with
national and provincial governments (and with their policies and practices), its relationships
with other local government organizations, to non-government organisations, schools and to
the local community.
In this institutional environment, a significant threat to the sustainability of the tools is the
policy „threat‟. Recent experience with the changing policy environment indicates the
necessity of constant monitoring of Indonesian government policy and the capacity to
interpret and reflect these changes quickly in the financial and management tools. The
available evidence from the cycles of development and adjustment of the tools so far
indicates that DBE1 has been able to do this well. But the challenging question is „who will
do this work after DBE1 has closed?‟ Ultimately, government must take responsibility for
this.
Another threat to sustainability in the institutional environment comes from alternative
software that is intended to perform similar functions. Here there are two types of threat. The
first type is from institutionally mandated software of which the Sistem Informasi Manajemen
Daerah (SIMDA) – the District Management Information System15
– is one example. This
tool performs some similar functions as DBE1‟s SIMA, but it seems to be more complex and
15
Software for asset management developed by the National Development Auditing Agency (Badan Pemeriksa
Keuangan Pembangunan (BPKB)) and advertised for sale to local governments for Rp.1 billion.
Evaluation of DBE1 District Interventions
44
less user-friendly than SIMA. The threat is that senior administrators in other government
agencies can mandate the use of a tool such as this that is less user-friendly and also does not
address the needs of schools and districts effectively.
The second type of threat is from software developed by other organisations and donors of
which the Tool for Reporting and Information Management for School (TRIMS) is an
example. TRIMS, according to the Purworejo District Education Office, is unsuitable because
of its complexity and its lack of continuing and accessible support, yet this software could
well be selected and mandated rather than on the basis of district need and proven
performance of the DBE1 tools.
Threats in the organizational environment
In the organizational environment, that is within an organization such as an education office,
a significant threat is from staffing turbulence. This threat is well known and is illustrated by
cases where committed and competent leaders and key, recently trained staff, are transferred
out to some other government organization. This results in the loss of essential leadership or
administrative skills to implement the tools effectively. This threat is now being addressed in
some districts where commitments have been made to retain key staff unless movement is
essential for reasons such as promotion in the local government service.
In Purworejo this threat is recognized and is also being addressed through a „layered‟
capacity development strategy to ensure skills are retained. This strategy requires trained
operational staff demonstrating the software to their supervisors who then undertake the same
kind of demonstration and advocacy to the level above them, up to the level of the head of the
district office who ultimately advocates the outputs of the software to local government.
District education offices could be guided to develop strategies for sustainability and the
institutionalization of tools in further development activities using good practice examples
such as this.
6.3. Addressing the Threats to Sustainability
Indonesian Organizations
The evidence from this evaluation is that there are examples of the Government of Indonesia
at provincial and district levels taking the responsibility that is theirs to ensure sustainability
of the tools and good practice applications. The use of SIMA by the West Java provincial
government and the funding support from the province for dissemination of tools in Central
Java is another.
Local service providers could, in principle, help with sustainability by offering support that
addresses these threats. They could also use organizational development expertise to help
districts manage the turbulence threat by adopting effective human resources management
strategies. But it is unlikely that any current service provider could extend its influence to the
national level and would likely be limited to its own province.
Evaluation of DBE1 District Interventions
45
External Donor Support
There may be a continuing role for external donor support in assisting to document and
disseminate good practice examples of the commitment and leadership shown by some
provinces and districts. However, this is a function that can be taken on by government with
little difficulty.
In terms of the general model of sustainability proposed by DBE1 as shown in Error! Not a
valid bookmark self-reference. assumes the technology is settled at Year 1. With later
implementation as shown in Figure 3, there will be limited time for districts and partners to
effectively transfer.
Figure 2, below, this may need to be adjusted in the light of recent experience to recognize
the relatively late development and implementation of some tools of which PBPSAP from
2011 is one example.
The general proposition for sustainability set out in Error! Not a valid bookmark self-
reference. assumes the technology is settled at Year 1. With later implementation as shown
in Figure 3, there will be limited time for districts and partners to effectively transfer.
Figure 2, is a sound one and is supported by the experience of several recent projects.
However, when the logic is applied to very recent developments and tool implementation, it
illustrates the challenge for sustainability and the great importance of some kind of robust
national responsibility and service provision for the time when on-going external project
support is not available.
Recommendation 6: To strengthen DBE1’s approaches to sustainability, a thorough risk
analysis should be undertaken of the tools and of the institutional and organization
environments is which the tools are used. The results of that analysis should be used to
develop evidence-based strategies to further strengthen the approaches to sustainability
currently being implemented. As the professional staffing stability of the education office is
critical in sustainability, this key organization should be guided to develop strategies for
sustainability and the institutionalization of tools in further development activities.
6.4. Models of Sustainability
Error! Not a valid bookmark self-reference. assumes the technology is settled at Year 1.
With later implementation as shown in Figure 3, there will be limited time for districts and
partners to effectively transfer.
Evaluation of DBE1 District Interventions
46
Figure 2: DBE Transition Strategy
Figure 3: Transition Strategy for Later Implementation of Tools (Year 4-5)
Year 1 Year 2 Year 3 Year 4-5
Lev
el o
f E
ffo
rt
Lev
el o
f ef
fort
Districts & partners
DBE1
Year 1 Year 2 Year 3 Year 4-5
Lev
el o
f E
ffo
rt
Lev
el o
f ef
fort
Districts & partners
DBE1
Evaluation of DBE1 District Interventions
48
7. Conclusion: The Development of District Management and Governance
7.1. The Role of Principles of Development
One important role of evaluations is to make contributions to theory building based on the
evidence of practices in the field. Such a contribution is important because it provides strong
support for development activities that will follow DBE1. It helps to build on past
achievements and to stop „re-inventing wheels‟16
.
DBE‟s approach is strongly grounded in good development theory and professional practices
based on the experience of lessons learned in the Indonesian schools and in the Indonesian
government context. An understanding of the principles and theory supporting change in
district educational development by all participants in the development and change process is
essential. It is essential for three reasons.
First, participants must understand the foundations of what they are doing and why
they are doing it otherwise their work will often be „directionless‟.
Second, theory is essential so that problems encountered in the implementation of
change can be solved independently and in logical ways by reference to this theory.
Third, truly professional practitioners must be able to explain what they are doing and
why. They can best do this if they have a good theoretical understanding and
knowledge of the principles of change and lessons learned from past experience in the
field.
7.2. DBE1 Principles for Technical Assistance
In a background document prepared for this study, DBE1 has articulated a set of „principles‟
for providing technical assistance as follows:
Focus on developing products that local governments are required to implement.
Products and procedures must be based on – and intended to implement – government
regulations; project interventions must be modified as regulations change.
Utilize technology (software and hardware) commonly in use or available in the
districts (Microsoft Office Excel or Access are the underlying software for all DBE1
applications).
On-the-job training must follow introductory training for local implementers until a
product of quality is achieved. (One-off training is never effective for developing
quality products.)
All interventions include review and inputs by local stakeholders to promote
participation, transparency and accountability.
All tools and procedures incorporate facilitation for policy alternatives based on data
analysis.
16
In this context, attention is drawn to Recommendation 3.
Evaluation of DBE1 District Interventions
49
All tools and procedures use currently available data; they do not require additional
specific data collection.
The quality of products is determined by the quality of data; local stakeholders
determine the quality of data sufficient for their purposes17
.
These principles are reflected in Figure 4 below. The figure suggests that it is possible to
begin the processes of change and development at most points in the cycle, provided there is
a constant process of feedback and interaction between intended beneficiaries and DBE1.
Figure 4: DBE1 Assistance for District Management and Governance
7.3. Lessons from Research
One key lesson from past experience is that many development initiatives fail because it is
assumed that a focus on the „obvious‟ technical merits of a proposed change will cause it to
work in the field. This approach fails because it ignores the complex social setting such as
institutional culture in which the change is intended to occur. There were several instances
from districts visited where this approach was reported to be the case with software tools
introduced by other local organizations and donors.
17
DBE1 Tools and Procedures to Support Local Government Education Management and Governance,
November 2011.
Evaluation of DBE1 District Interventions
50
In his book Indonesian Education (2005), Christopher Bjork presents a study of how teachers
and principals responded to a central government technical policy to introduce the „Local
Content Curriculum‟ in the 1990's18
. This was an early example of the decentralization of
education. Bjork‟s study indicates that a deep understanding of the cultural, social and
regulatory context in which policies are to be implemented is required rather than a focus on
technical matters alone. Decentralization of the curriculum required a change in institutional
culture, but government had only been addressing technical aspects of change.
Change is not simply implementing a technical solution, such as training in how to use a
particular software package. Change involves a complex set of social issues as well. These
issues may include government policy and practice (of which a concern for the outcomes of
government auditing was one practice that emerged in this study), local administrative and
political leadership, and the commitment to improve. It is essential for sustainable change to
understand these social settings and to take them into account when planning change. DBE1
has done this work. In fact, our conclusion from this evaluation study is that greatest threats
to sustainability come from this setting rather than from any technical or implementation
weaknesses in the tools. Whatever the clear technical merits of the various tools and the
application software and manuals, the ultimate success of them working as intended has
depended very much on a supportive social context.
Similar lessons were learned from implementing school based management change in
Indonesian schools. These lessons were identified and reported in an impact study undertaken
by DBE1 of its concurrent schools program19
. Eight factors were found to be associated with
project impact. All of these factors have direct parallels with factors observed at district level.
The eight factors are:
The program was school-based and involved all members of the school community.
Training was provided on-site.
Training was ongoing and included mentoring; one-off training events rarely result in
successful reform.
The project worked with and strengthened local systems and institutions.
The program was firmly and explicitly based on government policy.
Technical assistance rather than funding was provided.
The program was manageable and affordable for local partners.
Commitment was built at provincial and district level.
In 2001, USAID‟s Center for Democracy and Governance published a paper setting out
lessons learned from the Implementing Policy Change Project20
. The lessons are relevant to
DBE1‟s work because the implementation of the various tools is intended to either support
policy or to generate information that can lead to policy change. The paper identifies a series
of tasks that project experience found are integral to improving policy implementation.
18
Bjork, C. (2005). Indonesian education; teachers, schools and central bureaucracy. New York: Routledge. 19
Heyward, M.O., Cannon, R.A. and Sarjono (2011) Implementing school based management in Indonesia:
impact and lessons learned. Journal of Development Effectiveness, 3, 3: 371 – 388. 20
USAID. Centre for Democracy and Governance. (2001) Policy implementation: what USAID has learned.
Washington, DC: USAID.
Evaluation of DBE1 District Interventions
51
The evidence from this present evaluation of tools demonstrates how these tasks have been
supported by DBE1 in its work with the Government of Indonesia at all levels, and
particularly at district level, where the bulk of activity has been carried out. The identified
tasks, and the evidence of how DBE1 supported them are:
Policy legitimation: is the policy accepted as worthwhile and is there a „policy
champion‟?
DBE1: Policies have been generally „accepted‟ because they originate from the
highest levels of central government. In many cases observed in the evaluation,
there has been strong commitment to the tools and policies from district and
provincial leaders who have undertaken the role of „policy champion‟.
DBE1: As users at several different levels in the organization begin to see the
value of the data outputs in simplifying and enhancing the quality of the work that
has to be done, acceptance is further strengthened. The use of SIMA to reduce the
repetitive amount of data collection and entry is a good example of this.
Constituency building: what steps were taken to identify and mobilize support and
deflect criticism of those that opposed it?
DBE1: Much of the constituency-building has worked outwards from interested
organizations – sometimes operating alone or having been positively influenced
by the work of other organizations – rather than being the outcome of an active,
project-led initiative. As USAID has pointed out in their study (p. 5), active
participation is a key element in constituency building and this has been evident in
every organization visited in the present evaluation.
Resource accumulation: what are the resources required to implement the policy?
DBE1: In the case of the tools, the main resources required to implement the
policies have been skilled human resources and very basic computing equipment
and software. The human resources have been of two kinds – committed and
supportive administrative leadership and the technical skills to undertake the
necessary work including both computing skills and analytical work necessary to
use the data wisely.
Working through organizations and structures of implementing agents.
DBE‟s work in districts has been with and through existing organizational
structures, not creating new ones. These structures have been enabled to carry out
their technical functions and, through this enabling process, have also been
assisted to strengthen the organization. This idea has extended to the strategy of
establishing „service providers‟ through existing organizations that are intended to
be a sustainable resource for continuing development.
Mobilizing actions.
DBE1‟s actions build on the favorable constituencies assembled for policy
implementation and operate within the implementing organizations, harnessing
and focusing their commitment, their needs and their resources towards agreed
action strategies.
Evaluation of DBE1 District Interventions
52
Establishing and using monitoring systems.
DBE1 constantly monitors its work and this study is one example of that
commitment. It commissioned an internal evaluation of the tools in 2010. DBE1‟s
continuing contact and support of districts and provinces has enabled direct
monitoring of the implementation of tools and immediate support or adjustment of
approaches to meet local needs.
From their study of policy reform in developing countries, Brinkerhoff and Crosby21
came to
a similar conclusion that seems to fit the DBE1 experience: “Instead of identifying ideal
solutions up-front and top-down, policy implementers need to iteratively develop second or
third-best answers that collaboratively agencies and stakeholders can agree upon.” Although
DBE1‟s solutions have not been „second best‟, key concepts in this conclusion that do
describe DBE1‟s approach are „iteratively‟ and „collaboratively‟ as illustrated in Figure 4.
These concepts are also illustrated by the steady evolution of DBE1‟s technical approaches
over six years with feedback from local users and the progressive development of software in
consultation with users as well as with policy-makers at the highest levels in the relevant
Ministries.
21
Brinkerhoff, D. and Crosby, B. (2002). Managing policy reform; concepts and tools for decision-making in
developing and transitioning countries. Bloomfield, CT: Kumarian Press. p.6.
Evaluation of DBE1 District Interventions
53
Appendix 1: Evaluation Study Plan
EVALUATION STUDY PLAN
for
SHORT-TERM TECHNICAL ASSISTANT (STTA)
SENIOR RESEARCHER TO CONDUCT EVALUATION OF DBE1 DISTRICT
LEVEL INTERVENTIONS
1. Background to the assignment
The objective of DBE1 is to develop more effective decentralized education management and
governance. In order to achieve the overall project objective, DBE1 developed, piloted and
disseminated tools to improve the management and governance of basic education.
Specifically at district level, a comprehensive toolkit has been developed and District staff
trained to assist with financial analysis, data management, analysis and planning as follows:
Financial Tools:
District Education Finance Analysis (AKPK)
School Unit Cost Analysis (BOSP)
Calculation of Costs to Achieve Minimum Service Standards and Universal Access
(PBPSA), which incorporates and supersedes the former BOSP
Data and Planning Tools
District Education Planning Information System (SIPPK)
Education Asset Management Information System (SIMA)
Personnel Management Information System (SIM-PTK)
Integrated District Management Information System (SIMPK), which incorporates the
former SIPPK and SIM-PTK
District Strategic Planning manual (Renstra)
Personnel from universities are being trained and mentored to act as potential service
providers in the use of the tools in districts.
In 2010 an initial assessment was conducted by the DBE1 M&E specialist of the use of key
methodologies: BOSP, AKPK, SIPPK, and renstra (strategic planning).
2. Objective
"The objective of the assignment is to maximize the acceptance, widespread use and the
implementation of the DBE1 results at both school and district levels." To this end, the final
report, focusing on Districts, will include lessons learned and recommendations for the use of
DBE1, current and future development assistance programs and partners, and for potential
local service providers.
Evaluation of DBE1 District Interventions
54
To address the objective of the assignment, an evaluation of the tools’ acceptance by users
and the use of the tools will be undertaken. The key issues/questions to be answered in the
evaluation are as follows:
Clarify and describe the tools.
Do the tools work? Do they achieve the stated objectives of translating policy into
implementation action in Districts? If so, why and how? If not, why not?
Is there an appreciation of the need to work with an external service provider? Do the
districts, working with service providers, have the capacity to use the tools
effectively?
Are the tools and the implementation strategies sustainable? What is the level of
interest and need/demand for the tools and support services? Are there potential
service providers available with the capacity to meet that need/demand?
What is the experience with the dissemination, institutionalization and
adaptation/evolution of the tools? This must include the experience of other donors
(World Bank, UNICEF, AusAID).
What is the initial experience of potential service providers with the tools?
What is the early evidence of outcomes? (The study may consider early evidence of
outcomes – at District level and beyond at school level and national level – should
these exist. Outcomes for non-organizational phenomena should also be considered
such as on the quality of data and on people/participants in implementation. Is there
early evidence of outcomes and, if so, what are factors that appear to support these
outcomes?)
3. Methodology
The principal methods to be adopted to achieve the objectives will include:
Review and analysis of central documentary evidence from DBE1, including the 2010 study.
Interviews with key DBE1 staff involved in the design, development and implementation of
the tools and with users.
Good practice cases to be determined on a purposive sampling basis taking into consideration
(a) location, (b) tool and (c) Districts that may have begun working with potential service
providers. A key purpose in selecting cases will be their utility in demonstrating good
outcomes that support the achievement of the objective of the assignment (2, above).
Good practice case studies will be based on interviews, observations of how tools are used
and by whom, outcomes/issues emerging, local documents and group methods, and
exploration of links to both planned and unexpected outcomes and to emerging issues.
Sequence: Good practice (GP) case study by researchers followed by supporting studies by
field staff if determined to be feasible.
GP criteria will be adapted from earlier work by DBE/UNICEF/World Bank.
Evaluation of DBE1 District Interventions
55
4. Scope of Work
In order to achieve the above objective, the senior staff researcher will:
a. Coordinate with the DBE1 national team to design an evaluation study to address the
stated questions; the design will involve a case study methodology and field visits to a
limited number of districts for data collection, the use of existing data and interviews
with key actors
b. Place the development and implementation of the tools within the broader intellectual
framework of policy implementation
c. Conduct field work to collect data
d. Conduct data analysis and prepare a final report including lessons learned and
recommendations
e. Participate in workshops and other events to present the findings as appropriate.
Evaluation of DBE1 District Interventions
56
Appendix 2: Good Practice Examples in the Implementation of Tools22
Table 5: Good Practice Case Study 1 – SIMA: Subang District
District location Subang, West Java Type of good practice Implementation of Asset Management: SIMA
Why this example is considered good practice
SIMA is achieving expected outcomes plus business efficiencies. Subang’s activities demonstrate local leadership, active participation, dissemination and local ownership of school management.
Background and description of situation before change initiative
Asset data collection done manually on paper. Very time consuming. Data provided by schools was: limited by less than full school participation, incomplete, and inaccurate data from schools.
Description of good practice
DBE1 partner district Subang, was chosen to participate because DBE identified a need for support in management of assets because of impending audit visit. A demonstration of SIMA stimulated interest and commitment to proceed to implementation. DBE1 conducted training for two teams of 5 Dinas staff for 2 days. Staff had a good knowledge of assets, and possessed the essential skills and motivation to benefit from training. Teams trained district schools in February and March 2011 in SIMA. The results of the training led to outcome as per Table 4.
Does the tool work? Does it help to translate policy into action in Districts?
Yes. Evidence shows it working at school. District and provincial levels.
How outcomes were achieved: the strategies, processes, resources, etc.
Identification of need for support in management of assets because of impending audit visit. Training and support from DBE1. Leadership from Kepala Dinas Pendidikan. Job description of middle-level managers changed to focus on data systems.
Outcomes of practices
Refer to Table 4; major business efficiencies achieved. Dinas and schools now better able to manage their assets. SIMA facilitates data checking and quality improvement. Users are better able to meet external audit requirements.
Are the tools sustainable: what is the level of need for the tool and support services?
Yes, potentially high: Local regulations require schools to participate. School benefits by experiencing better asset management and support. Addresses external audit requirements.
What is the experience with the dissemination, institutionalization and adaptation/evolution of the tool?
Disseminated to all schools in district. Kepala Dinas leadership and support Institutionalized in Dinas through formal job descriptions, training, and employment of qualified computer support staff.
Are there potential service providers available with the capacity to meet that need/demand? Is there an appreciation of the need to work with an external service provider? Do the districts, working with service providers, have the capacity to use the tools effectively?
Need for service provider has not been clearly stated at this stage.
Lessons Learned Timely interaction of local needs for better data and planning, relevant support (DBE1), leadership, appropriate staffing and external stimulus (audit) facilitate change and development.
Source of information Visit by evaluation team, DBE1, 11 August 2011.
22
The framework to describe these good practice examples is adapted for this study of DBE1 tools from Good
Practices for Mainstreaming in Basic Education. Jakarta, UNICEF, 2009, Appendix 5.
Evaluation of DBE1 District Interventions
57
Table 6: Good Practice Case Study 2 – SIMA & SIMPTK: Purworejo District
District location Purworejo, Central Java
Type of good practice Implementation of Asset Management, SIMA and Personnel Management Information System, SIMPTK
Why this example is considered good practice
Demonstration of needs, local leadership, active participation, initiative and ownership of school management issues.
Background and description of situation before change initiative
Asset data collections done manually on paper. Very time consuming. Data was limited, inaccurate and incomplete. Validity of policies and practices questionable.
Description of good practice
Uses the outputs of SIMA and SMPTK together for integrated analysis and decision making about school facilities and human resources planning. SMPTK and SIMA require accurate data. Tools permit checking and verification; schools now more careful and comprehensive in their data collection and inputs to the district.
Does the tool work? Does it achieve help to translate policy into implementation action in Districts?
Yes; helps in formulating local policies and practices.
How outcomes were achieved: the strategies, processes, people, resources, etc.
Purworejo chosen to participate because DBE has been working with this district that has a strong record of collaboratively developing and trialing tools. The District is also seen at Provincial level as a ‘reference’ District and has received additional financial support through World Bank BEC-TF for the expansion of training. Key concepts of strategy that lead to positive outcomes include: leadership by Kepala Dinas; issue of supporting regulations; reliable and culturally
appropriate approach by DBE1 that provided continuous technical support and advice during development and testing; easy and widely used software (Excel); needs-based, collaborative development (not top down); 2-way feedback; a layered-competency approach in Dinas: operators develop
competency > teach head of section > teach head of sub dinas > when understood, explain to Kepala Dinas, Bupati & DPRD. Younger and computer-literate staff well able to see issues in analytical way and provide technical support to Dinas.
Outcomes of practices
Quality of data is higher: data can be checked, verified and corrected Reduces time for Dinas. Unexpected outcome: the interaction between need to achieve Minimum Service Standards and availability of tools has led to data collection from all schools in District so all schools can now be considered together in planning educational provision.
Unexpected outcomes
Teachers are willing to respond the demands of data collection because (a) intrinsic interest in computerized systems and (b) saves time compared to earlier manual systems. Schools have assets, purchased from a variety of sources including donor grants, whose ownership was unclear until now recorded. Now it becomes District property and funds may be used to maintain it. Recorded assets can also be depreciated, written off, and disposed. Until that happens, the asset must be retained for audit even though it may be useless. This explains the piles of broken furniture and materials to be seen in many schools in Indonesia.
Are the tools and the implementation strategies sustainable: what is the level of interest and need/demand for the tool and support services?
The sustainability of the tools is supported by many factors such as:
Sustainability of expertise is important, well-understood and now managed by policy from Bupati to retain staff in their positions in Dinas
to support tools unless to be promoted
a strong need to avoid a ‘disclaimer’ result from audits
satisfaction that comes from doing a job well in-house
strategic building of staff capacity at all levels to ensure continuity – building technical depth in the organization.
No demand for service provider. The only appropriate service-provider nominated was the Provincial Education Office Planning Unit.
Evaluation of DBE1 District Interventions
58
District location Purworejo, Central Java What is the experience with the dissemination, institutionalization and adaptation/evolution of the tool, including the experience of other donors
District has responded to requests for technical support from private schools including madrasah. What can be observed here is that SIMA is helping to
institutionalize the quality of the whole system and build local communities of practice among schools where they can be mutually supportive. Other Districts are seeking help and spending their own money to get this from Purworejo. Other Dinas would like to use tools as well.
Are there potential service providers available with the capacity to meet that need/demand?
The advice from Purworejo was clearly no. Dinas staff advised that it is a challenge for non-Dinas service providers to keep up-to-date with policy developments and also remain familiar with the data management and analysis processes – two tasks that are uniquely carried out in Dinas Pendidikan.
Is there an appreciation of the need to work with an external service provider? Do the districts, working with service providers, have the capacity to use the tools effectively?
The experience of the Dinas Pendidikan with service providers is not positive. They had support from a university that proved unsatisfactory: they used high-level and inappropriate assumptions especially about Districts and schools; designed software that was too complicated; and were not available for help when needed. The World Bank introduced TRIMS but this also proved to be too complicated to be of practical use. Moreover, on-going and accessible support of the kind readily available from DBE1 was not available at all. This district has demonstrated its own capacity to use tools well and the capacity to disseminate their knowledge and skills to others.
Threats
The Purworejo District Asset, Finance Management and Revenue Office is a powerful organization that sees SIMA to be in competition with SIMDA, whereas (correctly), it should be seen as a complementary tool. The Head of this Office believes one system should cover all. This illustrates the complex and competitive world in which IT solutions may fail.
Lessons Learned Excellent support (DBE1), leadership in Dinas, appropriate staffing, external audit facilitate change and development.
Source of information Visit by evaluation team from DBE1, 18 August 2011.
Evaluation of DBE1 District Interventions
59
Appendix 3: Alternative Policies and Costs
Policy: Minimum Service Standard No.7: Every elementary school should have two teachers
with the degree of Bachelor. (S1).
Example: Tuban District in East Java has 50 schools that require two S1 teachers and 163
schools that need one S1 teacher, or a total of 263 persons to meet the Minimum Service
Standard No.7.
Activity Type of Cost Volume Unit Cost
(IDR)
Total Cost 5 years (IDR)
Alternative I: Recruiting New Teachers
Recruiting New Teachers
Administration 263 0 0
Cost of New Teacher Salaries
Operational 263 Rp. 25 million/year Rp. 6.6 billion
Alternative II: Improving Qualifications of Teachers to S1 Degree
Improved Qualifications
Investment 263 x 2 years Rp. 3.5 million/year Rp. 1.85 billion
Alternative III: Redistribution Teachers With S1 Degree
Teachers Assignments
Administration 263 Rp. 1million Rp. 263 million
This example has been provided by DBE1.
When accurate data about teacher distribution is available, valid policy alternatives can be
considered. In this case, the policy alternatives indicate major cost differences to achieve the
same outcomes. See discussion in Section 3.2: Do the Tools Work?
Evaluation of DBE1 District Interventions
60
Appendix 4: List of Institutions and Resource Persons Met
DBE1
No Name Position 1 Dan Moulton Chief of Party
2 Basilius Bengoteku Deputy Chief of Party
3 Frank Hijmans Planning & Management Advisor
4 Mark Heyward Replication & Capacity Building Advisor
5 Sandra Triatmoko Budget & Finance Specialist
6 Ingga Danta Vistara Senior Data & Information Technology Specialist
7 Tita M. Rachmaniah ICT/EMIS Specialist
8 Dirga Prahadi EMIS/ICT Assistant
9 Rudi Sopiana Junior Data Information Specialist
10 Indra Gunawan Education Management Community Participation Specialist
11 Erna Irnawati Provincial Coordinator, West Java
12 Supriono Subakir Provincial Coordinator, East Java
13 Doly Hutapea Provincial Coordinator, North Sumatera
14 T. Meldi Kesuma CO Specialist, Office Manager, Acting PC NAD
15 Rimbananto BF Specialist. NAD
16 Muhibbudin Ed. Management Planning Specialist, NAD
17 Rusnaini Regional Coordinator Aceh Tengah & Utara, NAD
18 M. Khadafi Data & Information Assistant, NAD
Evaluation of DBE1 District Interventions
61
West Java
No Name Position West Java Province Education Office
1 WahyudinZarkasy Province Education Office Head
2 Pakpahan Infrastructure and Equipment Section Head
3 Diding G General Affairs and Human Resources Section Head
4 Drs. ArthaSetiawan, M.Si Infrastructure and Equipment Staff
5 SuparmanNatakarna General Affairs Staff
6 Hikmaturrohman General Affairs Staff
7 ZainalArifin General Affairs Staff
8 Wawan Sofwan General Affairs Staff
9 Haris Sirojudin General Affairs Staff
10 Gun Gunawan Staff
District Education Office – Kabupaten Sukabumi
1 Dede Daniel Secondary Education Division Head
2 Iik Nurulpaik Universitas Pendidikan Indonesia/UPI [Service Provider]
District Education Office – Kabupaten Subang
1 Dra. Titin Rosyatin DEO Vice Head
2 Eva Dahlia General Affairs Sub-Division Head
3 Gin Gin Ginanjar Rahayu General Affairs Staff
4 Hasta Satya Darma General Affairs Staff
District Education Office – Kabupaten Karawang
1 Waskita DEO Head
2 Tri Warakanti Head of Program Sub-Division
3 Iin Rachmawati District Coordinator
4 Dadang SIPPK Operator
District Education Office – Kota Bogor
1 Fetty Kondarsah DEO Head
2 Ritta Tresnayanti Head of Planning Sub-Division
Evaluation of DBE1 District Interventions
62
Central Java
No Name Position District Education Office – Kabupaten Purworejo
1 Bambang Aryawan DEO Head
2 Aman JokoWaspodo Head of Planning Sub-Division
3 F. Widhi Dewanto Planning Sub-Division Staff
4 BambangSurtianto Staff TKNK
5 Adi Condro Purwanto Planning Sub-Division Staff
6 Dwi Santosa General Divison Staff
7 Wahyu Eko Prihantoro General Division Staff
District Asset, Finance Management, and Revenue Office – Kabupaten Purworejo
1 Achmad Kurniawan K. Head of District Office
2 Amin Fadillah Staff
3 Setiyo Budi Wahyono Staff
Evaluation of DBE1 District Interventions
63
East Java
No Name Position District Education Office – Kabupaten Sampang
1 Heri Purnomo DEO Head
2 Mujali Head of Program Sub-Division
3 Emmi BOSP Operator
District Education Office – Kota Mojokerto
1 Muchtar Bagus Haryanto DEO Vice Head
2 Lely Head of Program Sub-Division
3 Suprapti Staff, Secondary Education Division
4 Sutjipto School Supervisor
5 Siti Ulfah School Supervisor
6 Astrid Staff, Basic Education Division
7 Paryono District Facilitator
8 Endang District Facilitator
District Education Office – Kabupaten Tuban
1 Endang Trimeidiya DEO Vice Head
2 Wignyo Hadi Head of Program Sub-Division
3 Retno Staff, Secondary Education Division
District Education Office – Kabupaten Gresik
1 Suwono Head of Program Sub-Division
2 Ratna Staff, Program Sub-Division
Evaluation of DBE1 District Interventions
64
North Sumatera
No Name Position District Education Office – Kota Tebing Tinggi
1 Drs. H. Pardamean Siregar, M.AP DEO Head
2 Zahidin, S.Pd, M.Pd DEO Vice Head
3 Darajat, S.Pd, M.Pd Head of Policy and Education Finance Division
4 Drs. Joner Sitinjak Head of Basic and Secondary Education Division
5 Nelce Kaloeti, S.Sos, M.Pd Head of Teacher Section, Quality Improvement of Teachers and Education Staff
6 Mhd. Dedi Lubis, S.Pd Operator SIM-NUPTK, SIMP-K, SIM-PTK & PBPSAP)
7 Rainbow Pratama Sitohang Operator SIM-NUPTK, SIMP-K, SIM-PTK & PBPSAP)
District Education Office – Kota Tanjung Balai
1 Drs. H. Hamlet Sinambela, M.Pd DEO Head
2 Harmeini, SH DEO Vice Head
3 Tety Juliany Siregar, ST, MT Head of Facilities and Infrastructure Division
4 Azhar, S.Pd Head of Data Collection and Processing Section, Program Division
5 Imron Butarbutar, S.Pd District Facilitator DBE1
6 Dewi Murni Operator DPISS/SIPPK and PBPSAP DBE1;
7 Julia Hafni Syambas Operator SIM-NUPTK, PBPSAP
Evaluation of DBE1 District Interventions
65
Nangroe Aceh Darussalam
No Name Position District Education Office – Kabupaten Aceh Besar
1 Fadlan DEO Head
2 Hayatun Nufus School Supervisor, District Facilitator
3 Amiruddin School Supervisor, District Facilitator
4 Rusli Principal of SD Haloum, District Facilitator
District Education Office – Kota Banda Aceh
1 Saridin Head of Program Division
2 Muslim Head of Section. Operator
3 Darmawati Staff, Monitoring
SEDIA - AusAID
1 Andrew Duncan Team Leader
2 Krishnayani Prawira Winata Deputy Team Leader/School Improvement Adviser
3 Anas M. Adam System Strengthening Adviser
4 Jony Chandra Education and Management Planning Specialist
5 Chairul Muslim Strategic Planning Adviser & Coordinator of SEDIA District Capacity Building
UNICEF
1 Jean-Pierre Paratore Youth & Adolescent Dev. Specialist
2 Umardi Program Officer
Evaluation of DBE1 District Interventions
66
Schools Visited
No Name Position SLBN Citeureup, Cimahi
1 Tito S Suharwanto Principal
2 Dimyati Inventory Staff
SDN Bendosari, Purworejo
1 Busroni Principal
2 Kusmono Teacher
SDN Pelutan, Purworejo
1 Suliyah Principal
2 Solikin Teacher
SDN Pakem, Purworejo
1 Ulis Indarti Principal
SD Negeri Manasuka Subang
1 Atin Surelawatin, S.Pd., MM.Pd. Principal
2 Rina Mugiatin SIMA Operator
SD Negeri Rosela Indah Subang
1 Nining Mulyati, S.Pd. Principal
2 Suharya, S.Pd. SIMA Operator
3 Susi Marlia, S.H. SIMA Operator
SMP Negeri 1 Subang
1 E. Heni Rodiah, S.Pd., MM.Pd. Principal
2 Asep Sugandi SIMA Operator
SMA Negeri 1 Subang
1 Dra. Hj. Yulia Gantini, M.Pd. Principal
2 Karja Nugraha Vice Principal for Infrastructure and Equipment