final evaluation of pep and operationalisation of pep ii · final evaluation of pep and...
TRANSCRIPT
0
Framework Contract Beneficiaries 2013 Lot 1
European Union Delegation to Zambia
FINAL EVALUATION OF PEP AND OPERATIONALISATION OF PEP II
Contract N° 2015/371442/2
FINAL EVALUATION REPORT
& STRATEGIC RECOMMENDATIONS FOR PEP II
JULY 2016
This project is funded by the European Union Project implemented by
The contents of this publication is the
sole responsibility of AESA Consortium and
can in no way be taken to reflect the views
AESA Consortium
Page | 1
of the European Union
EUROPEAN UNION DELEGATION TO ZAMBIA
Final Evaluation of PEP and Operationalisation of PEP II
Contract N° 2015/371442/2
Framework Contract Beneficiaries 2013 – LOT 1
FINAL REPORT
July 2016
Team Composition: Team Leader: Michael CONNOLLY
KE 2: Benjamine HANYANI-MLAMBO
This report was prepared with financial assistance from the Commission of the European Communities. The views
expressed are those of the consultant and do not necessarily represent any official view of the Commission or the
Government of this country.
Page | 2
Table of Contents
Summary ....................................................................................................................................................... 4
1. Introduction .............................................................................................................................................. 7
1.1 Background and Context ..................................................................................................................... 7
1.2 EU Support to Strengthen the Agricultural Sector and MAL Performance ........................................ 8
1.3. Objectives of the Evaluation .............................................................................................................. 8
1.4. Approach and Methodology .............................................................................................................. 9
2. Evaluation Questions Answered ............................................................................................................. 11
2.1 Relevance .......................................................................................................................................... 11
2.2 Effectiveness ..................................................................................................................................... 13
2.3 Efficiency ........................................................................................................................................... 14
2.4 Impact ............................................................................................................................................... 15
2.5 Sustainability ..................................................................................................................................... 16
3. Overall Assessment of PEP ...................................................................................................................... 17
3.1. Programme Context ......................................................................................................................... 17
3.2. Strengths and achievements............................................................................................................ 17
3.3. Weaknesses and challenges ............................................................................................................. 18
3.4. Overall insights ................................................................................................................................. 18
4. Programme Plans, Targets and Performance Outcomes ........................................................................ 20
5. Findings, Conclusions and Recommendations ........................................................................................ 27
6. Programme Oversight and Management .............................................................................................. 33
7. Financing and Budget ............................................................................................................................. 35
8. Cross-cutting issues: HIV/AIDS and Gender ............................................................................................ 37
9. Strategic Recommendations for the Operationalisation of PEP II .......................................................... 38
9.1 Building on the Lessons from PEP 1 .................................................................................................. 38
9.2 Facilitate, communicate and implement a Change Management Strategy for MOA & MoFL ...... 38
9.3. Strengthening Extension Services Systems in Collaboration with Research (Action Fiche 2.1 & 2.3).
................................................................................................................................................................ 41
9.4. Learning and Innovation Platforms for Research and Extension ..................................................... 43
9.6. Focal Districts for PEP II.................................................................................................................... 45
9.7 Knowledge Management in Ministries ............................................................................................. 46
Page | 3
Annex 1: Detailed Evaluation Method, Limitations and Tools .................................................................... 48
Annex 2: List of Persons/Organizations Consulted ..................................................................................... 58
Annex 3: Literature & Documentation Reviewed ....................................................................................... 63
Page | 4
Summary
1. Programme Context
As a programme, PEP was a response to the widely held view by national stakeholders
around 2009 that the agricultural sector was struggling to achieve its undoubted potential
to make a vital contribution to economic growth, rural development, and poverty reduction
in the country. At the centre of those concerns and challenges was the then MOAC – the
ministry expected to lead sector initiatives and development programmes. It had
capacities that were known to be weak and it needed substantial revitalization of its
policies, structures, personnel, technologies, and extension services systems.
The EU, with the strong support of Cooperating Partners (SIDA, JICA, IFAD & Finnish
Cooperation), engaged the ministry and it was agreed that a comprehensive assessment of
the Ministry’s capacities be conducted. Following from that exercise, the Performance
Enhancement Programme (PEP) was conceived and designed. It was not envisioned as yet
another project, but as a programme with the ambition, and indeed, courage to take on the
wide and deep system-wide capacity challenges facing the Ministry.
PEP eventually started in 2012 with four very overloaded key result areas across the
ministry covering change management; sector policy, planning and financial management;
human resources and ICT; and monitoring and evaluation. The goal was to tackle all the
major areas of capacity weakness together, with little prioritization and over a relatively
short time frame. This ultimately proved to be the most fundamental weakness of PEP –
trying to do too many things in too short a time – a commendable array of activities were
started but many were never properly or fully completed, while some unplanned activities
were added for implementation at the behest of MAL.
2. Strengths and achievements
PEP initiatives in management and staff training, providing computers, developing data
bases, strengthening planning, monitoring and evaluation skills, and updating the curricula
of training institutes were all well appreciated by ministry staff. This was partly because of
the extent of the previous protracted neglect of those areas, coupled with the hope and
impetus that those initiatives brought to the ministry staff which benefited. Given the
absence of modern support facilities and systems for middle-tier professional and support
staff in the ministry, the programme support to improve ICT workplace technology,
training, and the development data bases and information systems made most MAL staff
familiar and proficient with computer and internet use, knowledge of data bases, and
reporting in their day-to-day work.
Page | 5
3. Weaknesses and challenges
A central expectation of stakeholders and a core purpose of PEP was to facilitate and
manage change actively across and within MAL. To begin to fulfill this role satisfactorily,
the programme needed to have and communicate widely a comprehensive change
management strategy spearheaded by champions of change from the senior levels of the
ministry. The programme did not have such a strategy, including gaining and sustaining a
strong and evidence-based commitment from senior management. The Change
Management Team (CMT) operated in the prevailing administrative mode of MAL, as a
Head Office based coordination unit, rather than leading a dynamic system-wide change
process across all levels of the ministry. The programme was not communicated
adequately or cascaded effectively to the provincial or district levels as the feedback and
records on information, training and evaluations at those levels clearly reveal. Both those
shortcomings will need to be addressed in the next phase.
The programme should have conducted a benchmarking exercise, such as a ministry-wide
staff survey, at the outset. For a performance improvement programme centered on staff
proficiencies, such a comprehensive exercise should have been mandatory so as to enable
meaningful assessments of progress in skills acquisition by staff over the course of the
programme. The generally disenabling conditions in the ministries mitigated against
optimal performance improvement (e.g. limited and late budget allocations and lack of
opportunities to put skills gained from training into practice in their work places). Periodic
appraisals on the constraining effects of those latter factors will be necessary for inclusion
in the next phase. There were also instances of wastage of resources, despite the prevailing
budget shortfalls in MAL. PEP funded an exercise to devise improved budget resource
allocation criteria for MAL which the ministry subsequently decided not to accept or utilize,
casting some aspersions on programme decision-making processes and the quality of
engagement and ownership of PEP by MAL.
4. Conclusion
Overall, the PEP experience was a learning journey for the EU and MAL that revealed many
instructive lessons for the design, planning, facilitation and implementation of such
programmes in future. A central one is that a programme should be implemented in a well-
defined programme mode – not as a collection of projects. Another is that the result areas
should be limited and not overloaded – the so called “doability” mantra. While PEP brought
benefits across all levels (HO, provinces and districts) of MAL, the exclusion of large
numbers of staff (half of the frontline field extension complement) from the allocations of
computer tablets led to disappointment, and perceptions of favouritism and exclusion, by
half of a very important cadre of MAL staff. Change management programmes and their
strategies need to challenge existing norms and modes and bring about demonstrable
Page | 6
change in the way public service ministries perform. To do that, a change strategy and
process was needed together with a way of knowing or gauging the point of departure and
then how well it is progressing and achieving. PEP II needs to do just that and avert the
pitfalls of the first phase.
In a nutshell, PEP certainly sought to achieve noble and worthy deeds. There is no doubt
that PEP set out to do the right thing and had some very worthy accomplishments. It
facilitated increased awareness, knowledge and insights among many staff on management
and performance issues in MAL. Its major drawback was that it attempted to do some of
those “right things” in the wrong way. Examples included embarking on a staff
performance improvement programme without conducting a comprehensive baseline
study on existing proficiencies; seeking to improve MAL budget allocation criteria without
first securing purposeful and definite commitments by the relevant GoZ ministries;
providing wide-ranging support to improving ICT capacity across MAL while being unable
to secure the commitment of senior management to put in place a dedicated unit to
optimize the management and benefits of the investment.
Page | 7
1. Introduction
1.1 Background and Context
The Government of Zambia’s development priorities are framed by the vision of “a
prosperous middle-income nation by 2030” that “provides opportunities for improving the
well-being of all”. The revised Sixth National Development Plan (R-SNDP) 2013-2016 aims
“to accelerate growth further and to make it more relevant to improving the livelihood of
the Zambian people, especially in rural areas”.
Given that about 80% of the population depends on agriculture as their main source of
livelihood, the agricultural sector has the greatest potential to significantly impact poverty
reduction and to achieve increased economic diversification. This policy shift to refocus the
agricultural sector’s development strategies and programmes towards drivers of pro-poor
agricultural growth is at the core of the draft National Agriculture Policy (NAP) and the
National Agriculture Investment Plan 2014-2018 under the Comprehensive African
Agriculture Development Programme.
The NAP and the Agricultural Sector Chapter of the R-SNDP acknowledge that to achieve
this goal the Government must focus on providing public goods that are indispensable to
creating broad-based and inclusive agricultural growth. These include agricultural
research, effective agricultural extension services, control of pests and diseases of national
importance, and the development of basic rural infrastructure. Vision 2030 and the NAP
also recognise as key the "gender responsive sustainable development" of the agriculture
sector with an emphasis on women’s empowerment.
Beyond the sector policy framework, there is still a gap between stated policy intentions
and their realisation. Although budgetary allocations to the then Ministry of Agriculture
and Livestock have increased during the last seven years, they remain below the
recommended CAADP 2 minimum of 10% of the annual national budget and expenditure
within that remains overwhelmingly driven by the Government’s maize purchases through
the Food Reserve Agency (FRA) and input subsidies under the Farmer Input Support
Programme (FISP). The maize input subsidy programmes consumed 52% of the 2014
agriculture budget. Emphasis on the two programs has diverted resources from critical
programs with potential to contribute to the desired growth in the agricultural sector such
as irrigation development, extension services delivery and research while limiting the
scope for optimal private sector participation in national agricultural development.
Page | 8
1.2 EU Support to Strengthen the Agricultural Sector and MAL Performance
The underperformance of the agricultural sector is also linked to institutional weaknesses
and organizational constraints that prevent the Ministry of Agriculture and Livestock
(MAL) from delivering on its core functions such as extension and research, directly
affecting farmers' productivity and undermining the potential role of the private sector.
The EU provides consistent support to improve the capacity of the Ministry of Agriculture
and to the Ministry of Fisheries and Livestock to deliver quality services to smallholder
farmers towards increased productivity and enhanced diversity of Zambia's agricultural
sector.
Against this background, the EU in 2009, supported the then Ministry of Agriculture and
Cooperatives (MACO) to carry out a Participatory Review of the Ministry's Performance
that further fed into an in-depth assessment of MACO's capacity development needs and
challenges. The outcome of this exercise served as the basis for the design of the "Support
to Agriculture Sector Performance Enhancement Programme" (known as PEP)
implemented through four subsequent programme estimates under the 10th European
Development Fund (EDF). The project, worth €8.9M, aimed at ensuring that "the
agricultural sector contributes to growth and poverty reduction in Zambia" (Overall
Objective) by "strengthening the capacity of the Ministry of Agriculture and Livestock
(MAL) in an effective and sustainable way so that their operations are improved" (Project
Purpose). The project focused on four results areas namely 1) Change management and
service delivery, 2) Sector policy, planning and financial management, 3) Human resources
management and Information and Communication Technology, and 4) Monitoring and
Evaluation. Following a four-month closure phase PEP was completed in April 2016.
1.3. Objectives of the Evaluation
The assignment had two objectives and involved two separate, although interlinked
phases, consisting of 1) the final evaluation of PEP; and 2) the development of
recommendations for PEP II. The aim was to conduct an independent assessment of the
performance and impact of the "Support to the Agricultural Sector Performance
Enhancement Programme" (PEP), paying particular attention to the project's achievements
against its expected results and purpose. This involved an in-depth assessment of the
strategic objectives and implementation mechanisms of PEP as well as of the extent to
which these have been effective in delivering the desired results.
Page | 9
1.4. Approach and Methodology
Guided by the ToRs, and the Inception Report approved for the evaluation, the focus of the
methodology for this evaluation was on the completion of two separate, but closely linked,
assignments:
1) The final evaluation of PEP; and, 2) The development of recommendations for PEP II.
The central focus was on the assessment of PEP achievements under the national
prevailing economic and institutional realities; key indicators and modalities to achieve
result-oriented approaches; and core institutional capabilities and commitments required
to underpin improved performance by key sector actors; and the quality of outcomes,
results and lessons from the focal interventions.
This evaluation sought to deepen understanding of the cause and effect linkages between
activities and results, including the Theory of Change (ToC) as a key learning platform for
the European Union and its partners on agricultural sector performance enhancement and
capacity development programmes to understand not only (1) what works and what does
not, and (2) critically why, and under what circumstances. While the ToC per se was not
formally highlighted and elaborated in programme planning documents and reports, the
evaluation team (as per the Inception Report) used a ToC lens in assessing the programme
processes to achieve overall outcomes and impact.
In addition to the normative evaluation methodology for projects, centered on the purpose,
indicators and outputs of the logical framework, the methodology focused on key learning
outcome areas and operational attributes reflecting staff performance improvement in
MOA & MFL This focused on knowledge, skills and insights acquired by management and
staff on individual and collective performance during the course of the programme.
Examples of such Performance Enhancement Insights, as perceived and related directly by
Ministry staff, are described in the boxes included later in this report.
This cross cutting and multi-intervention evaluation was conducted in accordance with the
guidelines for the OECD DAC Evaluation Criteria on which the evaluation questions were
based and agreed with the Reference Group.
Those evaluation questions were formulated in the ToR to provide qualitative insights from
the overall programme experiences to inform future programme design, implementation
modalities, management, operations, and funding. The answers to those questions focus on
the overall programme performance outcomes in strengthening capacity in MAL, including
overarching lessons arising for future phases. Under the guidance of the Reference Group,
the team focused its work on evaluating the four key output areas as delineated in the
Page | 10
ToRs. Based on guidance & advice from the Reference Group, an itinerary of interviews,
meetings and discussions (see Annexes) were held with (1) key personnel involved in PEP
oversight, management, and implementation including relevant GRZ ministries (e.g.
Decentralization Directorate in the Cabinet Office), and (2) local institutions or
organizations (public sector/NGOs/Farmer Organizations) involved in PEP capacity
development activities, and (3) cooperating partners including IFAD & JICA.
Using a progress and performance matrix grid, an appraisal of all PEP key result areas
(targets, indicators & outputs ) as per the PEP Log Frame, was initially conducted, including
status of performance reviews/or evaluations, analyses of outcomes and results against
targets, and findings determined on PEP implementation. The four key result areas for PEP
were assessed comprehensively under the above matrix grid.
This was combined with more areas (also using a systematic matrix approach) to examine
the log frame, targets, indicators, achievements, and outcomes. The key lessons arising, and
recommendations covering institutional capacity development processes, including the
appropriateness of change management and organization development (OD) approaches
implemented by PEP were also assessed. The follow-through actions on the Mid-Term
Evaluation Report, (September 2014) were assessed in the context of the subsequent
implementation and completion of the programme. The overall programme was assessed
for consistency with the EU country strategy and complementarity with other donor
programmes. Two EU specific criteria (EU added value in design and implementation &
coherence with the EU country strategy, member states and other donors) were also
assessed.
The Consolidated FCRs (Findings, Conclusions and Recommendations) Matrix across all
outputs and activities was completed in the synthesis phase of the evaluation. The
evaluation questions were then addressed and answered. The priorities for
operationalizing PEP II were also identified, based on the lessons from the evaluation of
PEP I and both were presented on May 17th for validation to a stakeholder’s workshop in
close consultation with the Reference Group.
Page | 11
2. Evaluation Questions Answered
2.1 Relevance
To what extent was the project design appropriate for the then MAL's institutional capacity development needs? The parent MAL had remedial capacity needs in critical functional areas, as documented in needs assessment reports of 2009 and 2013. As such, the PEP programme design was appropriate to the overall capacity needs of the ministry. This is because the PEP Programme was designed to address the identified areas were capacity was lacking, such as, in policy analysis and decision-making processes; services provision; budgeting, planning and financial management; human resources management; and in knowledge management and Information and Communication Technology (ICT). As an illustrative example, while many MAL personnel were very highly qualified technically, leadership and management capabilities were lacking. This strongly justified training and capacity development on leadership, management and administration. A number of strategic documents viz: Vision 2030, the Revised Sixth National Development Plan (R-SNDP) 2013 – 201, the National Agricultural Policy (NAP), and the NAIP 2014 – 2018 also emphasize the importance of providing know-how and support to the development of the key ministry (MAL) and the wider agricultural sector. Have these needs changed since the mid-point of the project? No significant changes occurred within the programme context during implementation. Available development data, national indicators and web-based insights by development partners such as the African Development Bank and the World Bank show that there have been inconsequential variations in socio-economic indicators from project inception in 2012 till project completion in 2016. Despite efforts by the TA Team, Change Management Teams and implementation teams within MAL to continuously re-strategize, the experiences in the implementation of various activities did not seem to have evident influence/impact on prioritization and re-planning of activities on the programme. In other words, the programme continued throughout its implementation period to try to do too many activities, despite evidence that that it was struggling to achieve its targets. What amendments in the concept, theory or implementation modality should be made for a future phase of the project? The programme lacked a definite overall change management strategy, apart from the implementation of activities under programme estimates across the key result areas. PEP started in project mode with a number of activities but with no overall programme orientation and/or strategy. There is, therefore, need to understand Programme-Based Approaches (PBAs) to capacity development e.g. the need for a mindset shift from a project to programme mode with appropriate modalities. A future phase of the programme should also be more focused on fewer result areas; have a more robust communication strategy
Page | 12
between and within MOA & MoFL and with sector stakeholders; and have programme estimates with more coherent and integrated activities that facilitate more process-based approaches to implementation involving monitoring, learning, and evaluation (MLE). Were the project institutional arrangements and mechanisms relevant and have they contributed to increased ownership by the then MAL? The programme’s institutional arrangements and mechanisms were also ostensibly relevant. The institutionalization of the PEP programme, with the programme being housed and embedded in MAL rather than as a stand-alone and/or semi-autonomous endeavor, together with PEP programme interventions. put the programme in the best position to directly influence management and administration processes within MAL. However, these were largely functioning in administrative mode, with no evidence of commitment and active engagement with the processes of capacity development and ownership. As such, the programme did not contribute to increased ownership in any substantive way. The programme struggled with ownership issues, with evidence of limited commitment, participation and quality of decision-making by the programme steering committees and change management teams. A major factor diluting the quality of implementation throughout the programme cycle was that most of the implementation responsibilities and tasks, pertaining to those key senior roles, were delegated to lower ranking personnel in MAL departments whose authority levels, influence and proficiencies were limited. In some cases, this led to critical delays in the finalization of key processes e.g. the MAL Strategic Plan that remained unapproved up until the completion of PEP in April 2016. Another major mitigating challenge was that the programme tried to do too many activities concurrently and was trying to accomplish too much. The programme then quickly evolved into a set of often disconnected projects rather than a coherent programme. As discussed later, this reduced programme effectiveness, efficiency and overall impact. What changes should be considered for future interventions? For the future, similar interventions should be programme rather than project-based, hinged on an understanding of Programme-Based Approaches (PBAs) to capacity development. A future phase of the programme should also be more focused on much fewer activities across less result areas. There is also need for a delineated and phased approach to programme implementation (with orientation/inception, implementation phases, and an exit phase) and a very clear and comprehensive change management strategy (including a robust communication strategy given the limited vertical and horizontal communication between and within MoA and MoFL departments). A more focused approach, with fewer and less loaded result/output areas, will improve on programme relevance. Programmes such as PEP should also avoid making substantial ad hoc contributions to recurrent budget expenditure (e.g. E-voucher programme). Similar performance enhancement programme initiatives in Ghana and Rwanda have improved performance and service delivery, notably since these initiatives were integrated with policies and strategies for decentralization; lessons should be learned from those country
Page | 13
experiences in view of the impending decentralization in Zambia. Within the country, the MoA and MoFL can draw lessons in service delivery from the Ministry of Health.
2.2 Effectiveness
To what extent have the Ministry's capacity and performance been enhanced? The PEP programme enhanced the capacities of individual management, staff and units within MoA and MFL. This was largely through leadership training, induction and in-service training, M&E training, the development and establishment of an HR database, and the distribution of ICT equipment. However, the prevailing conditions in the ministries (e.g. staffing levels & field mobility, conditions of service, inadequate operational budgets), mitigated against optimal performance improvement. Key setbacks included the limited budget allocations and the frequent lack of opportunities to put skills gained from training into practice in staff work places. Has flexibility contributed to the achievement of the purpose or rather contributed to hindering it? The E-voucher system introduced innovation to farmer registration and FISP. On the other hand, emoluments payments enabled the ministries to fill positions and free up accommodation for new recruits. However, programme flexibility made the PEP programme a utility fund for shortfalls in government budgets which is neither desirable nor sustainable. In other words, programme flexibility has hindered rather than contributed to the achievement of the programme purpose. As an example, resources used for e-voucher pilots and emoluments payments (typical government recurrent expenditure) could have been used to complete planned and bona fide PEP commitments e.g. ICT equipment such as tablets made available to all district field staff, M&E training, and curriculum development. While recognizing the foregoing, it is also appreciated and acknowledged that the provision of some of this support was sometimes unavoidable given that this is a programme that was implemented within a set context i.e. the MoA and MoFL work in an environment where flexibility was imperative to ensure that critical deficit areas were addressed in the wider overall interest of progress by both MAL, PEP and the EU Country Strategy. The PEP programme had also previously supported the pilot electronic farmer registration process, which made support to the E-voucher system a logical follow-up programme to assure optimal benefits by MAL and recipient farmers from the initial investment. Were the project's design and approaches conducive to increased effectiveness? While the programme design facilitated increased effectiveness, the approaches adopted mitigated against effectiveness in some respects. Examples include the implementation of too many activities concurrently; the lack of communication on MTE recommendations; and the fact that most of the training conducted was not thoroughly evaluated for skills acquisition and application by trainees. Examples of cases where the project design improved effectiveness include the establishment of the Human Resource Information System (HRIS). This has gone a long way in providing timely and reliable information on and about the HR situation in the entire ministry. Similarly, the reinstatement of In-Service
Page | 14
and Induction Training Programmes has brought a sense of purpose to the Human Resource Development function in the ministries. However, there is no evidence yet of mainstream capacity and performance enhancement throughout the ministries. To what extent have the policies and guidelines developed on HIV/AIDS and gender been implemented and/or mainstreamed (as recommended by the MTR)? One of the programme’s milestones was the development of HIV/AIDS and gender policies and guidelines. Training conducted on the HIV/AIDS and gender policies and guidelines raised awareness at head office and provincial level and in the training institutes. However, more needs to be done at all levels of staffing. Furthermore, based on information in both the MTE and Final PEP Reports, there is no evidence yet of substantive implementation and/or mainstreaming of the policies and guidelines across the ministries.
2.3 Efficiency
Have the project implementation arrangements been efficient? Efficiency assessments by the evaluation team found that programme implementation arrangements have been unsatisfactory. This is because most of the planned activities were implemented late into the programme (leadership and management training, M&E training, procurement of ICT and other support equipment). In addition, some planned programme activities were not completed within the timeframe of the programme (e.g. M&E training and ICT training). Facilities (e.g. the ministry’s HR and M&E data bases, and website) were developed but were either not used at all, or under-utilized. As elaborated elsewhere, the programme tried to do too much with the consequent partial completion of some activities, while also concurrently implementing a lot of unplanned activities. Such approaches often propel ostensible efficiency in spending of programme estimate funds but raise searching questions about the planning and prudence of such spending in achieving desired core results and overall impacts. Budget figures (NAO/PEP Final Report) indicate a range of 81 – 100% expenditure under the four key result areas, which appears relatively efficient, but needs to be considered in conjunction with the overall findings on effectiveness and impact later. How should they be replicated or amended under PEP II? The programme estimates need to be better informed and planned in terms of budget allocations, activities, sequencing and monitoring. The introduction of incentives and awards should be a priority as a strategy for improving facility utilization and overall efficiency e.g. incentives for colleagues to make inputs for the MoA and MFL website. To what extent partnerships with other cooperating partners have contributed to the project performance? The collaboration with CPs enhanced programme performance (e.g. the Extension Strategy with JICA, the M&E Manual with IFAD, and the IFMIS with the Finnish Cooperation).
Page | 15
To what extent have lessons and recommendations from the MTE been implemented and/or followed up? The recommendations of the Mid-Term Evaluation (MTE) on the log frame were followed and contributed to its adjustment and updating (December 2014). However, the revised log frame was not officially adopted through an official rider at EU level and may not have brought about the proposed improvements set out in the MTE. In addition, there was no evidence that the proposals of the MTE for a more comprehensive change management process by the programme were engaged with. The MTE recommendations were also not shared, or communicated to, the provincial and district levels.
2.4 Impact
What impact (intended, unintended, positive or negative, direct or indirect) has the project had on the MAL's capacity and performance? Some individual beneficiaries of change management efforts (e.g. leadership and management training) exhibited a change of attitude and mindset shift towards performance issues (e.g. DACOs becoming involved in mentoring and coaching of their staff). According to ZARI reports, distributed ICT equipment also improved the timeliness, quality and consistency in reporting. As an example, the distributed ICT equipment facilitated the collection and utilization of real time data using the CABI Plant Wise which created plant disease clinics in farming communities run by camp extension officers. The payments of retirement packages indirectly contributed to employment creation. Worth noting is the fact that the direct impact of the PEP programme is largely restricted to individuals given its focus as a performance enhancement programme. impact assessments were also restricted to the activity/output levels. Impact at outcome level was difficult to assess due to the limited timeframe of the PEP programme. Performance enhancement is also a long-term process so the PEP programme should not be expected to have achieved much impact at the outcome level. On the other hand, some opportunities for learning and development that were presented to the ministry were not utilized (e.g. failure by a technical working group to develop proposals for a study tour to Brazil). Which of these effects have been observed or experienced by Zambia's small-scale male and female farmers? This evaluation also notes that the programme was primarily not designed to have impact at farmer level given that there are no direct targets or indicators at farmer level. The timeframe was also too limited for the programme to achieve any trickle down impact at farmer level. However, the programme was designed to enhance performance of ministry staff so that they would improve service delivery to farmers and improve their output in the long run. As already alluded, it was not possible to see major changes in the actual performance of staff during the tenure of the programme as attitudes and work culture change take time to happen. On the other hand, impact assessments based on the “with” and “without” project scenarios, show that access to ICT extension aids and service delivery improved much better as opposed to a scenario without the project. The PEP programme also made a difference overall within the ministry and with regards to service provision by
Page | 16
both the ministry departments and their sister institutions. These improvements would not have occurred without PEP programme support. What could have been done differently for the project to achieve wider impact? To achieve wider programme impact there was need for consolidation and strengthening of induction training. This is because the current approach is focused strongly on personnel and administrative procedures but remains weak on technical/subject matter content, There is also need for consolidation of staff skills beyond introductory training/ information giving (e.g. mentoring and coaching). This being a performance programme, there should also have been much greater attention paid to performance assessment across the ministry As an illustrational example, APAS was still as irrelevant at the end of the programme as it was at the beginning and no effective measures were taken to change that. Again, these are issues the follow-up phase (PEP II) could draw lessons from and improve upon.
2.5 Sustainability
To what extent has the support provided generated effects that can economically and institutionally be sustainable? The leadership and management training transformed the capacities of individual staff which can be cascaded down to the peers and subordinate staff (multiplier effect). Other examples of sustainability included an improvement in the capacity of HR staff and PPD staff (admittedly more at HQ than in the provinces and districts), an improvement in ICT skills at many levels, and improved efficiency as a result of the introduction of the HRIS. Beyond those accomplishments, the programme became involved in a lot of activities that should rightly have been undertaken and funded from the ministries’ recurrent budgets, and therefore objectively unsustainable (e.g. emoluments, induction and in-service training, and ICT equipment). In addition, unless there was a follow-up phase, as is the case, the chances for sustainability of a number of those initiatives were low. It is, therefore, paramount that the follow-up phase of the PEP programme should look at ways of ensuring the sustainability of key programme initiatives. Examples include the need for building upon the lessons from PEP I (a change management strategy, refocus attention from addressing the hardware issues to the software issues in agricultural extension, and use of Innovation Platforms to enhance learning and strengthen required capacities for innovation, and operationalising performance appraisal and incentive systems. ). Those, and other proposals, are elaborated under the later section on Strategic Recommendations for the Operationalization of PEP II. Sustainability of most of the PEP programme initiatives could also be assured if GOZ-MOF reinstates adequate recurrent budgets for both MoA & MoFL.
Page | 17
3. Overall Assessment of PEP
3.1. Programme Context
As a programme, PEP was a response to the widely held view by national stakeholders
around 2009 that the agricultural sector was struggling to achieve its undoubted potential
to make a vital contribution to economic growth, rural development, and poverty reduction
in the country. At the centre of those concerns and challenges was the then MOAC – the
ministry expected to lead sector initiatives and development programmes. It had
capacities that were known to be weak and it needed substantial revitalization of its
policies, structures, personnel, technologies, and extension services systems.
The EU, with the strong support of Cooperating Partners (SIDA, JICA, IFAD & Finnish
Cooperation), engaged the ministry and it was agreed that a comprehensive assessment of
the Ministry’s capacities be conducted. Following from that exercise, the Performance
Enhancement Programme (PEP) was conceived and designed. It was not envisioned as yet
another project, but as a programme with the ambition, and indeed, courage to take on the
wide and deep system-wide capacity challenges facing the Ministry.
PEP eventually started in 2012 with four very overloaded key result areas across the
ministry covering change management; sector policy, planning and financial management;
human resources and ICT; and monitoring and evaluation. The goal was to tackle all the
major areas of capacity weakness together, with little prioritization and over a relatively
short time frame. This ultimately proved to be the most fundamental weakness of PEP –
trying to do too many things in too short a time – a commendable array of activities were
started but many were never properly or fully completed, while some unplanned activities
were added for implementation at the behest of MAL.
3.2. Strengths and achievements
PEP initiatives in management and staff training, providing computers, developing data
bases, strengthening planning, monitoring and evaluation skills, and updating the curricula
of training institutes were all well appreciated by ministry staff. This was partly because of
the extent of the previous protracted neglect of those areas, coupled with the hope and
impetus that those initiatives brought to the ministry staff which benefited. Given the
absence of modern support facilities and systems for middle-tier professional and support
staff in the ministry, the programme support to improve ICT workplace technology,
training, and the development data bases and information systems made most MAL staff
familiar and proficient with computer and internet use, knowledge of data bases, and
reporting in their day-to-day work.
Page | 18
3.3. Weaknesses and challenges
A central expectation of stakeholders and a core purpose of PEP was to facilitate and
manage change actively across and within MAL. To begin to fulfill this role satisfactorily,
the programme needed to have and communicate widely a comprehensive change
management strategy spearheaded by champions of change from the senior levels of the
ministry. The programme did not have such a strategy, including gaining and sustaining a
strong and evidence-based commitment from senior management. The Change
Management Team (CMT) operated in the prevailing administrative mode of MAL, as a
Head Office based coordination unit, rather than leading a dynamic system-wide change
process across all levels of the ministry. The programme was not communicated
adequately or cascaded effectively to the provincial or district levels as the feedback and
records on information, training and evaluations at those levels clearly reveal. Both those
shortcomings will need to be addressed in the next phase.
The programme should have conducted a benchmarking exercise, such as a ministry-wide
staff survey, at the outset. For a performance improvement programme centered on staff
proficiencies, such a comprehensive exercise should have been mandatory so as to enable
meaningful assessments of progress in skills acquisition by staff over the course of the
programme. The generally disenabling conditions in the ministries mitigated against
optimal performance improvement (e.g. limited and late budget allocations and lack of
opportunities to put skills gained from training into practice in their work places). Periodic
appraisals on the constraining effects of those latter factors will be necessary for inclusion
in the next phase. There were also instances of wastage of resources, despite the prevailing
budget shortfalls in MAL. PEP funded an exercise to devise improved budget resource
allocation criteria for MAL which the ministry subsequently decided not to accept or utilize,
casting some aspersions on programme decision-making processes and the quality of
engagement and ownership of PEP by MAL.
3.4. Overall insights
Overall, the PEP experience was a learning journey for the EU and MAL that revealed many
instructive lessons for the design, planning, facilitation and implementation of such
programmes in future. A central one is that a programme should be implemented in a well-
defined programme mode – not as a collection of projects. Another is that the result areas
should be limited and not overloaded – the so called “doability” mantra. While PEP brought
benefits across all levels (HO, provinces and districts) of MAL, the exclusion of large
numbers of staff (half of the frontline field extension complement) from the allocations of
computer tablets led to disappointment, and perceptions of favouritism and exclusion, by
Page | 19
half of a very important cadre of MAL staff. Change management programmes and their
strategies need to challenge existing norms and modes and bring about demonstrable
change in the way public service ministries perform. To do that, a change strategy and
process was needed together with a way of knowing or gauging the point of departure and
then how well it is progressing and achieving. PEP II needs to do just that and avert the
pitfalls of the first phase.
In a nutshell, PEP certainly sought to achieve noble and worthy deeds. There is no doubt
that PEP set out to do the right thing and had some very worthy accomplishments. It
facilitated increased awareness, knowledge and insights among many staff on management
and performance issues in MAL. Its major drawback was that it attempted to do some of
those “right things” in the wrong way. Examples included embarking on a staff
performance improvement programme without conducting a comprehensive baseline
study on existing proficiencies; seeking to improve MAL budget allocation criteria without
first securing purposeful and definite commitments by the relevant GoZ ministries;
providing wide-ranging support to improving ICT capacity across MAL while being unable
to secure the commitment of senior management to put in place a dedicated unit to
optimize the management and benefits of the investment.
Page | 20
4. Programme Plans, Targets and Performance Outcomes
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
OVERALL OBJECTIVE
The agricultural sector contributes to growth and poverty reduction in Zambia
Target Increased national growth and reduced rural poverty. Indicators(s) Budget allocation to the sector according to CAADP objective to allow 6% growth and contribute to MDG 1 (halving poverty) Reduction in subsidies (FRA/FISP) to the sector to allow improvement of productive investment along CAADP lines.
Reduction in agricultural production, inter alia, is adversely weighing on growth prospects (TORs). Poverty levels among rural dwellers that make up the majority of the country's population have remained a significant problem with poverty levels at more than 77% for more than a decade and the gap between urban and rural poverty continues to widen (TORs).
Real GDP growth has been declining. This was 7.3% in 2012, 6.7% in 2013 and 5.7% in 2014. www.afdb.org 60% of the population lives below the poverty line, while 42% are considered to be in extreme poverty. www.worldbank.org Poverty levels have remained the same since 2010. Maize subsidy programmes consumed 52% of the 2014 agriculture budget.
PROGRAMME PURPOSE
The capacity of MAL is strengthened in an effective and sustainable way and their operations are improved
Target Strengthened MAL and improved service delivery Indicator(s): Number of new policies, legislation & plans approved / promulgated. Level of satisfaction of MAL staff with working environment. Percentage of filled positions against the establishment.
2 out of a target of 3 new policies crafted. Agricultural Transformation Plan and National Agricultural Policy. Revised Strategic Plans await approval. Baseline not completed. PEP final statistics indicated employment levels of 8,379 against a total staff establishment of 12,784 (66%).
PEP facilitated the development of a Strategic Plan by the ministry. PEP also supported the development of the National Agricultural Extension Strategy. Both are yet to be approved. No evidence of a baseline having been conducted. MOA & MFL statistics on staffing levels are inconsistent depending on the source. The number of extension officers in
Page | 21
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
Quality of planning, budgeting and reporting.
MoA now amongst best 50%, but still work to be done to ensure equitable distribution. Resource allocation work under way.
position increased from 48% to about 90% in the past 5 years (MOA KII). There are 1,400 camp extension officers in 1,810 camps giving a staffing level of 77% (PEP Ref Group). In ZARI the staff establishment is 60 – 70%. Significant changes and improvements still required in all three (3) areas.
OUTPUT 1
Change Management and Service Delivery improved
1. Change Management and Service
Delivery improved.
Target Adoption of a high performance culture focused on world class service delivery and customer satisfaction. Indicator(s): MAL structure reviewed and operationalised. Level of satisfaction of MAL staff with management performance. % of staff with reviewed job descriptions.
MAL structural review initiated. The programme put in place a Programme Steering Committee (PSC), Change Management Teams (CMTs), and Provincial Change Management Teams (PCMTs) to initiate and manage the change process. Survey not conducted. Old job descriptions, but there has been no updating since the last job evaluation exercise in 2010
Organizational review still at planning stage, awaiting approval of Strategic Plan (PEP Final Report). The PSC and CMT provided useful guidance to the programme, notably at the planning stages. The concept of “Champions of Change” was not integrated into the PEP programme. Baseline not completed (PEP Final Report). Restructuring has been delayed, awaiting approval of Strategic Plans (PEP Final Report).
Page | 22
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
OUTPUT 2
Sector Policy, Planning and Financial Management improved
2. Sector Policy, Planning and
Financial Management improved
Target MAL provides a clear policy framework for the sector, based on empirical evidence, disseminates this widely and uses it as the basis for planning its own operations and budget expenditures. Indicator(s): Number of policies/legislation reviewed or developed. Number of evidence-based policy advisory notes prepared annually. Number of policy dialogue meetings held annually at national, provincial and district level. Number of SAG meetings per year.
Three (3) policy documents have been completed out of the planned five (5). (PEP Final Report). Two (2) evidence-based policy advisory notes have been prepared. Three (3) policy dialogue meetings held annually. One (1) SAG meeting per year.
The two policies that have not been completed for various reasons are: The Strategic Plan and National Agricultural Policy. The former has been recently split into two strategic plans, one of each new ministry. The two (2) evidence-based policy advisory notes are the Effects of FISP reforms and Effects of Fish Ban on farmers’ livelihoods. Policy dialogue meetings at HQ level on E-voucher, Agricultural Transformation Plan, and M&E. At Provincial level annual review meetings held in three provinces with support from PEP. One AgSAG meeting supported each year, on specific themes including approval of the drafts of the Strategic Plan, National Agricultural Investment Plan and stakeholder contributions to the sector (GAFST framework).
Page | 23
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
MAL budget expenditure as % of budget released. % of departmental and district/provincial plans that are structured in line with the framework of the strategic plan.
Not specified. But should be well over target with increased FISP allocations alone (PEP Final Report). Whilst not approved, SPs remains among several policy documents used for MAL planning and budgeting.
MoF remain committed to PFM reforms. GRZ-wide M&E system under development, District and provincial plans being brought within the framework of the strategic plan, but also the NAP and NAIP. Progress has been made in harmonising these documents, but they still retain different structure and emphases.
OUTPUT 3
Human Resources Management and ICT improved
3. Human Resources Management
and ICT improved.
Target To begin to see a Human Resource Management function demonstrating a clear understanding of the Ministry’s business and become a strategic business partner, challenging and supporting the line departments and regions, and taking a lead on cultural and change management issues that inspire MAL employees to contribute to the continuous improvement in the service given to customers. Indicator(s): Payroll cleaned up for retired staff.
Completed in PE3 (2015). (PEP Final Report).
HRIS system modified, MAL staff trained, data entry complete for 95% of the staff. The activity would have been completed earlier if it were not for lack of commitment and sense of urgency on the part of Senior HRA Managers that were expected to lead and drive the process. (PEP Final Report).
Page | 24
Output(s) Target (Plan) and Indicator(s) Stated Result(s) Findings (to FCRs)
Outstanding payments of retirement packages for MACO/MLF retirees made. HR Strategy developed. Number of training programmes re-introduced. % of staff trained in line with re-introduced programmes. HRIS operational. Number of departments regularly updating their website module.
PEP contributions complete and audited. Development of HR Strategy still at planning stage. More than three (3). These include the leadership, management and ICT end-user training, three (3) in-service training programmes, and induction training. 90% of targeted provinces, 10% of staff trained nationally. Training included Leadership, Management and Supervisory courses (PEP Final Report). Data entry took long. All departments providing information to NAIS for uploading onto website.
Audit was lengthy due to poor record keeping, but eventually competed. HR Strategy not yet developed A number of new courses developed and run with PEP support. MAL should include these in their recurrent budgets in the future, not to rely so much on external funding and overseas training. Induction of new staff still remains a challenge owing to insufficient funds allocated to this activity. Data entry completed by the end of October 2015. Testing and commissioning the system followed. More pressure should be applied by senior management to ensure on-going updating of website.
Page | 25
OUTPUT 4
Monitoring and Evaluation improved
4. Monitoring and Evaluation
improved.
Target Effective and efficient M&E system leading to evidence-based policy, planning and investment strategies. Indicator(s): % of quarterly M&E reports generated timely i.e. in line with M&E system M&E database system operational
Results not specified. M&E database under development and being piloted.
The template to be used for M&E reporting across government ministries is still being worked out. (PEP Final Report). M&E database developed but yet to be commissioned. .
CROSS-CUTTING ISSUES
Gender, HIV & AIDS improved
Gender
Target
Gender awareness is mainstreamed in all MAL activities. Indicator(s)
MAL Gender Guidelines and Checklists developed and applied to planning to ensure equal opportunity and access to both women’s and men’s needs and interests.
Gender mainstreaming guidelines were developed and distributed at all levels. A TOT for mainstreaming for gender was also conducted for senior management and provincial staff.
Gender mainstreaming, sensitization and training conducted.
HIV & AIDS
Target
HIV and AIDS is mainstreamed in all MAL activities.
Page | 26
Indicator(s)
MAL Wellness and HIV/AIDS Workplace Policy developed and staff sensitised at all levels.
The MAL Wellness and HIV/AIDS Workplace Policy was put in place and senior management were sensitized.
MAL Wellness and HIV/AIDS Workplace Policy developed and sensitization initiated.
Page | 27
5. Findings, Conclusions and Recommendations Output(s) Findings
Conclusions Recommendations
OVERALL OBJECTIVE
The agricultural sector contributes to growth and poverty reduction in Zambia
Real GDP growth has been declining. This was 7.3% in 2012, 6.7% in 2013 and 5.7% in 2014. www.afdb.org 60% of the population lives below the poverty line, while 42% are considered to be in extreme poverty. www.worldbank.org Poverty levels have remained the same since 2010. Maize subsidy programmes consumed 52% of the 2014 agriculture budget.
Overall the programme has had very little direct impact so far on the agricultural sector, priorities, maize subsidies, economic growth and poverty reduction in Zambia.
There is need for the programme to play a more influential role in policy formulation, implementation and review.
PROGRAMME PURPOSE
The capacity of MAL is strengthened in an effective and sustainable way and their operations are improved
PEP facilitated the development of a Strategic Plan by the ministry. PEP also supported the development of the National Agricultural Extension Strategy. Both are yet to be approved. No evidence of a baseline having been conducted. MOA & MFL statistics on staffing levels are inconsistent depending on the source. The number of extension officers in position increased from 48% to about 90% in the past 5 years (MOA KII). There are 1,400 camp extension officers in 1,810 camps giving a staffing level of 77% (PEP Ref Group). In ZARI the staff establishment is 60 – 70%.
Efforts to provide strategic directions to operations have been hindered by the structural decisions to have two ministries and protracted decision-making processes. With no baseline, monitoring and evaluation of programme progress is difficult and cumbersome. There doesn’t seem to be coherent, centralized and reliable information on personnel and staffing in the ministries.
PEP II should facilitate the development, approval/promulgation of strategic plans by both the MOA and the MFL. PEP should also go beyond policy formulation by promoting implementation in order to improve service delivery. Baseline studies in target districts should be given priority during the orientation phase of PEP II. The Human Resource Information System (HRIS) and supporting data bases need to be more professional and reliable.
Page | 28
Output(s) Findings
Conclusions Recommendations
Significant changes and improvements still required in planning, budgeting and reporting.
While the programme made valiant efforts to improve these three areas, the ministry’s low capacity base at the outset, coupled with the ministry’s current dysfunctional resource allocation and disbursement system which undermines bona fide attempts to strengthen professional planning, budgeting and reporting.
The two ministries need to review and reform the existing budget system and take advantage of Core Functions Analyses exercises to guide and assist them in modernizing their budgeting process, especially in view of the impending decentralization process.
OUTPUT 1
Change Management and Service Delivery improved
1. Change Management and Service
Delivery improved.
Organizational review still at planning stage, awaiting approval of Strategic Plan (PEP Final Report). The PSC and CMT provided useful guidance to the programme, notably at the planning stages. The concept of “Champions of Change” was not integrated into the PEP programme. Baseline not completed (PEP Final Report). Restructuring has been delayed, awaiting approval of Strategic Plans (PEP Final Report).
While these committees were effective at the national level, they were not sufficiently effective at the provincial and district levels. Without the baseline, it will be difficult to make assessments of the level of satisfaction of MAL staff and PEP performance in general. Restructuring remains key for effective change management, enhanced performance and service delivery.
Organizational review to be a priority for PEP II, The PEP focal persons at provincial and district levels need to be included in higher level change management teams. There is also need for active change management and facilitation teams at provincial and district levels. It is critical that benchmarking surveys be conducted. These can take the form of staff surveys or farmer perception surveys. This can be implemented during the orientation or start up phase of PEP II. There is need to hasten the development, finalization and approval of separate strategic plans for the MoA and MFL that will serve as the basis for restructuring.
Page | 29
Output(s) Findings
Conclusions Recommendations
OUTPUT 2
Sector Policy, Planning and Financial Management improved
2. Sector Policy, Planning and
Financial Management improved.
The two policies that have not been completed for various reasons are: The Strategic Plan and National Agricultural Policy. The former has been recently split into two strategic plans, one of each new ministry. The two (2) evidence-based policy advisory notes are the Effects of FISP reforms and Effects of Fish Ban on farmers’ livelihoods. Policy dialogue meetings at HQ level on E-voucher, Agricultural Transformation Plan, and M&E. At Provincial level annual review meetings held in three provinces with support from PEP. One AgSAG meeting supported each year, on specific themes including approval of the drafts of the Strategic Plan, National Agricultural Investment Plan and stakeholder contributions to the sector (GAFST framework). MoF remain committed to PFM reforms. GRZ-wide M&E system under development,
The government has been dragging its feet in approving the policy documents. This can be a reflection of lack of commitment, lack of political will or a failure to assume ownership of the PEP programme. Policy impact assessments are critical for improving policy relevance, impact and harmonization. Policy dialogue meetings initiated but more still needs to be done. These are yet to be fully decentralized. The Agricultural Sector Advisory Group served as the all stakeholder platform. The PEP programme, however, failed to work with the Agricultural Consultative Forum (ACF) for various reasons. Very little progress has been achieved in adopting the recommended budgetary allocation criteria/formulae.
There is need for better commitment, political will and for the government to assume ownership of the PEP programme to ensure greater support of the programme’s initiatives. Policy formulation, implementation and review should never be once-off processes, but rather provide platforms for the inclusive formulation of objective and consistent policies, constant reviews, and linked communication systems. To achieve greater impact policy dialogue should be facilitated and supported at all levels including the provincial and district level. Such platforms should also be initiated in all provinces and districts. The AgSAG should be further re-vitalised to provide a continuous dialogue between GRZ, CPs and stakeholders, rather than only meeting when a critical document has to be developed. The ministries continual problems with their budgetary processes and disbursement shortfalls need to be considered by the AgSAG group.
Page | 30
Output(s) Findings
Conclusions Recommendations
District and provincial plans being brought within the framework of the strategic plan, but also the NAP and NAIP. Progress has been made in harmonising these documents, but they still retain different structure and emphases.
Policy formulation exercises in themselves have limited value unless they are linked to and used for implementation.
The MoA and MFL strategic plans should be finalized as a matter of urgency.
OUTPUT 3
Human Resources Management and ICT improved
3. Human Resources Management
and ICT improved.
HRIS system modified, MAL staff trained, data entry complete for 95% of the staff. The activity would have been completed earlier if it were not for lack of commitment and sense of urgency on the part of Senior HRA Managers that were expected to lead and drive the process. (PEP Final Report). Audit was lengthy due to poor record keeping, but eventually competed. HR Strategy not yet developed More than three (3). These include the leadership, management and ICT end-user training, three (3) in-service training programmes, and induction training.
Payroll cleaned up. All outstanding payments of retirement packages cleared. The HR Strategy is long overdue. A number of new courses developed and run with PEP support.
There is need for a change management strategy that can facilitate a mindset change during the orientation phase of PEP II to ensure a common vision and assumption of programme ownership by all MoA and MFL staff. Record keeping to be improved. The HR Strategy is the overarching guiding documents on how to proceed and deal with critical HR issues. Its development should therefore be a priority. MoA and MFL should include these new courses in their recurrent budgets in the future, not to rely so much on external funding and overseas training. This will ensure sustainability of the PEP initiative.
Page | 31
Output(s) Findings
Conclusions Recommendations
Induction of new staff still remains a challenge owing to insufficient funds allocated to this activity. Data entry completed by the end of October 2015. Testing and commissioning the system followed. All departments providing information to NAIS for uploading onto website.
Induction of new stuff remains a peripheral priority for government. HRIS operational. Innovation is required to ensure that websites are consistently updated.
The induction of new stuff is key for enhancing performance and for improving service delivery. It should be a top government priority for both implementation and funding. HRIS to be consistently maintained and updated. There is need for incentives and awards to ensure that websites are consistently updated. More pressure could also be applied by senior management to ensure on-going updating of website.
OUTPUT 4
Monitoring and Evaluation improved
4. Monitoring and Evaluation
improved.
The template to be used for M&E reporting across government ministries is still being worked out. (PEP Final Report). M&E database developed but yet to be commissioned.
The government, with EU support, is developing a government-wide M&E system. Encouraging progress has so far been achieved in M&E.
M&E, which is one of the three areas to be addressed by PEP II, should incorporate lessons learnt and recommendations from the government-wide M&E system. Commissioning, adoption and utilization of the M&E database have to be hastened.
Page | 32
Output(s) Findings
Conclusions Recommendations
CROSS-CUTTING ISSUES
Gender, HIV & AIDS improved
Gender
Gender mainstreaming guidelines were developed and distributed at all levels. A TOT for mainstreaming for gender was also conducted for senior management and provincial staff.
Policies and guidelines put in place, but there is no dedicated full-time officer. The function is currently under PPD.
There is need for a full-time gender officer, preferably under the HRA departments.
HIV & Aids
The MAL Wellness and HIV/AIDS Workplace Policy was put in place and senior management were sensitized.
Policies and guidelines put in place, but there is no dedicated full-time officer.
The position of an HIV & AIDS coordinator should also be a full-time position i=under the HRA departments.
Page | 33
6. Programme Oversight and Management A Programme Steering Committee (PSC) provided oversight and policy direction and was
to meet twice a year while a national Change Management Team (CMT), comprising the
departmental directors, met quarterly to deal with programme implementation issues.
Management arrangements, in line with the Financing Agreement, had programme
supervision provided by the Director of the Policy & Planning Division (PPD), MAL who
was the Imprest Administrator while the Chief Accountant was the Imprest Accounting
Officer. In addition to those arrangements, a member of staff of the Policy and Planning
Department was appointed as a MAL/NAO Liaison Officer. The PSC struggled to have
meetings in accordance with the frequency stipulated in its ToRs; supervised PEP in an
uninspiring, administrative mode; and displayed little innovation in communicating PEP
across MAL e.g. no evidence of national seminars on change management processes or
interaction or lesson learning from other countries implementing agricultural sector and
ministry change programmes in SSA.
Provincial Change Management Teams (PCMTs) were nominally established at all
provincial head offices, with representation from the districts, but were not actively
engaged in programme implementation as the programme was centrally managed.
Provincial staff from MAL departments, however, were involved in coordinating PEP
supported activities (e.g. ICT equipment distribution, SAO & SEO training, HRIS data
collection).
The programme was supported by a team of two long-term technical assistants, comprising
the Team Leader and Institutional Development/Change Management Specialist and the
Human Resource Management Specialist. Programme administration was provided by the
TA team and a Programme Support Unit based in the offices of PMTC (Zambia) Limited,
comprising an accountant, an administrator and a driver.
The CMT should give more priority to strengthening planning processes at provincial and
district levels and facilitate more interaction, inputs and meetings with ministries middle
management staff. Given the stated commitment for PEP II, supported by MoA & MAL, to
shift capacity development efforts and resources more substantially to the provinces
during the next phase, it is recommended that there be Provincial and District Change
Facilitation and Implementation Teams in target provinces/districts, and that the provincial
teams have formal representation on the PSC.
The MoA and MOFL management, at both provincial and district levels, should ensure
integrated planning i.e. PEP activities as part of the normative ministry quarterly/monthly
planning processes with reviews conducted by M&E officers at district and provincial
levels. As decentralization becomes imminent, it is suggested that PSC meetings be held
Page | 34
rotationally in focal PEP provinces and involve periodic direct presentations from the
Provincial & District Teams. This will facilitate the much needed closer interaction and
stronger mutual understanding between oversight and implementing entities in the second
phase of the programme.
Page | 35
7. Financing and Budget Despite its delayed start, the programme succeeded in spending a high proportion of its allocated funds (see 2.3 on Efficiency).
The amount of the overall budget allocated and disbursed under PE 2 was very high, most probably arising from the need to
compensate for the delayed start-up. While HRM & ICT allocations were the highest elements in the overall budget, the
inability to make provision for the procurement of tablet computers for half of the field extension staff complement was a
major omission in terms of financial planning, equitable treatment of staff, prudent use of funds, and programme reputation.
For the future, there is need for a timeous inception phase and the addition of an exit phase with a separate PE.
Component FA Budget Start-Up & PE.1
Expenditure
PE.2 Expenditure
PE.3 Expenditure
FA Budget Balance
% Usage
Operations (Programme Estimates) 6,990,000 1,828,414 4,102,399 411,952 647,235 80%
Change Management and Service Delivery 1,150,000 185,454 659,574 82,250 222,723 81%
Sector Policy, Planning and Financial Management
1,300,000 272,470 751,423 87,854 188,253 86%
Human Resources Management and ICT: 3,080,000 232,934 2,465,625 137,636 243,805 92%
Retirement Packages 1,120,000 1,101,913 18,087 0 100%
Monitoring and Evaluation 340,000 35,643 207,690 104,212 -7,546 102%
Technical Assistance Service Contract 1,050,000 1,050,000 0%
Project Monitoring, Evaluation and Audit 310,000 310,000 0%
Communication & Visibility 50,000 17,659 12,399 19,941 60%
Contingencies 500,000 500,000 0%
Page | 36
Source: NAO
37
8. Cross-cutting issues: HIV/AIDS and Gender
As described in the FCRs (Section 5 earlier) a Gender, HIV/AIDS, and Wellness policy was
put in place and guidelines for mainstreaming of gender were also developed, There was,
however, no dedicated full-time officer for either HIV/AIDS or Gender. The policies and
guidelines developed were not implemented across MAL during the course of the
programme. Without a dedicated function and an associated staff complement, the chances
of active and enduring actions in these areas are slight.
The function for those areas is currently under PPD. It is recommended that this function
be transferred to HRA and one officer be appointed to cover these issues in each of the two
ministries – perhaps a HIV/AIDS Officer in MOA for both ministries and a Gender Officer in
MFL to cover both ministries. Structural separation should not preclude inter-ministerial
planning and collaboration in important HR areas where there are resource constraints.
38
9. Strategic Recommendations for the Operationalisation of PEP II
9.1 Building on the Lessons from PEP 1
The major overall conclusions from PEP 1 were that the programme initiated and
implemented many worthy actions and had significant achievements in strengthening key
areas of weakness (HR, leadership and management training, ICT equipment, farmer
registration, M&E, curriculum updating) in the performance of MAL. The programme was,
however, overambitious in what it sought to accomplish in a normative project timeframe;
it did not have a robust Ministry-wide change management strategy; it needed stronger
commitment from senior management; it was implemented in component rather than
holistic mode; and, by its closure, had yet to witness substantive shifts in the way the
Ministry (now two, MOA & MFL) understood, and managed the proficiencies and
performance of its staff.
From the wider capacity development perspective, there is a need for all staff in the two
ministries to understand what capacity development means and entails. The entire
workforce need to know why it matters so much for key public services institutions in the
agricultural sector to train and motivate their staff so as they can fulfill their roles, provide
services to stakeholders, and be fulfilled and rewarded in doing their jobs at all levels in the
ministries.
Given the imminent GRZ moves to decentralization and the need for more local autonomy
in planning, decision-making and accountability – especially in Districts – line ministries
have been requested to draw up devolution plans which will involve major shifts for those
responsible for rural service delivery including MoA, MFL and other key service ministries
such as Health and Education. There will be integrated local planning processes and future
assessments of farmer demand and service needs will have to be more thorough and
rigorous than at present. Early in PEP II, MoA & MoFL should prepare a discussion
document on Decentralized Agricultural Extension Services Provision and include it in the
ongoing development of the National Agricultural Extension Strategy ( NAES).
9.2 Facilitate, communicate and implement a Change Management Strategy for
MOA & MoFL
There is an evident and vital need to foster widespread knowledge and understanding by
all ministry staff on the purpose, processes, and desired outcomes of the second phase of
PEP, and the change management process that will be facilitated and implemented to
consolidate and further enhance the performance of both ministries.
39
9.2.1. There is a need for an Orientation Phase to achieve better institution-wide
familiarisation on capacity development; focused awareness on the three result areas and
the planned thrusts and activities under them; and a widespread and embedded common
understanding on what key outcomes the ministries are seeking to achieve, and the likely
mitigating challenges, over the next four years. One of the key findings of the PEP Final
Evaluation Team was that communication was a major weak link in the general MAL
culture and operations, and that PEP had not sufficiently recognized this and had not been
able to take effective action to redress it. The following actions are needed to ensure a more
purposeful and systemic change process under the new phase.
• A purposeful ministry-wide change communication strategy (Action Fiche 1.4) is
essential and needs to be actively led by top management and cascaded right across the
entire ministries, down through the various levels to frontline extension staff. This
strategy needs to be developed at the outset of this phase (the first key deliverable) and
it needs to be clear that this differs substantially from the normative project
communications and visibility strategy which serves principally to promote the
presence, profile and contributions of projects. The change communication strategy is
focused centrally on internal ministry processes, including the so called three R’s – roles,
responsibilities and relationships – to strengthen communication, performance insights
and synergies within and between all units of the ministries. The PSC needs to ensure
that the CMTs at all levels devise and implement this strategy with guided TA facilitation
and support in the initial stages. For example, Organizational Conversation is one of the
most recently recognized ways to most effectively improve internal communications
processses in organizations. It involves four elements ( 4 I’s) :
1) Intimacy: Leaders must reduce the space, institutional as well as spatial,
between themselves and employees.
2) Interactivity: Conversation is an exchange; organizational communication also
needs to be an exchange, a dialogue — not a lecture from the top.
3) Inclusion: All participants must be allowed to contribute to the conversation in
equal portions. The goal is not feedback, but actual employee-generated content.
4) Intentionality: The best conversations are not idle; there is a goal or purpose to
them. Likewise, organizational conversation should have an agenda that aligns with
the strategic objectives of the organization As the authors explain: "While the
elements of intimacy, interactivity and inclusion serve to open up that conversation,
the element of intentionality serves to ‘close the loop’ on it."
40
Starting with senior management, there is need for knowledge-centered learning and
training workshops (short periodic modules supported by modern learning supports e.g.
web sources, case studies, CDs, guides, factsheets ) on capacity development.
Management and staff across the ministries should be basically familiar with modern
principles and practices for capacity development, including, for example, the UNDP
default principles for developing countries and what they mean for ministry staff.
• This knowledge enhancement on capacity development should include organization
development; individual and team performance, including high-performing and self-
managed teams; skills and proficiency profiles; key performance indicators; self-
assessment; and performance management in public sector organizations. These
“knowledge building workshops” should be cascaded to district staff where “local
champions of change” are identified and become the focal (and sometimes rotating)
nodes of learning and innovation on organizational and performance improvement
approaches for service delivery.
• Provide support to an ethos of strong “organic” ownership by ministries for their
systems and performance outcomes; this means integration of learning and knowledge
acquisition adapted to their priorities and their very evident needs for renewal and
modernization. Such ownership also means a progressive recognition by ministries of
the limitations of a long-term, over-reliance on discreet or separate projects as part of
the ministries ad hoc and tactical responses to meeting pressing demands for service
delivery to smallholders. Such projects should not just substitute or supplement the
ministries’ efforts – they need to be designed and implemented to generate relevant,
proven, and scalable learning and innovation (approaches, models, tools, systems) that
can be demonstrably integrated and mainstreamed across ministries to enhance wider
national performance in agricultural sector development. This also means placing a high
priority on “master trainer capabilities” across ministries at Head Office, Provinces and
Districts.
• Towards the end of the orientation phase, both MoA & MFL need to engage in a renewed
strategic planning process. The strategic plans should be more than just strategy
documents that are not linked to operational plans or MTEFs. The analysis (so called
environmental analysis exercise/section) for the strategic plans should include a
thorough services analysis and costings for all departments and units of the ministries.
This will provide the foundation for informed and evidence-based decision-making,
based on the longstanding maxim that “form follows function” -- structures and systems
should be based on well-interrogated and thought out functions. Subsequent skills
audits and profiles of staff categories will establish the existing knowledge base and
competencies, so as training can become more targeted and efficient. In addition,
ministries would be able to conduct capability assessments of partner service providers
41
such as NGOs and collaborate with them more purposefully in district services planning
and coordination.
9.3. Strengthening Extension Services Systems in Collaboration with Research
(Action Fiche 2.1 & 2.3).
9.3.1. Upgrading Competencies of Provincial & District Staff
A major lesson -- reinforced strongly during PEP I -- was the realization that extension
services nationally have been severely neglected for decades in training and skills
upgrading – a critical and continual requirement to ensure staff are updated, relevant and
motivated in their work. Virtually no refresher courses or skills upgrading were provided
for frontline staff who mostly had poor equipment and workstation facilities, and little
resources for mobility. Extension strategies and guidelines for agricultural service
providers are being revisited including the provision of supports such as diaries, manuals
and handbooks. However, frontline extension officers across the country critically need an
array of new skills to become more relevant and skilled for modern service provision to
smallholders.
During this evaluation, block and camp extension officers were found to be very aware of
their weak training and support and that they were often unable to respond to what is
termed “real farmer demand” – the more demanding and progressive smallholders who
need up-to-date advice on new technologies, business, finance, and markets. It was also
found that most CEOs need booster training and skills enhancement for their primary
extension duties to strengthen household food and nutrition security.
Arising from this long-term neglect, it is clear that capacity development of the extension
management and services delivery system is the overriding major priority for PEP II and it
is recommended that it merits the predominant share of the indicative budget allocation
for programme estimates (€5 750 000).
9.3.2 Extension Training in Facilitation and PEA
Given the predominant absence of upgrading and refresher courses over the past decades
for field staff, most extension officers across the country still practice outmoded “transfer
of technology” approaches which viewed smallholder farmers as passive recipients of
technical information and not active partners in co-learning with extension officers.
In the past, the “top down” approaches used in development activities in rural Africa often
consisted of farmers and communities being told what to do. Participatory approaches
have evolved from past poor experience with these “old” approaches, where farmers were
often instructed, while their views about research needs and extension activities were not
42
sought. Participatory approaches, on the other hand, are now widely accepted in
development practice as being concerned not only with improving farming practices but
also with issues of empowerment and giving a “voice” to farmers. Participatory approaches
recognize the importance of all stakeholders in identifying and analysing problems,
achieving consensus on responses, generating and disseminating knowledge, and in
removing systems constraints.
PEA was implemented by GIZ in Siavonga District in the 90’s and JICA promoted
participatory approaches under RESCAP from 2010 in some Districts such as Kafue. Those
experiences, lessons and approaches have not yet been mainstreamed or taken to scale
across the national system but the accruing lessons need to be taken on board.
Under PEP II, there needs to be a guiding extension training strategy which focuses on (1)
facilitation skills for core trainers at provincial and district levels and (2) district-level
training of field staff in PEA with strong emphasis on learning by doing with farmers.
Guidelines for benchmarking (staff survey, skills audit and KAP studies at the outset and
after 3 years) need to be developed
The facilitation training will need to be conducted by experienced professional
practitioners in facilitation skills development for modern agricultural service delivery
systems. The PEA training for district staff would be conducted by training teams including
the trained facilitators (1), the designated district MoA/MFL training officers, supported
initially by the TA Advisor in Extension Systems, and short-term TA specialists in
participatory approaches in research and extension. The training programmes in
facilitation would be a minimum of one week in duration while the district PEA training
would be a minimum of two weeks duration over two modules. Rigorous evaluation, of
those training programmes should be conducted and skills profiles of all participants
compiled, shared with management, and retained in renewed District profiles/databases as
well as in the HRIS.
In line with GOZ national economic and social development goals, he “new districts” (of
which there are circa 30 with most in remote and disadvantaged areas) should receive
priority in planning and provision for the above training/skills programmes in the
intended pluriannual programme estimate. It should focus its resources predominantly on
the so called “software” of extension system performance (strategies & skills) as it cannot
contribute to or supplement substantially (as some discreet projects partially do) the
recurrent operational funding of ministries.
9.3.3 Strengthening District Agricultural Profiles
Given the significant investments and achievements by PEP I in MAL information systems
and supports – including M&E systems – and the continuation of support proposed under
43
PEP II (Action Fiche 1.3), there is a crucial need to optimize on this investment and achieve
improvements in the data/information base of most districts. Districts’ data culture and
related information capture remains weak and patchy and does not provide a sufficiently
reliable and informed basis for either local extension programme planning or as input to
higher level planning and policymaking. In a change management programme seeking to
improve performance at frontline delivery levels, this is a task that requires a local and
integrated team response. With decentralization down the line, the reinvigoration of the
District Agricultural Committees (DACs) should be pursued to give much needed impetus
to local planning, implementation and evaluation processes.
District-based Task Teams comprising relevant ZARI research station managers, DACOs
from MOA& MFL, and M&E designated officers, should be set up under the second PE (after
the orientation phase) in 10 districts initially (mostly the “new” remote ones) to bring
district profiles up to date to better inform upcoming decentralized district planning
during the off-season (winter). The profiles, which would start from area levels, should
cover inter alia land area, classification, arable ha, suitability, precipitation data, farm sizes,
farmer categories and numbers, crop & livestock enterprises, yields, smallholder gender
and age structure, profile of farmer groups and associations and commodity foci, profiles of
NGOs active in district, MOA/MFL staff lists, agri-input companies and dealers, names and
enterprises of lead/high performing farmers, young farmers groups/clubs ..et al. Again,
depending on the performance outcomes achieved in the 10 districts, this can be extended
to other districts – and with lessons and insights -- probably can be completed more cost
effectively than for the first 10.
As recommended in the final evaluation of PEP 1, baseline surveys will be conducted in all
focal districts to benchmark the extension capacities and provide a basis for assessment of
progress during the second phase.
9.4. Learning and Innovation Platforms for Research and Extension
The need to “institutionalize joint research-extension planning and strengthened
dissemination” (Action Fiche 2.2) is fully acknowledged by ZARI, MoA & MFL and how best
to do this is the key challenge for the ministries, their agriculture and livestock
departments, research institutes, farmers associations, and private sector stakeholders.
It is recommended to facilitate and support the setting up of innovation platforms ( Box
XX) comprising personnel from research, extension, farmers and their associations, ago-
dealers, NGOs, and the private sector. Zambia has some experience of such platforms at
district levels to promote the adoption of conservation agricultural practices (2010) and
this, coupled with considerable SSA experience over the past 15 years, should be harnessed
to inform the setting up of innovation platforms under PEP II.
44
There are 10 ZARI research stations located around the country and three should be
selected during the Orientation Phase ( in close consultation with ZARI, MOA & MFL, ACF,
NFU) to establish learning and innovation platforms, each with circa 10-12 members
initially, including PACOs and DACOs from the hinterland province and districts. Together,
preferably under a rotating chairperson (extension, research, farmer association leader),
and meeting twice a year, the members facilitate multi-stakeholder input and feedback on
local agricultural development priorities, farmer demand, and the technical services and
training responses needed from research and extension for the focal hinterland farming
systems and enterprises. Based on identified needs, joint and integrated training
programmes covering technical subjects, farmer organization & market development, and
value chains (3-5 days involving ZARI, DOA/Livestock, SCCI, NGOs) should be planned and
organized for District staff, CEOs and farmer associations/groups.
Innovation platforms should arrange for topic/subject sessions and invite agricultural
practitioners and experts periodically to provide inputs and advice on chosen subjects. The
learning and innovation experiences with the three initial innovation platforms should be
carefully evaluated after two years and comprehensively documented as case studies and,
depending on the outcomes, further platforms considered for initiation and support in the
subsequent phases of the programme.
9.5. Incentives to Recognize and Reward Distinctive Achievements in Performance
Long-run experiences with workforces globally (in both public and non-public
organizations) provide ample evidence of how incentives can spur and boost the
performance of individuals and teams across all cultures and nations. Awards have been
successfully used in some countries in the SADC region over the past 20 years as one way
to recognize and spur improved performance in public extension service systems.
While not included per se in the PEP II Action Fiche, it is recommended that incentives and
awards should be included, probably under the following areas (Action Fiche: 1.4, 2.1, 2.2,
& 2.3). The incentive/award system would be first discussed with public service ministries
in the context of revitalizing the public service performance appraisal system. If given the
go-ahead, it would be introduced under PEP II for the first 2 years after which there would
be eventual co-sponsorship (50:50) by the ministries and PEP over the last two years, with
the ministries then mainstreaming the principles and practices of incentives across their
key performance areas,
• Under 1.4, MOA/MFL individuals that achieve outstanding merit as “Champions of
Change” in leading the change process or highly motivated teams would receive, for
example, the PEP Change Management Award (annual?). PEP peers, a HR representative,
45
and the TA TL would develop the award criteria for approval by the PSC . The prize
could be a sponsored place on an appropriate training course/programme linked to
advancing their professional or personal development.
• Under Result 2, an annual award for the Best Block or Camp Extension Officer of the
Year is commended for each ministry. The criteria would be drawn up by a group
comprising peer CEOs, Chief Extension Officer, HR Officer, and TA Extension Advisor.
The award could be sponsored participation in a relevant study tour related to field
extension systems or an appropriate training course/programme linked to advancing
their professional or personal development. There is plenty of scope for awards for
other job categories and for other areas e.g. Best District Extension Performance –
possibly a biennial award.
Other possible areas for awards could be Best District Profile Award, Best District Team of
the Year -- especially for the new districts targeted under PEP II. Under Action Fiches 1.3,
there could be an annual award for Best Input to the Ministry Website. Other worthy areas
and awards can be chosen or nominations/ideas sought from staff by the Change
Management Teams.
9.6. Focal Districts for PEP II
Based on extensive discussions during the evaluation with senior management and staff in
MOA/MFL, it is proposed to focus primarily on the new districts of which 30 have been
created over 10 years or so and are spread across most provinces. It is proposed to focus
PEP II on circa 30 districts in total, 20 of which would be the new districts. An overall
guiding criterion, based on discussions with ministry management, for all PEP II focal
districts would be that there is no other CP or NGO active in them currently.
The principal reason for focusing decentralized development on 20 “new districts” is that
they are recognized to have “foundation capacity development needs” in developing basic
skills and supports for agricultural extension planning and services provision. A GRZ
programme like PEP needs to begin to correct what have been described by PPD as the “the
imbalances in development” where remote and peripheral regions/districts have been
marginalized and experienced relative neglect over the past 15-20 years.
The key guiding criteria for targeting these districts is that:
• Poverty levels are high among communities
• The nutrition status of households is generally poor with relatively high levels of child malnutrition and stunting.
46
• They were created relatively recently
They have relatively high numbers of newly recruited staff (especially MFL) all of
whom require foundation training in providing extension services for resource-poor
communities.
So there is both a need and an opportunity to develop core capacity based on modern
extension service delivery systems, including management and staff training in
participatory extension approaches that focus strongly on assuring food and nutrition
security at household levels. Those districts present an opportunity to provide integrated
extension training in communities and implement the operational guidelines for food and
nutrition officers in collaboration with CEOs, thus mainstreaming nutrition into the
extension systems of both MOA and MFL.
The other 10 districts would be selected based on the following criteria:
• Management and staff have had “threshold” capacity development to provide
extension services based on their experiences over 5- 10 years with donor-funded
projects (e.g. SIDA, JICA, IFAD, World Bank) or with NGOs (e.g. Concern Worldwide,
World Vision, Self-Help Africa).
• The districts have considerable potential to achieve impacts in the commercialization
of smallholder agriculture and in contributing to local economic and sector growth.
• There is scope to develop public-private partnerships for commodity enterprises and
value chain development.
• The districts have relatively good infrastructure including feeder roads and proximity
and access to markets.
Case studies on learning and innovation from all 30 districts would be continuously and
comprehensively documented; used in training programmes and for out scaling to other
districts; and be on ministry websites.
9.7 Knowledge Management in Ministries
PEP I had major achievements in improving farmer registration, contributing to the
enhancement of the M&E system, renovating and relaunching the website, and supporting
the provision of introductory training in ICT in MAL. PEP also advised and advocated for a
fully-fledged ICT Unit in the ministry, but this was not acted upon by the end of the project.
Further strengthening of the ministries information systems across an array of areas is
proposed (Action Fiche: 1.3). To operationalize this commitment effectively, it is proposed
47
that the ministries adopt a more coherent and integrated approach to their “itsy-bitsy” and
disjointed collection of activities in data and information systems management and
operations. PEP II should have a strong knowledge management component to
professionalize and bring cohesion to the ministries information systems from data
collection/capture right through to storage, access and dissemination.
Advantage should be taken of the CFA to assess the current situation thoroughly and, based
on previous PEP I consultancy exercises and lessons from experiences from other
comparable ministries in COMESA countries, develop and capacitate a knowledge
management system for both ministries, including appropriate structures, systems and
staffing for modern information management across all departments and units.
48
Annex 1: Detailed Evaluation Method, Limitations and Tools
APPROACH AND METHODOLOGY
Preamble
The focus of the methodology for this evaluation was on the completion of two separate, but
closely linked, assignments:
1) The final evaluation of PEP; and,
2) The development of recommendations for PEP II.
Phases & Tasks Involved
The central focus was on the assessment of PEP achievements under the national prevailing
economic and institutional realities; key indicators and modalities to achieve result-oriented
approaches; and core institutional capabilities and commitments required to underpin
improved performance by key sector actors; and the quality of outcomes, results and lessons
from the focal interventions.
This evaluation undertook to further enhance and deepen understanding of the cause and
effect linkages between activities and results as a key learning platform for the European Union
and its partners on agricultural sector performance enhancement and capacity development
programmes to understand not only (1) what works and what does not, and (2) critically why,
and under what circumstances – the latter providing qualitative insights to inform future
programme designs, implementation modalities, management, operations, and funding..
This cross cutting and multi-intervention evaluation was implemented in accordance with the
guidelines for the OECD DAC Evaluation Criteria, including those on capacity building and
development. Under the guidance of the Reference Group, the team focused its work on the
four key output areas as delineated in the ToRs. The methodology to complete those phases
included the following sequential steps and processes:
INCEPTION PHASE
1. Initial Meetings with EU Delegation staff and key MOA/MFL personnel involved with PEP,
including implementing partners as advised by the Reference Group/EUD.
2. Inventory and review (desk studies) of all relevant project documents including
preparatory processes; contracts; proposals; log frames; annual plans of operations; work
49
packages; baseline surveys, needs assessments; training programmes, reports, and interim
evaluations.
3. Briefing meetings before the start of the field phase at the EU Delegation/ Reference Group
with staff and key stakeholders presenting the findings of the desk phase, as well as the work
plan -- envisioned approach, methodology and itinerary/organization -- for the field phase. The
team proposed, discussed and agreed with the Reference Group the key evaluation questions
(as per the 5 OECD DAC Criteria) to be addressed during the evaluation.
FIELD PHASE
4. Field Phase: Based on guidance & advice from the Reference Group, an itinerary of
interviews, meetings and discussions was held with (1) key personnel involved in PEP
oversight, management, and implementation including relevant GRZ ministries (e. g. PSMD),
(2) institutional implementers and participants in PEP and (3) local institutions or organizations
(public sector/NGOs/Farmer Organisations) involved in PEP capacity development activities,
and (4) cooperating partners including IFAD, JICA, and Finnish Cooperation and (5) multilateral
agencies involved in agricultural sector programmes such as the World Bank, & Africa
Development Bank.
5. Analyses of Programme Performance: Using a progress and performance matrix grid, an
appraisal of all PEP key result areas (output & outcome) as per the PEP Log Frame, was initially
conducted, including status of performance reviews/or evaluations, analyses of outcomes and
results against targets, findings, conclusions and recommendations on PEP implementation
effectiveness. The four key result areas for PEP was assessed under the above matrix grid,
coupled with the key evaluation questions.
This was combined, as appropriate, with more in-depth performance assessment(s) in key
process areas (also using a systematic matrix approach) to examine the log frame, plans of
operations, targets, indicators, achievements, and outcomes. The key lessons arising, and
recommendations covering institutional capacity development processes, including the
appropriateness of change management and organization development (OD) approaches
implemented by PEP was also assessed.
The overall programme was assessed for consistency with the EU country strategy and
complementarity with other donor programmes. This responded to the key evaluation
questions derived from the five OECD-DAC criteria and the two EU specific criteria (EU added
value in design and implementation & coherence with the EU country strategy, member states
and other donors). Following the above comprehensive analysis, the hypothesis for PEP as a
performance enhancement programme was also systematically examined and verified.
50
6. The follow-through actions on the Mid-Term Evaluation Report, (September 2014) was
assessed in the context of the subsequent implementation and completion of the programme.
A perspective was elaborated on how effectively and efficiently the PEP four output areas were
implemented, including overall resource utilization, using the OECD DAC evaluation criteria.
SYNTHESIS PHASE
7. Across all key outcome areas, a Consolidated FCRs (Findings, Conclusions and
Recommendations) Matrix across all projects was completed by the team and the Reference
Group was fully debriefed via a slide presentation on the key findings, conclusions, and
emerging recommendations coming from the evaluation. The priorities for operationalizing PEP
II was also identified from the FCRs and presented for validation to the stakeholders workshop
at 15 below.
8. Drafting of final report including narrative based on FCRs including the emerging priorities
for PEP II.
9. Assessment of the major lessons learnt for ongoing/future design and implementation of
performance enhancement programmes. This included the priorities for design of such
programmes; types of framework(s) to guide programmes; modalities for planning and
operations, including M&E; and approaches and processes for capacity development. The
lesson arising from PEP was fed into into strategic recommendations for PEP II.
OPERATIONALISATION OF PEP II (FIELD PHASE)
10. Briefing meeting with Reference Group
11. Meetings with key stakeholders
12. Field Visits (3 days)
13. Drafting by team of provisional recommendations for PEP II
14. Debriefing of the Reference Group
15. FCRs was presented to a half-day Stakeholder Validation Workshop, broadly representative
of key principals and partners of PEP to discuss, obtain feedback on, and finally validate the
FCRs, including key lessons and recommendations for PEP II. The team was advised by the
Reference Group on stakeholder representation at the workshop.
REPORT FINALISATION
51
16. Following the completion of steps 1 – 15 above, including feedback from the Stakeholder
Validation Workshop and approval of recommendations for PEP II; and a final debriefing
meeting with the Reference Group, the draft final report of the evaluation was prepared by
the team within two weeks of the stakeholder validation workshop.
4. Guiding Evaluation Criteria
As per DAC OECD guidelines for programme and project evaluations, the following five criteria
guided the overall process:
Relevance – appraise congruence between the perception of what is needed as
proposed by the programme planners, and the reality of what was needed based on the
demands and economic circumstances of beneficiaries, government priorities and the
prevailing donor strategy. The intended outputs and outcomes of PEP was also be
assessed in terms of cross-cutting issues on gender, HIV/AIDS and environmental
sustainability.
Effectiveness – measure the extent of progress towards outputs or outcomes that has
been achieved. This includes three steps;
Measuring change in the observed output or outcome (i.e. progress made
towards changes to be observed for outputs or outcomes).
Attributing observed changes in output or outcomes to progress made in the
implementation of programme activities.
Judging the value of the change for the component beneficiaries – principally
staff in MoA & MFL.
Efficiency – measures how economically resources are being converted to results to
date. The focus is on exploring the extent to which resources are being used to produce
the intended outputs and if/how the resources available could have been used more
efficiently to achieve the intended results.
Impact – assesses the progress in changes (positive or negative, intended or unintended,
direct or indirect) brought about by the programme in counterpart capacity
development and changes in people’s well-being. Specifically, impact assessments
dwelled on what and how much change occurred at implementation level that is
attributable to the programme?
52
Sustainability – assessed the capacity to which present relevant social, economic,
institutional and other factors/players present are able to maintain, manage and ensure
project results endure into the future.
5. Theory of Change
The Theory of Change (ToC) was used to analyze the intervention logic of the PEP programme
and its translation into the logical framework, by exploring the linkages between the PEP
activities and the programme results. This is based on the understanding that an approach such
as the ToC – or one based on similar principles and logic – was used during the programme
design in identifying programme-level outputs, outcomes and impacts and linking these to the
activities under each of the four result areas under PEP: Change Management and Service
Delivery; Sector Policy, Planning and Financial Management; Human Resources Management
and ICT; Monitoring and Evaluation. The ToC is also used as a framework for analyzing the
linkages between PEP programme activities, the logical framework (log-frame) and PEP’s
Monitoring and Evaluation (M&E) system, all of which are built around a common set of
outputs, outcomes and impacts. This approach ensures a logical and systematic analysis of
cause-effect relationships in assessing and evaluating the programme.
The logic embedded in the ToC as outlined above is represented graphically in Figure 1.
Figure 1: A Generalized Theory of Change for PEP.
53
Sources: Patrizi, H and Patton, M.Q. (2010) “New Directions for Evaluation”, Evaluating Strategy, 128. Clark, H. (2004) Deciding the Scope of a Theory of Change, New York: ActKnowledge.
DIFFICULTIES ENCOUNTERED AND METHODOLOGICAL LIMITATIONS
The adoption of multiple evaluation approaches and tools was designed to improve on
analytical rigor, validation, accuracy and the reliability of collected information. As such,
reviews of literature from and related to the PEP programme, key informant interviews, focus
group discussions with secondary beneficiaries and the validation workshop were used to
ensure content validity, encompassing guidance on theoretical, conceptual and empirical
insights. Detailed discussions of the evaluation tools during the Inception Phase and their
subsequent revisions were also designed to improve on the reliability of collected information.
However, where challenges were encountered this was due to the adoption or use of the case
study in selecting provinces and the districts to visit during the Field Phase of the final
evaluation. For various reasons, it was unfeasible to conduct several case studies to allow for
greater generalization. Thus the generalization of findings might be an issue for debate. The
second phase of the assignment, i.e. operationalization of PEP II, was also conducted within a
“time squeeze”. This made it difficult for the consultants to undertake another field visit. The
consultancy team, however, used the opportunities presented by the initial key informant
interviews and field visit, to make search for insights for PEP II.
____________________________________
54
DATA COLLECTION TOOLS
PEP Final Evaluation - Evaluation Matrix based on Mission ToRs & DAC-OECD Criteria
Key evaluation areas Focal questions Data collection methods and sources of information
a) Relevance Core Evaluation Questions To what extent was the project design appropriate for the then
MAL's institutional CD needs? Have these needs changed since the mid-point of the project? What amendments in the concept, theory or implementation
modality should be made for a future phase of the project? Were the project institutional arrangements and mechanisms
relevant and have they contributed to increased ownership by the then MAL?
What changes should be considered for future interventions?
Literature Review (Programme Proposal, Project Estimates, etc.)
KIIs. FGDs.
b) Effectiveness Core Evaluation Questions To what extent have the Ministry's capacity and performance
been enhanced? Has flexibility contributed to the achievement of the purpose
or rather contributed to hindering it? Were the project's design and approaches conducive to
increased effectiveness? To what extent have the policies and guidelines developed on
HIV/AIDS and gender been implemented and/or mainstreamed (as recommended by the MTR)?
Literature Review (log frame, baseline & MTR reports)
HH Questionnaire Survey. KIIs. FGDs. Success stories.
55
c) Efficiency Core Evaluation Questions Have the project implementation arrangements been
efficient? How should they be replicated or amended under PEP II ? To what extent have partnerships with other cooperating
partners contributed to the project performance? To what extent have lessons and recommendations from the
MTE been implemented and/or followed up?
KIIs FGDs Budget reviews Review of instruments for
partnerships Strategies and reports subsequent to
the MTE.
d) Impact Core Evaluation Questions What impact (intended, unintended, positive or negative,
direct or indirect) has the project had on the MAL's capacity and performance?
Which of these effects have been observed or experienced by the indirect beneficiaries i.e. Zambia's small-scale male and female farmers?
What could have been done differently for the project to achieve wider impact?
KIIs FGDs Beneficiary Assessment(s) HH Questionnaire Survey Case Studies Capability + Skills + Results
(performance appraisals, management actions, meeting reports + circulars).
Staff Surveys Stakeholder Feedback Perception Studies
e) Sustainability Core Evaluation Questions To what extent has the support provided generated effects
that can economically and institutionally sustainable?
KIIs FGDs
56
CHECKLIST: Key Learning Outcome Areas and Operational Attributes to Assess Staff Performance Improvement in MOA & MFL.
1. Raised Awareness
Attitude
Confidence
Intention to act
Motivation (including hygiene and motivation/satisfier factors) 2. Enhanced Knowledge and Skills
Acquisition of new knowledge e.g. leadership, management, ICT, M&E.
Application of new knowledge
Improvement in understanding on knowledge needs and development of proficiencies (incumbents and colleagues)
Training, coaching and mentoring 3. Improved Consensus and Teamwork (Workplace Colleagues & with other Depts.)
Communication
Coordination
Contributions
Team leadership (task & process orientation and rotation)
Cohesion 4. Strengthened Coalitions (Across Units/Depts. and for implementation/collaboration with Cooperating Partners)
Common areas/agendas for action
Commitment to act
Trust
Leveraging diversities (for creativity & synergies)
57
5. Enhanced Networks
Common interest
Processes for collaboration
Incentives for participation
Generating new partnerships or coalitions 6. New Implementation Know-How
Evidence in formulated plans, strategies and operations at unit/dept. levels
Evidence in implemented plans, strategies and operations at unit/dept. levels Sources: EU Toolkit for Capacity Development, 2010. UNDP: Capacity Development Guides, 2004- 2009. World Bank Institute: Capacity Development Results Framework, 2009.
58
Annex 2: List of Persons/Organizations Consulted
Name Position Organization Telephone/Cell Number
EUD and PEP Reference Group
Matteo Sirtori Head of Section (Agriculture) EU Delegation + 260 211 250 711 [email protected]
Marion Michaud Advisor (Agriculture & Rural Development)
EU Delegation + 260 971 719 734 [email protected]
Friedrich Mahler Advisor (Agriculture & Rural Development)
EU Delegation +260 211 251 140 [email protected]
James McNulty M&E (Agriculture & Rural Development)
EU Delegation + 260 211 255 585 [email protected]
Bazak Zakeyo Lungu Agricultural Advisor National Authorizing Office + 260 977 329 765 [email protected]
Katupa Chongo Chief Agric Extension Officer Ministry of Agriculture + 260 950 769 412 [email protected]
Kondwani Gondwe Senior Planner Ministry of Fisheries & Livestock + 260 974 512 366 [email protected]
Mary Chilala Principal Economist (Planning and M&E)
Ministry of Agriculture +260 974 335 151 [email protected]
MOA & MFL National Offices
Dr. Julius Shawa Permanent Secretary Ministry of Agriculture +260 211 254 645 [email protected]
Ms Emma Malawo Director Dept of Policy & Planning +260 977 545 309 [email protected] Lilian C. Chomba Chief Planner Dept of Policy & Planning +260 977 598 983 [email protected]
Mwangala S. Chizongo
Senior HRM Officer HRA Department +260 977 700 896 [email protected]
Jane Samuel Phiri Assistant Director HRA Department - [email protected]
Joyce Nyama HRA Director Ministry of Agriculture +260 977 833 265 [email protected] Henry Mgomba Principal Agricultural Officer Ministry of Agriculture +260 977 246 661 - Karen Mukuka Chief Food & Nutrition Officer Ministry of Agriculture +260 977 383 275 [email protected] Timothy Tonga Principal Budget Analyst Ministry of Fisheries & Livestock +260 979 523 261 [email protected] Lois Mulube HRA Assistant Director Ministry of Fisheries & Livestock +260 966 955 765 546 [email protected] Esnat Nyangu Chief HRA Officer Ministry of Fisheries & Livestock +260 977 482 253 [email protected]
59
Chitalu Zimba Principal Statistician MOA & MFL +260 211 250 532 [email protected] Benson Mwenya Director Department of Livestock
Development +260 973 444 086 [email protected]
Derrick Sikombe Deputy Director Dept Policy & Planning +260 977 147 230 [email protected] Dr Martin Muyunda Principal Extension
Methodologist Ministry of Agriculture +260 967 875 008 [email protected]
Cabinet Office (Office of the President) Alfred S. Sakwiya Director Decentralization Secretariat +260 976 364 562 [email protected]
Southern Province
Charles Lwango Provincial Fisheries and Livestock Coordinator
Ministry of Fisheries & Livestock +260 977 413 270 [email protected]
Wakung’uma Mukumbuta
Provincial Senior Livestock Production Officer
Ministry of Fisheries & Livestock +260 955 939 990 [email protected]
Evans Nkhoma Provincial Agricultural Planner Ministry of Agriculture +260 954 741 204 [email protected]
Max Choombe PACO Ministry of Agriculture +260 977 795 652 [email protected]
Zandonda Tembo Senior Marketing Officer Ministry of Agriculture +260 977 466 705 [email protected]
Charity Chabaya HRA Executive Officer Ministry of Agriculture
Fundi Banda Senior Extension Methodologies officer
Ministry of Agriculture +260 968 732 699 [email protected]
Choma District
John Kaira District Fisheries Officer Ministry of Fisheries & Livestock +260 979 483 172 [email protected]
Limose Shamvu District Livestock Technician Ministry of Fisheries & Livestock +260 976 696 415 [email protected]
Dr. Jacob Bothma District Senior Livestock Officer Ministry of Fisheries & Livestock +260 977 342 898 [email protected]
Robert Tembo District Agricultural Coordinator (DACO)
Ministry of Agriculture +260 977 577 976 [email protected]
Sharald Malilwe Block Extension Officer Ministry of Agriculture +260 977 543 900 [email protected]
Anastacia Mulamfu Camp Extension Officer Ministry of Agriculture +260 977 425 783 [email protected]
Gwenny Nanyangwe Camp Extension Officer Ministry of Agriculture +260 976 734 674 -
Tommy Zimba Camp Extension Officer Ministry of Agriculture +260 976 356 816 [email protected]
60
Racheal Hangwemu Camp Extension Officer Ministry of Agriculture +260 967 493 187 [email protected]
Charity Hazemba Camp Extension Officer Ministry of Agriculture +260 968 783 999 -
Lusaka Province
Dr. Linous Munsimbwe
Provincial Agricultural Coordinator (PACO)
Ministry of Agriculture - [email protected]
Cathrine Kamushila HRA Executive Officer Ministry of Agriculture +260 977 601 715 -
Pascal Chipasha Principal Agricultural Officer Ministry of Agriculture +260 977 406 700 [email protected]
Dereck Chimangu Provincial Agricultural Planner Ministry of Agriculture +260 977 851 481 [email protected]
Kafue District
Kapwepwe Nthele Senior Agriculture Officer Ministry of Agriculture +260 978 920 845 [email protected]
Aggrey Chipuluka Block Extension Officer Ministry of Agriculture +260 979 371 977 [email protected]
Clara Chiluba Camp Extension Officer Ministry of Agriculture +260 979 465 150 [email protected]
Jacobina Mulenga Camp Extension Officer Ministry of Agriculture +260 979 616 026 [email protected]
Dr Perfecto Buyamba Kabanshi
District Fisheries and livestock Coordinator
Ministry of Fisheries & Livestock +260 977 820 964 [email protected]
Kasaka Fisheries Training Centre
Caiphas Mpaka Principal Ministry of Fisheries & Livestock +260 977 291 913 [email protected]
Libanga Ochola Training Officer Ministry of Fisheries & Livestock +260 975 453 131 [email protected]
Justina Kasabila Training Officer Ministry of Fisheries & Livestock +260 975 383 700 [email protected]
Zambia College of Agriculture (ZCA) – Monze
Bornwell Mupeyo Acting Vice Principal ZCA +260 971 413 105 [email protected]
Dr. C. Namwobe HOD - crops ZCA +260 978 746 766 [email protected]
Zambia Institute of Animal Health (ZIAH) – Mazabuka
Arnold Mulenga Acting Principal ZIAH +260 975 334 527 [email protected]
Dr. Webby Acting Vice Principal ZIAH +260 977 600 427 [email protected]
61
Chiobomba
Zambia Agricultural Research Institute (ZARI) & Seed Control and Certification Institute (SCCI)
Moses Mwale Director ZARI +260 966 766 395 [email protected]
Monde Siyandwa Zulu
Deputy Director ZARI +260 977 848 092 [email protected]
Mable M. Simwanza Director SCCI +260 977 783 943 [email protected]
NAO & PEP Technical Assistance Team
Jim Parker Team Leader TA Team PEP/ PMTC Zambia Limited +260 211 250 495 [email protected]
Bursch Nketani HRM Consultant PEP +260 977 173 474 [email protected]
Kanyifwa Mvulu Programme Accountant NAO/PEP +260 977 781 987 [email protected]
Cooperating Partners, Consultants & Other Stakeholders
Martin Liywalii S3P Programme Manager IFAD +260 658 615 571 [email protected]
Junji Takahashi Chief Adviser – Rice Dissemination Project
JICA +260 968 830 140 [email protected]
Tokutaro Lino National Project Coordinator - Rice Dissemination Project
JICA +260 972 391 161 [email protected]
Goichi Sasaki Extension/Training Officer – Rice Dissemination Project
JICA +260 977 791 385 [email protected]
Nigel Nicholson Nutrition Consultant Independent +33 553 918 460 [email protected]
Sam Mwaura M&E Consultant Independent +260 977 857 341 [email protected] Masiye Nawiko Executive Director Agricultural Consultative Forum +260 966 455 696 [email protected]
Smallholder Farmers FGD Participants (Mbabala Agricultural Camp – Choma District)
Livious Hampwaye Chairperson Mbalala Camp Cooperatives +260 965 437 015 -
Jennifer Munsaka Treasurer Mbalala Camp Cooperatives +260 972 056 244 -
Winslow Chabota Chairperson Mbalala Camp Cooperatives +260 969 554 607 -
Orally Hampwato Secretary Mbalala Camp Cooperatives +260 967 917 011 -
Royo Muntanga Member Mbalala Camp Cooperatives - -
Rosemary Mugimba Member Mbalala Camp Cooperatives - -
62
Phillip Sikalemezya Chairman Mbalala Camp Cooperatives +260 974 957 082 -
Deciah Mweeta Member Mbalala Camp Cooperatives - -
Babie Muntanga Member Mbalala Camp Cooperatives - -
Centros Munkombwe Secretary Mbalala Camp Cooperatives +260 959 203 057 -
Berry Sibalwa Chairperson Mbalala Camp Cooperatives +260 974 865 296 -
63
Annex 3: Literature & Documentation Reviewed To date the following documents have been fully or partially reviewed1:- 1. Vision 2030.
2. Revised Sixth National Development Plan (R-SNDP) 2013-2016.
3. National Agriculture Policy (NAP) 2004 – 2015.
4. Comprehensive African Agriculture Development Programme - Zambia Compact.
5. National Agriculture Investment Plan (NAIP) 2014-2018.
6. Draft Strategic Plan of the Ministry of Agriculture and Livestock, 2013-2016.
7. PEP Proposal, Financing Agreement (FED/2010/022-057), Start Up Programme Estimate and Programme Estimates no. 1, 2 and 3.
8. PEP Logical Framework – 18 July 2013 + December 2014.
9. Project deliverables including thematic studies, training materials, policy briefs, etc.
10. PEP Training Reports.
11. Results-Oriented Monitoring (ROM) Reports and Background Conclusions Sheets, 2012 and 2013
12. Participatory Review of the Ministry of Agriculture and Co-operatives Performance [...]", including an "Assessment of Current MACO Capacity, Capacity Development Needs and Challenges" and a "Strategy and Roadmap for a Performance Enhancement Programme in MACO/GRZ", Cardno Agrisystems Ltd, July 2009.
13. PEP Mid-term Evaluation Report – September 2014.
14. PEP Final Report – February 2016.
15. Activity and lessons Learnt Report.
16. PEP Quarterly Progress Reports.
17. PEP Meeting Minutes.
18. Participatory Extension Approaches (PEA) Assessment & Harmonization Study – July 2015.
19. National Agricultural Extension and Advisory Services Strategy (NAEASS) 2016.
20. HIV & AIDS and Wellness Policy for Agriculture Sector Document – February 2014.
21. Agriculture Sector M&E Inception Phase Synthesis Note – Sam Mwaura.
22. PEP II Project Fiche.
1 List of literature and documents reviewed over time.