imf central america- panama-dominican republic regional ... · imf central america-...

IMF Central America- Panama-Dominican Republic Regional Technical Assistance Center (CAPTAC-DR)

Independent Mid-Term Evaluation Phase I: July 2009 to date

Volume I: Final Report November 2013

Consultants Robert Woodbridge Subhrendu Chatterji Pablo Lo Moro James Hanson Pedro Belli Alejandro Garcia Caballos

Independent Interim Evaluation of CAPTAC-DR: Volume I: Main Report

ii | P a g e

ACKNOWLEDGEMENTS We have found this work on evaluating CAPTAC-DR extremely interesting. We have been very impressed by the very helpful and highly professional approach taken by the IMF team in this evaluation exercise both at CAPTAC-DR and at headquarters and in the high quality of the Center’s advisory services. We thank the IMF, especially Ugo Fasano and his team of resident advisors at CAPTAC-DR and Katarzyna Kardas and her colleagues in the Institute of Capacity Development, Global Partnerships Division. We are also very grateful for the support and comments from the Project Managers and Backstoppers at the Fiscal Affairs Department, the Monetary and Capital Markets Department and the Statistics Department, as well as from the team at the Western Hemisphere Department and at the Finance Department. Finally, we appreciate the support from the many individuals who helped us during our four case studies both at CAPTAC-DR and in the field.


iii | P a g e

Contents

EXECUTIVE SUMMARY ................................................................................................................. i

1. PURPOSE OF EVALUATION, METHODOLOGY AND SCOPE .................................................... 1

1.1 Objective of the Evaluation ........................................................................................................... 1

1.2 Overall Approach and Scope ......................................................................................................... 1

1.3 Evaluation Criteria and Rating System .......................................................................................... 2

1.4 Evaluating TA Projects and Workshops ........................................................................................ 3

1.5 Case Studies .................................................................................................................................. 4

1.6 Evaluation Approach for Strategy, Operations, RBM and Governance ........................................ 6

1.7 Limitations and Other Issues......................................................................................................... 6

2. CAPTAC-DR BACKGROUND AND CONTEXT ........................................................................... 8

2.1 CAPTAC-DR in Context .................................................................................................................. 8

2.2 Current Landscape of CAPTAC-DR Activities ............................................................................... 10

3. EVALUATION OF CAPTAC-DR AT PROGRAM LEVEL ............................................................ 11

3.1 Introduction and Summary of Results ........................................................................................ 11

3.2 Approach for Evaluating CAPTAC-DR Outcomes ........................................................................ 11

3.3 Ratings from Case Studies and DeskTop Reviews ....................................................................... 12

3.4 Portfolio Analysis by Type of Project .......................................................................................... 14

3.5 Feedback from Surveys ............................................................................................................... 15

3.6 Key Evaluation Questions ........................................................................................................... 16

4. EVALUATION RESULTS FOR TA CASE STUDIES AND DESKTOP REVIEWS .......................... 19

4.1 Overview ..................................................................................................................................... 19

4.2 Key Findings ................................................................................................................................ 19

4.3 Case Study: Costa Rica – Customs Administration ..................................................................... 21

4.4 Case Study: Cross Border Consolidated Supervision (Regional) ................................................. 22

4.5 Case Study: Operational Risk Supervision (Regional) ................................................................. 23

4.6 Case Study: El Salvador – External Sector Statistics .................................................................... 24

4.7 Desktop Reviews ......................................................................................................................... 25

4.8 Highlights of Results from Survey of TA Beneficiaries ................................................................ 27


iv | P a g e

5. EVALUATION RESULTS FOR REGIONAL WORKSHOPS ........................................................ 31

5.1 Overview ..................................................................................................................................... 31

5.2 Review of Selected Workshops ................................................................................................... 32

5.3 Results of Workshop Participant Survey ..................................................................................... 33

5.4 Evaluation KEQs and Ratings....................................................................................................... 37

6. ASSESSMENT OF CAPTAC-DR STRATEGY .......................................................................... 39

6.1 Introduction and Overview ......................................................................................................... 39

6.2 Alignment of CAPTAC-DR and SC Objectives for the Center....................................................... 39

6.3 Achieving the Advantages of CAPTAC-DR’s TA Delivery Approach ............................................ 40

6.4 Program Delivery Compared with Annual Work Plan ................................................................. 41

6.5 Implementation Challenges ........................................................................................................ 44

6.6 Regional vs Country-Focused TA and Training ............................................................................ 45

6.7 Role of National Authorities........................................................................................................ 46

6.8 Coordination with Other IMF Activities and Donors/IFIs............................................................ 47

6.9 Capacity Building ......................................................................................................................... 49

6.10 Comments on Overall Portfolio Approach .................................................................................. 49

6.11 Strategy Development for the Next Phase ................................................................................. 50

7. CAPTAC-DR FINANCIAL REPORTING .................................................................................. 51

7.1 Introduction and Summary ......................................................................................................... 51

7.2 Distribution of Trust Fund expenses ........................................................................................... 51

7.3 Analysis of Actual vs Budget Costs .............................................................................................. 52

7.4 Operational and Funding Issues Relating to Financial Reporting ............................................... 54

8. ASSESSMENT OF CAPTAC-DR OPERATIONS ...................................................................... 55

8.1 Overview ..................................................................................................................................... 55

8.2 Development of the Work Plan .................................................................................................. 55

8.3 Scale and Resource Mix .............................................................................................................. 58

8.4 Review of the CAPTAC-DR Center Coordinator’s Role ................................................................ 59

8.5 Resident Advisors ........................................................................................................................ 61

8.6 Short-Term Experts ..................................................................................................................... 62

8.7 Leveraging RA Time ..................................................................................................................... 63

8.8 Role of IMF HQ Departments ...................................................................................................... 63

8.9 Internal Reporting ....................................................................................................................... 65


v | P a g e

8.10 CAPTAC-DR’s Overall Project and Program Management .......................................................... 65

8.11 Professional Support for Center Coordinator & Resident Advisors ............................................ 67

9. REVIEW OF RBM IMPLEMENTATION AND CAPTAC-DR LOGICAL FRAMEWORKS ................. 68

9.1 Introduction ................................................................................................................................ 68

9.2 Status of IMF RBM Introduction ................................................................................................. 68

9.3 Comments on IMF RBM Framework ........................................................................................... 69

9.4 Illustrative Reporting Framework for CAPTAC-DR ...................................................................... 71

9.5 The Project Management Cycle .................................................................................................. 72

9.6 Comments on RBM Implementation and LOGFRAMES at CAPTAC-DR ...................................... 73

10. COMMENTS ON CAPTAC-DR GOVERNANCE ........................................................................ 76


10.2 CAPTAC-DR and Steering Committee Survey Responses/Objectives ......................................... 76

10.3 Steering Committee Role ............................................................................................................ 77

10.4 Observations on Steering Committee Meetings ......................................................................... 79

10.5 CAPTAC-DR Reporting to the Steering Committee ..................................................................... 80

10.6 Sustainability of the CAPTAC-DR Funding Model ....................................................................... 81

11. RECOMMENDATIONS .......................................................................................................... 83


11.2 Summary of Recommendations .................................................................................................. 84

11.3 Background to and Details of Recommendations ...................................................................... 87

11.3.1 Development and Implementation of CAPTAC-DR’s Strategy ................................................ 87

11.3.2 Strengthening CAPTAC-DR’s Monitoring and Reporting......................................................... 89

11.3.3 Improving CAPTAC-DR’s Operations ....................................................................................... 90

11.3.4 Developing a More Results-Oriented TA Approach ................................................................ 91

11.3.5 Enhancing CAPTAC-DR TA’s Sustainability .............................................................................. 92

11.3.6 Improving CAPTAC-DR’s Cost-Effectiveness ........................................................................... 92

11.3.7 Enhancing the Effectiveness of RBM Implementation ........................................................... 93

11.3.8 Increasing Standardization of Practices and Information Sharing Across RTACs ................... 95

11.3.9 Improving CAPTAC-DR’s Governance ..................................................................................... 95

11.3.10 Enhancing Effectiveness of CAPTAC-DR’s Workshops and Training ................................... 96

11.3.11 Implementation of Recommendations ............................................................................... 96


vi | P a g e

ABBREVIATIONS AND ACRONYMS ............................................................................................. 97

INDEX OF TABLES AND CHARTS ............................................................................................... 99

APPENDICES ........................................................................................................................... 101

I. Terms of Reference for the Evaluation ..................................................................................... 102

II. Responses to TORs Questions in the Report ............................................................................. 110

III. Evaluation Tools and Major Task Groups ................................................................................. 112

IV. Key Evaluation Questions ......................................................................................................... 115

V. List of Meetings and Interviews ................................................................................................ 119


i | P a g e

EXECUTIVE SUMMARY Overview CAPTAC-DR’s overall Evaluation results are very strong, ranging from Excellent for relevance to Very Good for the other rating criteria.1 CAPTAC-DR has achieved a great deal in its first three years of operations. Buy-in has been demonstrated at senior level from the country members and regional organizations. There has been regular collaboration and consultation with other technical assistance (‘TA’) providers in the region. Working arrangements are functioning well, with the Western Hemisphere Department (‘WHD’) guiding medium-term strategy for the region. This work is combined with the diagnostic assessments and backstopping at topical level from the IMF’s TA departments, as well as with active consultations by CAPTAC-DR with its members to determine TA implementation needs and project demand. This teamwork has enabled the Center to generate a highly focused series of TA projects supported by related regional workshops and seminars. The Centre has also been taking proactive approach to focus on areas which maximize the impact of its interventions. Our survey of TA recipients has yielded very positive feedback on the Center’s quality of advice, results, speed of response and the IMF’s partnership approach. At this stage, the main assessable results have been at the output level. It is likely to take a few more years, with some exceptions, to clearly demonstrate medium- to long-term outcomes and sustainability of the TA. Nonetheless we believe that the ‘direction of travel’ looks positive. We recognize that CAPTAC-DR has from the outset focused on reporting results and placing a strong emphasis on the transparency of its operations through proactive consultations with stakeholders, as well as through its website and its regular newsletters. We have made a series of suggestions on improvements to implementation of Results-Based Management (‘RBM’), particularly on how project2 and program3 design as well as management and monitoring could be strengthened further to deliver strong and measurable results at the outcome level. In addition, we have made recommendations relating to CAPTAC-DR’s strategy, operations and governance that the IMF and the Steering Committee (SC) may wish to consider. Purpose of the Evaluation and Summary of the Evaluation Methodology The purpose of the Evaluation is to assess the extent to which: CAPTAC-DR is achieving its objectives in an efficient and effective manner, and The TA and training delivered by the Center is sustainable.

1 As per our methodology, scores from 2.5 to 3.5 (out of four) are considered Good. All of the CAPTAC-DR level scores in are at 3.0 or above. 2 Defined as an intervention or a set of interventions aimed at achieving an objective or set of closely related objectives (e.g. introducing National Account Statistics, or drafting capital adequacy regulations). 3 Defined as a set of closely-related projects aimed at achieving a higher level objective or set of higher level objectives (e.g. the Center’s PFM projects in a country, or the CAPTAC-DR program as a whole).


ii | P a g e

We carried out the Evaluation by means of: In-depth study of nine TA projects, including four detailed case studies that required country visits. The coverage by

topical area was: three projects under Revenue Administration; three under monetary operations and financial sector supervision; one under Public Finance Management (‘PFM’) and two under statistics.

Desk-based research on eight training workshops delivered, or sponsored by CAPTAC-DR. Interviews with all relevant departments at IMF HQ. Detailed review of published and internal reports. Field trips to Guatemala to interview the Center’s coordinator, administrative staff and resident advisors and to Costa

Rica, El Salvador and Guatemala to conduct our four case studies. Online surveys of beneficiaries of the TA projects, participants in the eight selected workshops and SC members. Background and Context CAPTAC-DR is a recently established IMF Regional Technical Assistance Center (‘RTAC’) with its launch taking place in July 2009, as compared to five of the other eight operating RTACs which started operations between 1993 (Pacific) and 2004 (Middle East). In common with the other RTACs, CAPTAC-DR is a multi-donor trust fund managed by the IMF with funds totaling US$35.6 million pledged by Canada, the Central American Bank of Economic Integration (‘BCEI’), the European Commission (‘EC’), Germany (through GiZ), the Inter-American Development Bank (‘IADB’), the IMF, Mexico, Spain as well as seven member countries. The current phase is expected to last five years. There is a general move towards greater regional economic integration and a stronger momentum towards regional harmonization of standards and practices within the CAPTAC-DR region focused on Central America and the Dominican Republic, although the pace varies depending on the sector. To this end several regional organizations have been established, namely: the Central American Monetary Council (‘CAMC’); the Central American Council of Supervisors of Banks, Insurance and Other Financial Institutions (‘CCSBSO’); and the Central American Council of Treasury and Finance Ministers (‘COSEFIN’). One of CAPTAC-DR’s central missions is to support regional integration and harmonization at both the country and the regional level. The Center’s seven member countries share many characteristics but also exhibit economic diversity. For example, in 2011 GDP per capita ranged from US$8,600 (Costa Rica) to US$1,600 (Nicaragua). GDP growth varied from 10.6 percent (Panama) to 1.5 percent (El Salvador). Inflation is reasonably subdued, at single digit in all cases while fiscal deficits do not exceed 3.5 percent. Dollarization of the banking system is a feature of all seven economies, with the lowest level at 32 percent in Guatemala and Honduras (whereas two of the countries, namely Panama and El Salvador, have adopted the US dollar as their national currency). The global financial crisis set back the pace of reform but growth has returned since 2011. The IMF World Economic Outlook of October 2012 forecasts average annual GDP growth for 2013-17 for all seven countries (2.5 percent for El Salvador, 3-4 percent for Guatemala and Honduras, four percent for Costa Rica, the Dominican Republic and Nicaragua, as well as six percent for Panama).


iii | P a g e

CAPTAC-DR Level Evaluation Results The Center-level evaluation results are summarized below. These results are very strong in our view, especially given that CAPTAC-DR has only been in operation for four years. Many of the projects in the portfolio are multi-year in length. CAPTAC-DR’s reports, unlike those for say West AFRITAC (‘AFW’), do not clearly distinguish whether projects are short or long-term. We therefore performed our own analysis to arrive at an acceptable approximation. Multi-year projects are more likely to produce significant outcomes over the medium-term. Of our nine case studies and desktop samples reviewed, we selected seven multi-year projects. Our methodology for arriving at the aggregate Center-level ratings above was based on: Consolidating the ratings from the nine case and desk sample reviews with a weighting of 60/40 for the case studies;

because these were analyzed in greater depth through field trips; A higher-level review of all projects reported in 2011 and 2012, including discussions at the Center and HQ; and Analysis of responses to our surveys of TA recipients and workshop participants.

Our analysis of projects by duration indicated that 37 percent could be classified as short-term and 63 percent as multi-year. 26 percent of the projects were in statistics (mainly external and national accounts) and 37 percent were in other sectors. The latter were narrowly focused and regional in scope: large taxpayer units, risk-based systems for customs, single treasury accounts, cross-border consolidated supervision, and liquidity management. The survey responses supported the ratings: 86 percent agreed that the TA was supporting priority areas; 97 percent that the TA was clear, practical and implementable; and 90 percent felt that workshops were pitched at the right level. The workshops also attracted a 96 percent satisfaction rating. Results from Case Studies and Desk Top Reviews Eight of the TA projects selected had a regional dimension in that they were delivered in many, if not all, of the member countries. The CAPTAC-DR model is particularly suited to the regional approach given the Center’s location and the fact that its team of resident advisors (‘RAs’) is able to maintain close communication with their TA counterparts. The model is also well suited for sustained TA delivery in some of the more difficult topical areas, such as tax and customs administration, because the RAs (supported by short term experts (‘STXs’)) are better able to keep up the reform momentum and respond quickly to any setbacks. Some of the projects in selected countries were able to achieve measurable outcomes more easily. The CAPTAC-DR reports have summarized some of the success stories. Nevertheless, when assessing outcomes at this stage, we can mostly consider the “direction of travel”. This progress needs to be sustained. These studies have also informed some of our recommendations.


iv | P a g e

Evaluation Results of Workshops During the period 2010-2012 CAPTAC-DR delivered or arranged 56 regional workshops attended by 1,438 participants with an average of 26 participants per event. Our survey of workshop participants elicited very strong approval ratings for their quality and content. Most respondents were highly satisfied and indicated that the main benefits were an improvement in their theoretical knowledge and their knowledge of international best practice. In two of the technical areas an encouraging by-product was that a majority of the participants used the knowledge gained to develop action plans (risk-based customs administration 75 percent, and public sector accounting 67 percent). Some of the responses indicate possible

areas for improvement in the design and conduct of the workshops. For example, 70 percent of participants thought that the workshops needed to be longer and about one-third felt that the content was too theoretical and would benefit from more practical content such as real case studies. Assessment of CAPTAC-DR Strategy Overview of Strategy CAPTAC-DR’s Phase I strategy was succinctly phrased in its strategic logical framework: “The Strategic Objective: Improved institutional capacity of the beneficiary countries and strengthened financial and economic regional integration.”4 CAPTAC-DR plays an important part in the IMF’s holistic strategy to deliver demand-driven TA to its member countries. The IMF provides diagnostic and strategic TA from HQ (a medium-term framework from WHD and topical area strategy from the TA departments) and CAPTAC-DR uses largely local and regionally based expertise to deliver on-the-ground capacity building. Supported by IMF HQ, CAPTAC-DR has run a very clearly focused operation, helped by clarity of TA demand arising from its careful national consultations, but also regionally through cooperation with such bodies as CAMC and CCSBSO and strong early cooperation with IADB. CAPTAC-DR has developed in a short time a track-record of being a well-functioning and successful operation whose activities are owned by country authorities. During the Center’s first three years of operation TA has been delivered in most topical areas, with focus on a few longer-term themes, which we have mentioned earlier: large taxpayer units, risk-based customs procedures; single treasury accounts; liquidity management; consolidated cross border supervision in banking; capacity building in external and national accounts statistics; and so on. We believe this level of focus has been an effective way to work towards CAPTAC-DR’s strategic objective. At the same time CAPTAC-DR has retained the flexibility to respond to specific TA demands from its members, often through shorter-term interventions. The Resident Advisor (RA) role is a unique modality provided by CAPTAC-DR and the other RTACs. There are long-term advisors, as distinct from CAPTAC-DR’s resident advisors, operating in some areas and countries in the region whether employed by IMF, World Bank or other organizations, but these tend to be anchored in a particular area and country. The long-term resident advisor role is sometimes less suitable for certain TA purposes for a variety of reasons including the absorptive capacity of the beneficiary. CAPTAC-DR’s RAs step in at intervals appropriate to the case and maintain a role

4 CAPTAC-DR 2012 Annual Report.


v | P a g e

over an extended period. The model is particularly relevant to the regional context where the member countries share aspirations of regional integration and harmonization of practices. Having said the above, we believe that CAPTAC-DR needs to increase the range of its TA delivery modalities to improve effectiveness. SC Members’ Views on Strategy We carried out a survey of SC members. 80 percent of the donors considered that the maximization of the Center’s developmental impact was a very important priority (40 percent of member countries). All donors considered it very important (40 percent) or important (60 percent) to focus both on countries most in need and on countries with less access to other TA. All member countries either agreed with these assertions (80 percent) or strongly agreed (the remainder). Somewhat in contradiction though, the member countries also considered that equal distribution of TA is an important issue. All country officials agreed (60 percent) or strongly agreed (40 percent) that resources should be distributed equally among technical topics. 80 percent of the member countries also felt that support should be provided equally to all countries. There have been several comments from individuals interviewed conveying that donor coordination by CAPTAC-DR needs to be improved. Whilst there is always room for improvement, we believe that CAPTAC-DR has done a good job so far. We understand that for Phase II CAPTAC-DR plans to strengthen collaboration with other TA providers by building strategic working partnerships. The survey responses indicate that donor coordination continues to be regarded as a weak area by the donor members of the SC. One particular concern has been expressed over the timely release of mission reports especially to donors active in the same technical areas. Overall, we believe that CAPTAC-DR is well on track with its strategy. Nevertheless, budget areas should be reviewed and balances struck between HQ versus Center controls. Additionally, projects should be differentiated more clearly as between multi-year and short-term (less than one year). Challenges The main strategic challenges relate to (i) issues relating to building sustainable institutional capacity; (ii) funding of CAPTAC-DR; and (iii) funding and provision of TA in areas not covered by CAPTAC-DR. The institutional capacity issues largely relate to the ease of recruiting, training, motivating and retaining good caliber staff. This in turn requires strong and enlightened human resource management. This does not seem an issue for the central banks in the region but the Center Coordinator believes it is a more likely challenge in the areas of tax, customs and public finance management, as well as in selected statistics departments. We have noted that in some of CAPTAC-DR’s multi-year projects going forward, there is now a planned component for strengthening HR management, e.g. in tax and customs administration projects. CAPTAC-DR’s funding has been adequate so far in Phase I, with US$35.6 million committed. Spain had pledged 20 percent of this total but has now declined to participate in Phase II, the target for which has been set at US$38.9 million. Whilst there is a risk that CAPTAC-DR will not be able to fill the donor funding gap, we believe the risk will not be one of continuity but rather how and in what areas to cut back the program.


vi | P a g e

Provision of TA in topics not covered by CAPTAC-DR is important because some of the necessary interventions affect the effectiveness and sustainability of the Center’s contributions. The TA gap matters because many donors and IFIs have their own set of priorities either by topic and/or by country. On the other hand CAPTAC-DR’s remit is to cover all seven countries with a menu of topical areas, but with some limitations such as the funding of elements of IT (both hardware and software). Under a generic objective, such as strengthening tax administration capacity or banking supervision, a number of TA interventions may be needed. They could involve legislation, automation or strengthening of HR management, which CAPTAC-DR’s work plans do not always cover. In our case study on cross-border consolidated supervision, for example, Panama needs to draft and pass legislation to empower its supervisors to demand information from financial holding companies. CAPTAC-DR had planned no TA to address this issue nor reported on who else would cover it. Strategy Development for Phase II CAPTAC-DR has already set out its draft strategy for Phase II. There are several interesting strategic developments proposed: Encouragement of peer to peer learning e.g. Costa Rica’s experience with the Single Treasury Account; Forging of strategic TA partnerships e.g. with IADB, GiZ and the World Bank; A systematic outreach towards regional organizations and donor field offices; Consideration of fewer but more extensive projects; and Involving TA beneficiaries in the Results Based Management processes e.g. by agreeing on outcome indicators and

milestones. We strongly support all of the above initiatives. In addition we have some other suggestions: 1. Spinning out and separately identifying the multi-year TA projects would be a positive step forward, recognizing the

need for longer-term capacity building and providing a better framework for RBM.

2. There needs to be substantially more consideration given to the context within which CAPTAC-DR’s interventions, at the sector level in each country and at individual project level, are intended to be delivered. It is unlikely that CAPTAC-DR’s interventions will on their own deliver important outcomes. The relevant dependencies and associated risks need to be clearly assessed, documented and their potential impact on achieving outcomes analyzed, for example by way of results chains.

3. CAPTAC-DR seeks to support countries in implementing their macroeconomic and financial sector strategies.

Member countries do not have PRSPs and, as far as we know, the completeness of relevant strategies varies from country to country. Some of the regional organizations such as the CCSBSO and CAMC have a clear regional overview in their particular areas of expertise and the IMF TA department missions have made recommendations. What seems to be lacking in some cases is an overall macroeconomic management strategy for each country that highlights priorities, even though some may have benefited from FSAPs (covering the financial sector) that are not too out of date. We suggest consideration is given to assistance to the development of an overall macroeconomic strategy, which would help countries to see how particular TA fits their priorities. This would help firm up the strength of demand for TA if it can be seen within the context of a prioritized strategy and action plan.


vii | P a g e

4. The Center should attempt to plan its interventions over the medium-term and these plans (reflected in the Phase II Program), whilst they will vary from time to time as circumstance and TA demands change, should also be reflected in the Regional Strategy Note prepared by WHD.

5. Benchmark and learn where appropriate from the performance of other RTACs. There was a recommendation with

respect to the 2009 AFRITACs evaluations that ICDGP should develop some procedures manuals to guide the operations of all RTACs. We also recommend that success stories and workshop material could be shared among RTACs.

CAPTAC-DR Financial Reporting

Of the aggregate under-spend of about US$4.5 million in Phase I to-date, 97 percent arose in the first two years. During the 2012 fiscal year the Center was operating at almost 100 percent of that year’s annual budget. The financial reporting system has a number of shortcomings most of which can only be addressed by HQ. How HQ costs are allocated to CAPTAC-DR for items such as backstopping, project management and RA costs are not readily available to the Center Coordinator. Budget versus actual comparisons are insufficiently

detailed. Reconciling financial results against work plans is frustrated because the former are denominated in dollars, whereas the latter are expressed in person/time. Assessment of CAPTAC-DR Operations Overall CAPTAC-DR’s project and program management is strong for several reasons: RAs are usually highly experienced in their areas of expertise and are known and respected in the region by

beneficiary clients; and IMF HQ exercises quality control by requiring consultation and approval of terms of reference for each mission (the

briefing papers) and then quality and peer review after the end of each mission (mission reports). A key issue affecting our review of CAPTAC-DR’s operations relates to the implementation of an effective RBM system, and a greater focus on defining, delivering, monitoring and reporting results. CAPTAC-DR has made substantial progress in this area, though more needs to be done to deploy RBM at project level and to develop ‘smarter’ indicators. CAPTAC-DR is making effective use of STXs with an excellent leverage ratio. We believe this significantly enhances the pace and efficiency of TA delivery. The work plan process is comprehensive and effective but focuses on one year at a time even though many projects are multi-year. The Center Coordinator is largely engaged in administrative issues despite qualifications and experience which could be better deployed in stakeholder consultations (in addition to the extensive consultations held in developing the work plan with member countries) briefings and coordination. The RAs spend up to 20 percent of their time in mission preparation and reporting: Briefing Papers (BPs) and Back-to-Office Reports (BTOs) even for projects as short as one or two weeks. In addition they spend time on internal reporting. Backstopping and project management have been significantly underspent. This could be due to several reasons or a combination thereof: over-


viii | P a g e

budgeting, under- charging, the Center RAs and STXs not needing or receiving as much oversight, or resources were stretched at IMF HQ. In particular it appears to us that there is a variation between the TA departments in the rigor with which they allocate backstopping and project management charges to CAPTAC-DR. Review of Results Based Management Implementation and Logical Frameworks CAPTAC-DR, and the CC in particular, have taken considerable ownership of this initiative and have spent substantial effort in developing a results reporting system. The implementation of RBM was an objective referred to in the Program Document but CAPTAC-DR had to wait until FY2012 when the IMF had designed a logical framework system that would be common to all RTACs. In the spirit of RBM, CAPTAC-DR developed comprehensive results reporting from the outset and from FY2013 had identified indicators and milestones for its logframes. Substantial progress has been made to date, and our assessment is that the Center is providing meaningful information on the results being achieved on its TA activities. Nevertheless, substantial work remains to be done to complete the process, and a thorough reassessment of its objectives needs to be made. RBM needs to be seen and implemented as a management tool rather than just a reporting framework. Experience at other donors and IFIs suggests that strengthening and embedding this entire process may take a number of years. Comments on CAPTAC-DR Governance In general, there seems to be some lack of clarity on the definition of the SC’s role. In fact 37.5 percent of all respondents felt that the SC’s role and responsibilities are not clear (or expressed that they didn’t know). The 2009 Program Document describes SC’s function as “to provide strategic guidance and contribute to the setting of the Center’s priorities, including the endorsement of the Center’s work plans. It is expected that the Committee will look at broad country priorities and provide its views in assisting beneficiary countries in the core areas of its expertise”. So far the Minutes of SC meetings don’t record many comments on strategic issues, TA priorities or achievement of results. There is an issue with continuity. Only the Dominican Republic and Nicaragua have been represented by the same individual at three of the four meetings. There was no country where the same individual was present at all four meetings spanning the past three years. A slightly more volatile picture arises with the donor members: none of these members have managed continuity beyond two meetings and the IADB and BCIE have been represented by a different individual every time. The observer members have the best record, with CAMC fielding the same representative at each of the last three meetings. We believe there may be some drawbacks in the reporting because of the length of the reports, the shortage of time at SC meetings and the impact these factors could have on the SC’s governance priorities. There does not seem adequate time to discuss results as opposed to merely receiving a report. The results themselves are still presented in most cases in too generic a manner. There also seems to be little time to discuss strategic issues.


ix | P a g e

Summary of Recommendations The recommendations below are listed broadly in order of importance, both between and within each headline recommendation. Recommendation 1: A more detailed and costed strategy should be developed for the next phase, the implementation of which should be easy to monitor. This should be done through: Drafting a coherent, costed strategy for each sector in each country, rather than some illustrative projects. Using a

streamlined, country/sectoral results chain approach, they should reflect interventions that are appropriately sequenced; take into account other reform needs; provide assessment of other complementary donor programs and consideration for partnerships; and incorporate objectively verifiable milestones and indicators. To the extent that CAPTAC-DR needs to retain flexibility to address member country needs as they arise, say, 25 percent of the budgets and resources could be allocated for such needs.

Having a strategy that whilst consistent with the RSN, is capable of standing on its own. The Program and annual budgets being clearly built up from and linked to expected interventions and the work plans. Clearer linkages with country/sector strategies in member countries, especially in areas such as PFM. More detailed research into partner regional and donor institutions and their medium-term plans and incorporating

these into the strategy. Also, incorporating in the CAPTAC-DR strategy an action plan for donor coordination for the Center based on clearly defined expectations and objectives from the Steering Committee and member countries.

Differentiating more clearly between multi-year and short-term (less than one year) projects and monitoring the mix between them with the view of maximizing the use of the former.

Avoiding the possible temptation to cut back on difficult areas in favor of easier pickings in others. Developing a financial sustainability plan for CAPTAC-DR as part of the next phase strategy, should it be a medium or

long-term objective of the Steering Committee.

Recommendation 2: Strengthen CAPTAC-DR’s monitoring and reporting through the entire project and program cycle by: Introduction of a robust accounting package, tailored to and suitable for the use of all RTACs, to enable standardized

and comparable financial information, which can be consolidated easily. It should cover all financial information, including those generated by HQ. Ultimately, it should also be integrated with the RTACs’ operational work plans and activities.

Elevation of budget monitoring from the current accounting level to more of a business level. The CC should take overall charge of this, and information barriers need to be removed to ensure that the Coordinator has adequate breakdown of underlying costs to ensure he can discharge this role effectively.

Identifying information needs for each key user group and designing reports to meet their needs, with a focus on analyses and explanations, deviations from expectations, and exception reports. Appropriate systems to produce such reports should be developed.

Integrating the work plan and annual budget Until IMF systems are capable of producing them, development of a CAPTAC-DR-level system to budget and track

actual project costs. Providing in annual reports more detailed and qualitative financial analysis of actual spends and variations from

budgets and explanations thereof. Streamlining internal reporting.


x | P a g e

Implementing more robust processes in ICDGP, AFR and TA Departments to ensure all backstopping and project management costs are charged fully.

Recommendation 3: CAPTAC-DR’s operations should be streamlined and the efficiency of resource use improved through: Recruiting research assistants to support RAs. Extending the contract term of RAs beyond the current one-year. Removing barriers to IMF staff taking up RA positions in RTACs, such as the requirement to take unpaid leave. Improving efficiency of STX recruitment and use through inter alia using them for longer missions; targets for minimum

use of STXs; contracting them for projects or periods (rather than missions only); widening their recruitment pool; and giving RAs access to the STX database.

Freeing up more time of the CC on internal reporting, budgeting and the compilation of work plans to allow him to focus more on areas such as RBM, liaison with regional organizations, other stakeholders and donor coordination.

Recommendation 4: TA delivery should be improved and made more results-oriented through a combination of: Complementing the short-term mission based TA delivery approach with medium-term assistance. Incorporating implementation plans in all TA reports with clear milestones and clearly defined, verifiable outcomes. Establishing a system for ensuring that outcomes and milestones can be monitored effectively and timely action taken

to address slippages, outside the mission windows and after the project’s delivery has been completed. Encouraging the national authorities to play an active role in donor coordination and disseminate what is being done

and by whom. Installation of tools such as Aid Management Platform software created by the Development Gateway should be considered.

Recommendation 5: Enhance sustainability of CAPTAC-DR’s TA interventions through: Undertaking staff capacity assessment (numbers, ability) as part of the diagnostic phase of capacity building projects

and tailoring nature and length of TA appropriately. Where needed, improving TA absorptive capacity and sustainability in client institutions through implementation of HR

strategies (by CAPTAC-DR or partners) that improve staff quality and retention. Embedding medium and long-term projects into the host institution’s annual or long-term institutional plan or strategy. Taking steps to partner with regional and country training institutions to host/organize workshops, in order to build

sustainable local training capacity. Reviewing the current usage of capacity building tools such as attachments, training of trainers, use of regional STXs,

etc., the scope for increasing their use considered, and clear annual targets set for each tool. Recommendation 6: Improve CAPTAC-DR’s cost-effectiveness through: More accurately tracking budgets and costs (all costs, including overheads and HQ, particularly for projects);

identifying underlying causes of variances; and implementing a system to take quick corrective action. Extending mission durations and workshop lengths. Reassessing the numbers of CAPTAC-DR/IMF staff attending SC meetings. Ensuring the CC and RAs are focused on higher value-added roles by engaging appropriate support staff.


xi | P a g e

Recommendation 7: The effectiveness of the implementation of Results Based Management should be enhanced by extending it from primarily a reporting to a management tool through: Implementing the use of sector/country results chains as highlighted under Recommendation 1 to provide underlying

information for developing logframes. (i) extending the logical framework approach to the project level within each country; (ii) incorporating project level

input/cost information; (iii) improving the technical robustness of the logframes by ensuring that all projects/countries that are contributing to an outcome are accommodated in a logframe; (iv) incorporating relatively small interventions in a simplified format, rather than being excluded; and (v) placing more emphasis on monitoring milestones.

Drafting Briefing Papers on a project (rather than mission) basis, incorporating full project logframes including outcomes. Once information is available, such logframes should be augmented with: (i) TA advice implementation milestones and dates; (ii) TA timetable, including an outline of future missions in relation to milestones; (iii) project budget, in sufficient detail, broken down by mission; (iv) TA implementation risks identified during the mission and steps necessary by client (in implementation plan) or CAPTAC-DR (e.g. timing of mission, increasing STX support) to address them; and (v) proposed monitoring approach for the TA.

Systematically gathering lessons learnt and incorporating them in future interventions. Using the RBM framework as a TA management tool, rather than simply a reporting tool. Engaging a logframe specialist, either for CAPTAC-DR or say across the RTACs, to review logframe entries and

provide technical feedback to improve users’ practical understanding of their use. Recommendation 8: Improve standardization of practices, reporting and knowledge sharing between RTACs by: Carrying out a study of best practices across all RTACs, especially focused on RBM implementation, reporting, cost

effective approaches, and different components of project and program management. Establishing and a number of Key Performance Indicators to monitor the relative performance between the RTACs, in

addition to them serving as useful monitoring tools for each RTAC. Sharing TA reports and workshop presentations. Circulating to SC members executive summaries of the evaluation of other RTACs. Recommendation 9: The effectiveness of CAPTAC-DR’s governance arrangements should be enhanced through: Re-examining the RTAC organization and management structure again in the light of experience. The reporting lines

are in several different directions depending on whether considering a CC, an RA, an STX or local staff. Financial controls are equally diffuse. The role of the Steering Committee is diffused. In these circumstances it is difficult to pin down accountability, transparency or responsibility for success or failure.

Streamlining the size and reviewing the seniority of participants in the SC or exploring alternative models. Improving continuity of membership among SC members. Streamlining reporting, with a focus on analysis and explanation of variances from plan, rather than mainly raw data. Devoting more space in the SC meeting agendas to discuss and query results monitoring, discuss project results and

strategic matters. Introducing SC sub-committees (e.g. results and specific strategic issues) to enable the SC to assess results and

contribute to CAPTAC-DR strategy more effectively.


xii | P a g e

Recommendation 10: The effectiveness of workshops and training should be improved by further integrating them with CAPTAC-DR’s TA work, through: Budgeting adequate preparation time for, in particular, in-country workshops to develop tailored case studies. Develop a more structured approach to post-workshop follow-ups including where appropriate: (i) setting up a virtual

network of participants; (ii) providing follow-up training to the same participants on the same topic; (iii) setting up mentorship arrangements; (iv) providing on-the-job training; and (iv) organizing study tours.

Recommendation 11: To ensure the Evaluation’s recommendations are implemented: CAPTAC-DR management should provide a formal response on them. Other IMF departments should also comment. An implementation plan with quarterly milestones should be developed and presented to the Steering Committee. A Steering Committee sub-committee to guide and monitor implementation should be considered.


1 | P a g e

1. PURPOSE OF EVALUATION, METHODOLOGY AND SCOPE 1.1 OBJECTIVE OF THE EVALUATION The objective of the Evaluation is to assess the extent to which: The Central America, Panama and the Dominican Republic Regional Technical Assistance Center (‘CAPTAC-DR’) is

achieving its objective in an efficient and effective manner, and The TA and training delivered by the Center is sustainable. Additionally, the evaluation seeks to identify lessons learned for the governance, operations and implementation of CAPTAC-DR’s strategy, to improve TA design and delivery at project and program level. The Evaluation Terms of Reference (‘TORs’) expand on the purpose to ask the evaluators to consider whether CAPTAC-DR is achieving the advantages typically associated with the RTAC model: better response to country needs; rapid and flexible TA delivery; closer coordination with authorities; stronger country ownership; and better coordination with other TA providers in the region. The Evaluators also seeks to identify the main challenges and risks facing CAPTAC-DR and consider how these are being or could be dealt with. 1.2 OVERALL APPROACH AND SCOPE The Evaluation was guided by its ToRs, which set out inter alia its purpose, scope, methodology, and a set of questions that we needed to respond to. The ToRs are provided in Appendix I. Appendix II provides an index, referring to each of the issues required to be addressed by the ToRs and the relevant section in the Report where we have provided a response. Appendix III provides a table summarizing the key evaluation tools we have used. Our general approach for the Evaluation has been as follows: 1. Agree Evaluation rating criteria, Key Evaluation Questions (‘KEQs’), and methodology for evaluating both individual

projects and the CAPTAC-DR program with the Institute for Capacity Development Global Partnership (‘ICDGP’) and the Evaluation Sub-Committee, including appropriate adjustments to the ToRs.

2. Develop and agree with ICDGP and the Evaluation Sub-Committee a work plan to implement the agreed Evaluation methodology.

3. Implement the work plan, which included: Desk-based research focusing on nine TA projects, including four in-depth case studies that required country visits.

The coverage by topical area was: Three projects under Revenue Administration (FAD); three under monetary operations and financial sector supervision (MCM); one under PFM (FAD) and two under statistics (STA).

Desk-based research on eight training workshops delivered or sponsored by CAPTAC-DR. Interviews with staff from the IMF’s Western Hemisphere Department (‘WHD’), TA Departments, support

departments such as Finance and ICDGP staff in Washington, DC with respect to CAPTAC-DR’s strategy, operations, governance and the Center’s relationship with Head Quarters (‘HQ’).

Reviewing reports to the CAPTAC-DR Steering Committee (‘SC’). Reviewing internal CAPTAC-DR reports and other relevant IMF reports. Field trip to Guatemala to discuss the Center’s strategy, operations and TA interventions with the Centre

Coordinator (‘CC’), Resident Advisors (‘RAs’) and office staff.

http://www.imf.org/external/cap/


2 | P a g e

Online surveys of beneficiaries of the TA projects reviewed in detail; participants in the eight selected workshops; and SC members.

Field trips to Costa Rica, Guatemala and El Salvador to research and get feedback from stakeholders on the selected case study projects.

4. Conclude on lessons that could be learnt and recommendations for the future. 5. Draft and finalize the Report.

We have listed the persons interviewed in Appendix V. The four outputs of the Evaluation have been: 1. An Inception Report. 2. A Draft Evaluation Report. 3. This Final Evaluation Report (‘the Report’), comprising two volumes. Volume II is substantially providing evidential

backup to the conclusions in Volume I and covers in more depth the Case Studies and detailed Desk Top Reviews. 4. Presentation of evaluation results and recommendations to the CAPTAC-DR Steering Committee. The trip to IMF HQ was carried out in November 2012; the final draft of the Inception Report was issued in December 2012; the field trips to were carried out in November and December 2012; the first draft of the Report was issued for comments in March 2013; and the presentation to the SC was made in April 2013. 1.3 EVALUATION CRITERIA AND RATING SYSTEM The TORs set out a series of Key Evaluation Questions (‘KEQs’) for each of the OECD DAC criteria that the Evaluation is expected to address. It proposed a quantitative rating methodology, along with a framework for assigning weights to the scores for the different DAC criteria in order to arrive at a unitary score for each evaluated project, workshop or the CAPTAC-DR program as a whole. We adopted the rating system for individual DAC criteria of: Relevance (‘R’) Effectiveness

o Impact (‘E/I’)5 o Outcomes (‘E/OC’) o Outputs (‘E/OP’)

Efficiency (‘E’) Sustainability (‘S’) We deployed a rating system for each of the above evaluation criteria that is set out in Table 1 opposite to the extent feasible based on the project reviews, field trip case studies and an overall CAPTAC-DR-level assessment. Achievements of targets within these criteria have been rated as Excellent (‘E’); Good (‘G’); Modest (‘M’); Poor (‘P’); or Not Demonstrated (‘ND’). In addition, we have given numeric scores to allow for gradation within each rating criterion.

5 Impact was assessed, but not scored, as provided for in the Evaluation ToRs.

CAPTAC-DR Evaluation Rating Criteria Rating Achievement Base

Score Range

Excellent (‘E’) All or substantially all objectives met

4 3.6-4.0

Good (‘G’) Majority of objectives met

3 2.5-3.5

Modest (‘M’) Few/minority of objectives met

2 1.5-2.4

Poor (‘P’) Very few objectives met

1 1.0-1.4

Not Demonstrated (‘ND’)

Criteria cannot be assessed

ND


3 | P a g e

We had some reservations about assigning weightings to the scores for individual DAC criteria and arriving at a consolidated score. We detailed these in our Inception Report. After discussions with ICDGP and the Evaluation Sub Committee, we have provided weighted scores for projects, workshops and the CAPTAC-DR program, in addition to our preferred un-weighted ratings. 1.4 EVALUATING TA PROJECTS AND WORKSHOPS We evaluated CAPTAC-DR’s TA and workshops at four levels: Portfolio; Four Case studies (see Section 4); Five Desk Top samples (Section 4); Eight workshops (Section 5); Surveys of TA recipients and workshop participants. The TA projects and workshops selected for detailed review, which we believe provided the opportunity for a broad overview of projects by sector, covered revenue administration, PFM, monetary operations, financial sector supervision and statistics. Portfolio Reviews The reporting systems adopted by CAPTAC-DR do not readily offer detailed information on activities at a ‘project’ level (the logframes do provide a summary of activities). We sought to develop a portfolio level view of the CAPTAC-DR’s projects, by extracting relevant information from the annual reports and from reports presented to the respective SC. We discussed the performance of the overall portfolio with each RA. We also analyzed the portfolio across a number of criteria. For workshops, we reviewed the overall participant ratings scored by each of the 28 workshops undertaken by CAPTAC-DR. These findings were augmented by more detailed review of selected workshops. Case Studies and Desk Top Samples Section 1.5 outlines our work relating to case studies. Section 4 provides summaries of our detailed analysis of the TA desk top reviews. Volume II provides full details of both. Surveys We conducted two targeted online surveys to obtain feedback on the selected TA projects and workshops6. The TA beneficiaries’ survey comprised 27 questions, with many requiring responses to a number of sub-questions on respondents’ views. 129 individuals representing stakeholders for seven TA projects were invited to participate in the survey. 35 responses were received. 27.3 percent survey feedback was received in respect of all projects targeted. The workshop participants’ survey had 20 questions, structured along the same lines as that for the TA beneficiaries. 54 out of 173 invitees (31.2 percent) responded, and response rates for individual workshops ranged between eight and 31 percent. We were able to analyze responses at individual TA project and workshop levels, as well as on a consolidated basis. 6 On surveymonkey.com.


4 | P a g e

1.5 CASE STUDIES TA projects for the four case studies were selected to provide topical spread, size of resource commitment, longevity (more likely to see progress towards outcomes) and regional impact. Based on the above criteria, we selected the following projects: Customs Administration – Costa Rica External Statistics – El Salvador Cross Border Banking Supervision – regional: field trips to Guatemala and Costa Rica Operational Risk, banking sector – regional: field trips to Guatemala and Costa Rica. Both banking supervision projects were extended to the whole CAPTAC-DR sub region. One of the key challenges faced during the Evaluation was the absence of logical frameworks that provided sufficient and comprehensive enough data on which to base our assessment of project-level results. The BPs for missions only provide activities and outputs for individual missions. The implementation of RBM in CAPTAC-DR started only in 2012, and in any event the outcomes proposed for missions are difficult to evaluate against as they lack precision in definition. For each case study (and also desk top review), we therefore needed to construct ex post logical frameworks, based on available activity and output information, and providing illustrative outcome indicators for these projects.


5 | P a g e

We worked with a results chain approach to provide the logical underpinning for the logframes used in the case studies. Chart 1 below illustrates the sector level results chain we constructed for one of the case studies.

Illustrative Sector-Level Results Chain As part of the case study evaluations, we developed sector-level results chains. Ideally, these would have been developed ex ante, and for effective implementation of RBM, we believe that they are necessary going forward. They are a planning tool for TA interventions in a sector, intended to highlight the key TA and reform needs beyond the particular CAPTAC-DR proposed project, and the requirement for appropriate sequencing, that need to be met in order for the necessary overall TA program for the sector, including the particular CAPTAC-DR project, to achieve its higher level outcomes and impacts. For example, in the customs area illustrated below we have a higher level outcome of growing revenues faster than imports and facilitating trade with CAPTAC-DR’s project focusing on risk-based control systems but other interventions from IADB etc. and further steps required by the national authorities. They also help identify related risks and assumptions. The illustrative Results Chain for CAPTAC-DR’s Risk-Based Supervision intervention in Costa Rica is presented below.

We then developed the sector results chain to place the project into context, and assess the degree to which the project was appropriately sequenced, and the wider reform needs to achieve higher level outcomes were being addressed, and identification of any risks that could be mitigated. Volume II provides our detailed analyses.


6 | P a g e

1.6 EVALUATION APPROACH FOR STRATEGY, OPERATIONS, RBM AND GOVERNANCE CAPTAC-DR’s strategy and operations are deeply integrated with those of IMF HQ. This makes evaluating its performance on an isolated basis impractical as, in many areas, strengths and weaknesses are drawn from what’s happening at HQ. This is particularly true of CAPTAC-DR’s operations. Although we have commented on HQ-based activities where they directly impact on the Center, such as RBM, the Evaluation has necessarily needed to focus on CAPTAC-DR matters. We assessed CAPTAC-DR’s modus operandi through interviews with the WHD, TA Departments and ICDGP staff. Interviews were also conducted with the CAPTAC-DR CC, all seven RAs and Center’s Office Manager. A key benefit of the Evaluation has been the fact that the team is also concurrently evaluating West AFRITAC (‘AFW’) and East AFRITAC (‘AFE’). We have been able to draw conclusions based on relative performance of the three RTACs and the wider population of all RTACs. A number of IMF working groups have been deliberating on different aspects of RTACs’ governance, operations, funding and TA delivery approach respectively. Where available, reports or recommendations produced by these groups were also reviewed. In assessing the status of RBM implementation, we had discussions with ICDGP, the TA Departments and CAPTAC-DR. We also reviewed reports from the IMF interdepartmental RBM working group, a consultancy report on introducing RBM in the RTACs, and logical frameworks produced by a number of RTACs. Governance was mainly assessed through a review of SC annual reports and minutes of SC meetings, as well as the analysis of feedback from the online survey of SC members. There were some variations in questions posed to SC members representing donors, and those representing member countries, reflecting potential differences in goals and interests in participating in CAPTAC-DR. 37 individuals representing the Steering Committee members were invited to participate in the survey. 12 responses, representing five (out of seven) member countries; and five of the nine donor countries or institutions, responded.7 1.7 LIMITATIONS AND OTHER ISSUES We experienced the following limitations in carrying out the Evaluation: 1. CAPTAC-DR has made considerable progress in seeking to report on results since starting the implementation of RBM.

Outcomes are not yet well enough defined to make them suitable for evaluating the achievement of results. We reconstructed logframes and outcome indicators for the projects selected for detailed review. However, this has not been possible to do at a portfolio level, and our assessment of portfolio level effectiveness results should be treated with some caution.

2. The lack of consolidated performance based reporting at a project, country, sector and portfolio levels could make portfolio evaluation difficult. The lack of concrete verifiable indicators, the absence of vague actual performance against plans, and absence of consolidated analyses, impair the reported results’ usefulness for portfolio evaluation.

7 Two representatives from the same donor country kindly provided their individual views.


7 | P a g e

3. The RTAC (including CAPTAC-DR’s) financial and project cost reporting systems (i.e. the system for allocating costs

such as RA/STX time and travel, HQ time for backstopping and project management to specific TA projects) suffer from a number of limitations, which we believe requires further elucidation. One consequence of this is that it is not possible to easily establish the cost of projects and therefore assess project cost effectiveness and efficiency.

4. Under DAC definitions, actions relating to implementation of TA recommendations, such as the use of a manual, or

implementation of a process, would be classified as an indirect result, and an output. Given the nascent stage of implementing RBM at the IMF and CAPTAC-DR, and the need to shift focus from advice to results (of any kind), we felt that for the purposes of the Evaluation there should be a clear line drawn between TA activities within the CAPTAC-DR’s control (such as drafting a manual or a regulation), and results dependent on other people’s action (using the manual or adopting the regulation). We have defined the former as outputs and the latter as short-term or intermediate outcomes. These intermediate results are defined as milestones under IMF RBM terminology. Over time, to the extent independent evaluations are expected to report on performance against DAC criteria, we believe the IMF and RTACs should consider moving to more widely accepted definitions, including separating its definition of objectives into outcomes and impact.


8 | P a g e

2. CAPTAC-DR BACKGROUND AND CONTEXT 2.1 CAPTAC-DR IN CONTEXT

The IMF runs a number RTACs across Africa, the Caribbean, the Middle East, and the Pacific, as listed in Table 2 opposite. In addition, the IMF has also set up CAPTAC-DR, which covers Costa Rica, the Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua and Panama. See Chart 2 below. All RTACs seek to help their member countries in strengthening human and institutional capacities to support both economic development and poverty reduction. This novel regional approach seeks to

improve both the adaptation of the technical support to individual country needs and the responsiveness of TA delivery. CAPTAC-DR began its activities in July 2009 and is based in Guatemala City. There were considerable delays in recruiting resident advisors and as a result CAPTAC-DR was not fully operational until the end of 2009. The Center’s work program covers Tax and Customs Administration, Public Financial Management, Financial Sector Supervision, Monetary Operations, and Statistics (i.e. National Accounts, External Sector Accounts, Economic and Financial, as well as Public Finances). The contribution from a group of countries and organizations has made this initiative possible. The Center’s donors are Canada, the Central American Bank for Economic Integration (‘BCIE’), the European Commission (‘EC’), Germany, the Inter-American Development Bank (‘IADB’), the IMF, Mexico, Spain, as well as CAPTAC-DR member countries (with a substantial contribution in kind from Guatemala, the host).

List of IMF RTACs

No. Region Base Set up Countries 1. Pacific Suva 1993 16 2. Caribbean Bridgetown 2001 20 3. East Africa Dar es Salaam 2002 7 4. West Africa Abidjan 2003 10 5. Middle East Beirut 2004 10 6. Central Africa Libreville 2006 7 7. CAPTAC-DR Guatemala 2009 7 8. Southern Africa Port Louis 2011 13 9. West Africa

(English-speaking) Accra Expected

Soon 6

Source: IMF & CAPTAC-DR websites

http://www.imf.org/external/cap/


9 | P a g e

Unlike longer standing RTACs, such as those for the Caribbean or for the Pacific, CAPTAC-DR is still in its first funding cycle, which started in FY2010. Its current budgetary phase runs for a total of five years, with a working budget of US$35.7 million. The Center’s average annual budget amounts to slightly over US$7 million based on the 2009 program budget but annual budgets have risen from US$5 million in FY2010 to US$ 8.2 million in FY2012. By way of comparison, yearly budgets for the current phase of East and West AFRITAC respectively amount to US$8.9 and US$9.0 million, although in the latter case the annual spend has only averaged about US$4.5 million. In regions where traditional IMF TA is relatively large, an important comparative advantage of the RTAC model lies in complementing diagnostics and strategically focused headquarters-based TA, and thereby enhancing follow-up and implementation. Where traditional Fund TA has been more limited, the Centers have undertaken a mix of both strategic and follow-up TA. On average, an RTAC roughly provides around two-thirds of the IMF TA delivered to the countries covered by that particular regional Center. The RTACs are one among the various components of a range of TA delivery modalities that the IMF uses, such as Topical Trust Funds and TA funded from bilateral sub accounts. As Chart 3 above shows, they have taken increasing importance over the last five years. In FY2012 RTACs are expected to deliver about 44 percent of externally financed IMF TA.8 As Chart 4 opposite illustrates, CAPTAC-DR’s TA delivery was relatively evenly spread across its member countries, measured in dollar terms. El Salvador absorbed approximately 13 percent of CAPTAC-DR TA, followed by Guatemala and Honduras each with 12 percent. Nicaragua and Costa Rica received about 11 percent each, whereas the Dominican Republic and Panama took up approximately 10 percent. The remaining 21 percent was directed towards activities at the regional level. TA was also well-distributed among technical topics. Support on Tax Administration required approximately 16 percent of total funds. At the same time, about 15 percent of total resources were spent each on, Public Financial Management and Monetary Operations. Customs Administration and Financial Supervision accounted for 14 percent, while Statistics 8 CAPTAC-DR Draft Program Document 2014-19.


10 | P a g e

absorbed 26 percent. Of the latter most of the work was directed to External Sector and National Account Statistics (respectively receiving 11 percent and 13 percent of the total). The remainder (less than 2 percent of the total) was spent on Public Finance and Monetary and Financial Statistics. Section 7 provides further information and analyses regarding the Center’s finances. 2.2 CURRENT LANDSCAPE OF CAPTAC-DR ACTIVITIES Generally, CAPTAC-DR countries have a long history of regional integration efforts and accomplishments. Yet this still remains a highly diverse region. Although all countries are relatively small, populations range from 14.8 million in Guatemala, to 3.6 million in Panama. Some have relatively high per capita incomes, such as Panama and Costa Rica (with respectively US$7,498 and US$8,646 in 2011) whereas Nicaragua’s only amounted to US$1,587. Inflation has been relatively subdued in recent years, though the Dominican Republic grappled with consumer price increases of 54 percent in 2004. Even literacy rates vary strongly between CAPTAC-DR countries: from 75 percent in Guatemala (during 2005-9) to 96 percent in Costa Rica. Finally, almost half (48 percent) of the population in Honduras live in rural areas, whereas three quarters are urban dwellers in Panama. Table 3 below, provides a summary of these indicators across the region.

The onset of the 2008 global financial crisis generally delayed CAPTAC-DR member countries’ progress in a number of areas, though a limited recovery ensued. High-level development related indicators show positive trends. Per capita incomes at current price have been increasing since 2010, though still trail below the average for Latin America and the Caribbean (‘LCN’),

for all CAPTAC-DR member countries. This is the case even for Costa Rica and Panama, the region’s best performers for this particular indicator. At the same time, child mortality rates (for children under five) have fallen broadly in line with LCN averages. Costa Rica constitutes one notable exception with much lower child mortality than the regional average, though the decrease is now much slower. The case of El Salvador is also noteworthy, as the country has seen a fall in this indicator by 14 deaths per thousand between 2002 and 2011, thus placing it well below the LCN average (though still above Costa Rica). CAPTAC-DR countries avoided major banking crises, reflecting their banking sectors’ lack of toxic assets, strong capital, and adjustments toward liquid assets.

CAPTAC-DR Member Country Development Indicators

Country CRI DOM SLV GTM HND NIC PAN

Per capita GDP, curr. prices 2011 8,646 5,530 3,702 3,178 2,193 1,587 7,498 GDP Growth 2011 (% est.) 4.2 4.5 1.5 3.9 3.7 4.7 10.6 Inflation CPI 2011 (%) 4.9 8.5 5.1 6.2 6.8 8.1 5.9 Dom. Credit Banks/GDP (%) 53.1 38.9 44.6 39.2 48.4 58.8 91.6 Fiscal Def % GDP 2010. 2.8 3.5(09) 2.7 3.3 2.8 1.0 3.4 Gov. Debt %GDP 2010 42 42 49.5 24.4 32.1. n.a. n.a. Dollarization (%) 50 30 100 32 30 90 100 Inter. Res. US$bn 2011 4.8 4.1 2.5 6.2 3.0 1.9 2.3 Urban population (%) 65 70 65 50 52 58 75 Literacy (2005-2009, %) 96 88 84 74 84 78 94 Source: World Bank; “Dollarization Declines in Latin America” Finance & Development, 2010.


11 | P a g e

3. EVALUATION OF CAPTAC-DR AT PROGRAM LEVEL 3.1 INTRODUCTION AND SUMMARY OF RESULTS

This section seeks to evaluate CAPTAC-DR’s performance at an aggregate level, taking into account our assessment of the Center’s TA portfolio, its workshops and wider issues. Our analysis shows that CAPTAC-DR responds to pressing TA needs by means of high quality technical work. The Center has efficiently used the resources at its disposal while also producing sound TA outputs. Because project design has usually taken place on the basis of appropriate diagnostics, CAPTAC-DR’s interventions are also likely to bring about quality outcomes, thus contributing towards attaining CAPTAC-DR’s goals.

CAPTAC-DR’s work is expected to be broadly sustainable. Sections 3.2 to 3.6 present our approach and its application to arrive at our ratings for the actual and potential levels of achievement of CAPTAC-DR’s outcomes at the program level. Section 3.7 provides our responses to the CAPTAC-DR-level Key Evaluation Questions (KEQs), and provides our ratings across the DAC criteria. Of all the rating criteria, outcomes have been the most difficult to establish. Although CAPTAC-DR closely reports project results, it remains very difficult to relate annual logical frameworks compiled with the work plans back to the report of actual TA delivered for the year in retrospect. The logframes included in the Center’s 2013 Annual Report are a good step forward in that they compare plan against actual and include comments on whether objectives have been achieved wholly or in part. Outcomes are not clearly defined in some cases and indicators often lack specific means of measurement. The evidence of systematic monitoring of indicators is improved in the 2013 Annual Report but could be further expanded by explanations of variance (see Section 9 on RBM). In any event, the result of the assessment of TA projects selected for detailed review cannot be extrapolated to the CAPTAC-DR portfolio-level directly. 3.2 APPROACH FOR EVALUATING CAPTAC-DR OUTCOMES In order to generate an aggregate perspective of CAPTAC-DR’s portfolio, we sought to identify its constituent parts. Thus we have singled out a number of categories for the Center’s TA activity, namely:

Evaluations, assessments, needs analysis, diagnosis and the resulting strategies and action plans (some of which

may feed into CAPTAC-DR multi-year TA projects); Formal training activities at regional or national level; Support to building capacity on technical areas related to statistics; Short-term, ad hoc, specialized interventions; and Multi-year projects (we created this category by reviewing the reports).


12 | P a g e

Broadly, we would expect missions involving diagnostics to generate least outcomes directly (follow-up TA projects would); training on its own would generate some sustainable capacity building especially if linked more directly to TA interventions, but our assumption is that more sustainable capacity would be built on TA projects, and of the latter, multi-year projects would generate most. In addition, we have combined evidence from a number of sources in order to triangulate, estimate and cross-check our ratings. We used both primary and secondary sources of information, namely: Consolidated ratings of the long-term CAPTAC-DR TA projects for which we have carried out case study and desk

top assessments. We weighted the case studies to the desk top reviews on a 60/40 basis because we were able to analyze these in more depth with field trips;

A review and analysis of all of the projects in CAPTAC-DR’s project portfolio for financial years 2011 and 2012. These projects have been classified as per the above-mentioned categories; and

Inputs from the three surveys conducted during the course of the evaluation (namely on TA projects, workshops, as well as the SC members).

3.3 RATINGS FROM CASE STUDIES AND DESKTOP REVIEWS Table 4 below highlights the evaluation ratings given to the CAPTAC-DR projects reviewed in detail and are reasonably narrowly ranged. This consistency points out to an overall coherence and homogeneity in the quality of CAPTAC-DR’s long-term TA portfolio (and at a consistently high level of quality). The average outcome and output scores are respectively 3.3 and 3.5 (in the top end of the Good category). Seven of the case studies and desktop reviews were multi-year projects representing a majority of on-going projects as of December 2012. This sample accounted for a large proportion of CAPTAC-DR projects with approximately over a third of TA outputs delivered.

Summary Evaluation Ratings for TA Projects Country Project Relevance Effectiveness Efficiency Sustainability

Outcomes Outputs Case Studies CRI Customs Administration E E E G G REGIONAL Cross Border Consolidated Supervision E G E G G REGIONAL Operational Risk Supervisions G G G G ND SLV External Sector Statistics E G E G E Desktop Reviews HND + SLV Large Taxpayers Units E G G G G NIC Tax-Customs Information Sharing E G G G G REGIONAL Single Treasury Account E G G G G CRI + DOM + GTM Liquidity Management E G G G G REGIONAL National Accounts Statistics E G E G G


13 | P a g e

Outputs, and particularly outcomes, can be best assessed as project implementation approaches maturity. For this reason, the above ratings should be taken with particular caution. It should be pointed out that the TA projects were not selected for evaluation randomly, and by themselves do not necessarily accurately represent the performance of the portfolio. There was a bias towards projects that were longer-term, where results of CAPTAC-DR’s interventions could be more easily discerned. This meant that projects that may have been aborted (if any), or did not demonstrate results (if any), as well as short-term projects that were successful and had a catalytic effect for further TA, were screened out to a certain extent. Nevertheless, we did review the overall portfolio, through review of annual and periodic progress reports, as well as regional seminars and workshops in order to achieve a balanced rating at the program level. Overall, our assessment of the TA projects demonstrates strong results being achieved, especially for the longer-term projects. This is reflected in generally high ratings across all the DAC evaluation criteria.

Table 5 opposite lists the estimated number of projects assessed as part of this evaluation by means of a case study or of a desktop review, which were in turn illustrative of a larger sample of CAPTAC-DR initiatives. This table shows that projects appraised during the evaluation are

representative of 45 undertakings by the Center, across all of its topic areas, and covering all of the member countries. This is particularly the case for projects with regional coverage, such as those on External Sector Statistics, or Cross Border Consolidated Supervision.

Topical Area Project Assessed in Description Countries

Applicable No. of projects

PFM Region Single Treasury Accounts SLV, GTM, HND, PAN, & DOM

5

Customs CRI Risk management All CAPTAC-DR 7 Taxation SLV & HND Large taxpayer units SLV, PAN, & HND 3

NIC Data sharing: tax & customs NIC 1 Supervision Region Cross-border consolidated supervision All CAPTAC-DR 7

Region Operational risk SLV, HND, DOM, NIC, & PAN

5

Monetary Operations

Region Liquidity management CRI, DOM, & GTM 3

External Statistics

SLV PRAESE project All CAPTAC-DR 7

Nat’l accounts statistics

Region National accounts, quarterly, annual, monthly, PPIs etc.

All CAPTAC-DR 7

Total 45


14 | P a g e

3.4 PORTFOLIO ANALYSIS BY TYPE OF PROJECT Table 6 below provides information on the composition of CAPTAC-DR’s projects, which we categorize as either short- or multi-year. The former includes evaluations, needs assessments, ad hoc short-term TA, as well as formal training. Multi-year projects include all TA projects lasting more than one year, as well as CAPTAC-DR’s support to its members on issues related to statistics.

Type of project 2011-2012 Plan 2012-2013

Weeks % Weeks % I – Short-term projects 253 43 220 37 Evaluations / needs assessments / diagnoses 67 12 11 2 TA short term 126 21 164 27 Formal training / national workshops 60 10 45 8 II – Long term TA projects 329 57 372 63 Statistics 163 28 161 26 Multi-year 166 29 211 37

Total (short- & long-term) 582 100 592 100

Source: CAPTAC-DR

We reviewed two Statistics projects (one as case study on external statistics in El Salvador and one as a desktop assessment on National Accounts). We also scanned the Annual Reports since inception, both reports on TA delivered for the years as well as work plans going forward. Our table above is approximate because it was not always easy to match up project descriptions between planned and that actually performed and not always easy to determine whether a project was short-term or long-term. We believe it is nonetheless a good approximation. The CAPTAC-DR multi-year projects (including statistics) were narrowly focused on certain topics: the main areas reviewed in the evaluation were: Large Taxpayer Units, Risk Management Systems in Customs, Single Treasury Accounts, Cross Border Consolidated Supervision, Liquidity Management, External Statistics and National Accounts. Aggregating these by sector weeks and applying the topics covered to the total number of weeks in 2011 and 2012 indicates that the case study and desktop reviews covered about 46 percent of the multi-year projects by topical area. With Statistics and multi-year projects combined we believe there is sufficient evidence from the process followed so far to support a rating of those areas between 3.1 and 3.3 for outcomes, as well as 3.4 and 3.5 for outputs. The short-term TA projects accounted for approximately 28 percent of the portfolio for the two years reviewed. We found it difficult to rate these for reasons mentioned elsewhere, namely the difficulty of marrying up reports of actual TA delivered versus that planned. Two of the desktop reviews did cover short-term projects: i.e. Tax and Customs information sharing in Nicaragua and Operational Risk for banks in three countries. The results of these two projects were rated Good. Overall, we would expect direct outcomes from these short-term interventions to be less deep than those of the long-term TA projects i.e. tax and customs information sharing in Nicaragua should eventually generate more investigations into discrepancies as between customs declarations and tax returns (yet to be demonstrated) and the development/ amendment of regulations on operational risk should eventually show up in supervision inspection reports.


15 | P a g e

3.5 FEEDBACK FROM SURVEYS TA beneficiaries A total of 35 persons (out of 129 invited) replied to an online survey of TA beneficiaries. In absolute terms, most replies came from beneficiaries of National Account Statistics TA (eight responses). In proportional terms, replies on PFM only commanded a reply rate of 11 percent, partly due to the large number of invitees. At the other end, Tax Administration attracted a reply rate of 57 percent. Given the targeted, rather than random survey methodology, these replies must be interpreted with caution. They do however provide interesting insights. With respect to designing of the TA, 17 out of 35 strongly agreed (and 30 agreed or strongly agreed) that CAPTAC-DR TA is assisting in one of their country’s/institution’s priority areas. A similar number of respondents, more specifically 18 out of 35 strongly agreed that the TA was carefully sequenced to ensure results. In addition, 21 out of 35 respondents strongly felt that CAPTAC-DR and their respective institutions had worked in partnership to design and prioritize the TA. In addition, all but one respondents strongly agreed, or agreed, that TA advice was provided in clear, practical and easily implementable ways; although there was a lower level of support for adequacy of follow-up after delivery. Encouragingly, it was generally felt that CAPTAC-DR understands respondents’ countries and institutions needs and capacities. Workshops A total of 54 respondents completed the survey on CAPTAC-DR’s workshops and seminars (31 percent). There was a relatively even distribution of results among topics, with Tax Administration yielding the highest number (10) and proportion (27 percent) of replies. The lowest representation was from PFM at 17 percent. As with the TA survey, results are of indicative value only. Nine out of ten respondents confirmed that the workshop was pitched at the right level for the participants, with a 100 percent response rate for Tax Administration and for Monetary Operations. The main exception to this trend was the respondents to the Public Accounting workshop, as half of them considered that the technical level was pitched too high. Survey results suggest that CAPTAC-DR workshops impart solid and useful knowledge. In addition, over 96 percent of the respondents expressed satisfaction with the training (half of them being highly satisfied).


16 | P a g e

3.6 KEY EVALUATION QUESTIONS Key Evaluation Questions Comments Relevance Consistency with Program Document and Government Priorities

The implementation of CAPTAC-DR’s activities has been in line with the Program Document. CAPTAC-DR has cooperated well with regional organizations. Survey results and case/ desktop reviews support the claim of strong ownership. Generally the work plans seem well sequenced but there are some instances where weaker countries could be left behind a little and where some obstacles could cause delays e.g. legislation in Panama to ensure effective cooperation on cross border consolidated supervision. The strategy development processes seem robust. It seems very clear from the Annual Reports to the SC that CAPTAC-DR coordinates well with other donors. There is room for improvement.

Consistency with IMF Headquarter/other activities Whether CAPTAC-DR’s activity is appropriately focused in terms of subject area, taking into account the IMF’s expertise and integration with HQ and other Fund activities (e.g. other training and TA delivered including through the TTFs). Coordination with Development Partners Context of donor/TA landscape of IMF TA

Score for Relevance 3.6 Excellent Effectiveness Impact CAPTAC-DR and aggregated project-level Impact9 as defined in the Program Document, achieved or likely to be achieved

It is too early to judge impact.

Outcomes CAPTAC-DR and aggregated project-level Outcomes (short, medium and long term, is/as defined), as defined in the Program Document, achieved or likely to be achieved, including e.g., implementation of advice.

We do consider there is an outcome risk that is difficult to assess and predict in CAPTAC-DR’s region during a single Program cycle, especially as the present cycle is only about 3 years old. There are some areas like revenue administration where medium-term results can be achieved within the life cycle of a single funding phase. In other areas such as PFM and Supervision even medium-term outcomes lie beyond full implementation. This exercise is very much an overview because RBM is still at a relatively early stage of implementation by the IMF and by CAPTAC-DR.

Outputs

Score for Outcomes 3.1 Good

Score for Outputs 3.4 Good Efficiency Process and implementation efficiency The process is good at involving HQ, CAPTAC-DR and beneficiary countries

for developing work plans. After a slow start CAPTAC-DR has caught up with the aims of the Program Document and executes close to budget. The difficulty we have in evaluating this particular aspect is in comparing TA actually delivered with TA planned. Some categories are clear e.g. large taxpayer units but others are not. Our workshop review section endorses a solid rating. Backstopping: This area needs looking into more, especially given the low

Efficient use of resources (human and financial) & attention to cost effectiveness

Monitoring and reporting

9 Impact assessed qualitatively only, and only where practical.


17 | P a g e

Key Evaluation Questions Comments actual versus budget of backstopping, and we have recommended that is done with a possible view to increasing decentralization and or the scale / volume / periodicity of reporting. Similarly project management was running at only 73 percent of budget through December 2012. Appropriateness of staffing levels seems good with a high level of regional/local expertise. The ratio of RA to STX time is very good. We have suggested additional support for RAs. It is not easy to compare budgets with actual at project level and inputs to actual expenditure come from different sources and are not under a common budgetary control (e.g. allocation of some HQ costs are not controlled by CAPTAC-DR). Substantial progress has been made in establishing logical frameworks and reporting on results but there is still a lot of improvement needed. We comment on this in the RBM/logical frameworks section. Given activities are not organized by projects, financial budgeting and monitoring of project costs is relatively weak, contributed to by shortcomings in HQ financial reporting systems which render accurate allocation of some direct (e.g. RA time on projects outside missions) and indirect costs (e.g. Center overheads) difficult.

Score for Efficiency 3.1 Good

Sustainability Sustainability of TA activity There are already some good examples of tangible results in areas such as

revenue administration. It is more likely that in some of the topical areas results will be more easily measured and sustainability of outcomes will also be easier. For most projects it is still too early to form a clear judgment, albeit trends are good. Sustainability risks remain high in some sectors. Recruiting and retaining quality staff in beneficiary institutions is the biggest challenge in some cases. Recognition and assessment of absorptive capacity should be analyzed thoroughly at the project design stage and resource allocation determined accordingly.

In addition to the TA risk there is the funding risk for CAPTAC-DR. About 20 percent of the pledged funding has come from Spain in Phase I and it seems this will not be forthcoming for Phase II.

Sustainability of the CAPTAC-DR funding model Contribution to building sustainable regional TA, implementation capability.

Score for Sustainability 3.0 Good


18 | P a g e

Rating Based on Previous Evaluation Methodology

DAC criteria Sub-criteria Score Weight

(%) Weighted

score Weight

(%) Total score

Rating

Relevance Consistency with Program Document and Government Priorities

3.7 60 2.22

Consistency with IMF Headquarter/other activities 3.6 20 0.72 Co-ordination with Development Partners 3.4 20 0.68 3.62 32 1.16 Excellent Effectiveness Impact - Outcomes: TA 3.1 30 0.93 Outcomes: Regional capacity building 3.1 30 0.93 Outputs 3.5 40 1.40 3.26 28 0.91 Good Efficiency Process and implementation efficiency 3.3 40 1.32 Efficient use of resources (human and financial) and

attention to cost effectiveness 3.3 40 1.32

Monitoring and reporting 2.9 20 0.58 3.22 22 0.71 Good Sustainability Sustainability of RTAC/TA activity 3.2 75 2.40 Sustainability of funding model; Contribution to

building sustainable regional TA, implementation capability

2.5 25 0.63

3.03 18 0.54 Good TOTAL 3.32 Good


19 | P a g e

4. EVALUATION RESULTS FOR TA CASE STUDIES AND DESKTOP REVIEWS

4.1 OVERVIEW We evaluated nine CAPTAC-DR TA projects in detail, four as case studies and a further five as desktop reviews. The case studies included the construction of a results chain to highlight the sequencing and wider TA/reform needs for the project and also included country visits and interviews with the TA beneficiaries and wider stakeholder groups. The desktop reviews were primarily carried out based on documentation available at the Centre and, in some instances, interviews with the relevant RAs. Finally, the TA beneficiaries were also invited to provide feedback by way of a structured survey on their opinions about CAPTAC-DR’s approach to their project and the TA received. Our detailed analyses of the case studies and desktop reviews are provided in Volume II, which also provides our responses to the KEQs in respect of each of the projects reviewed. The Evaluation did not provide any indication that there were systematic differences between the TA results achieved by the different TA Departments. Whilst drawing on these results, this section also accommodates other factors that are important at the aggregate level but do not influence individual TA project ratings. 4.2 KEY FINDINGS We present below our principal findings from the case studies and desktop reviews. More detailed project-specific findings and recommendations are presented in Volume II. Relevance 1. All the projects reviewed respond to pressing TA needs, which constitute a priority for the TA recipient institutions. 2. In the case of Operational Risk Supervision, doubts may linger on whether the choice of this particular project was

fully demand-driven, as this complex area is most relevant to OECD banks (than to CAPTAC-DR banks). In addition, the support to customs in Costa Rica came at a time when the government was rather focusing on the tax administration in the context of its reform initiatives and was paying less attention to customs. These marginal comments should however not detract attention from the fact that CAPTAC-DR’s interventions have been extremely relevant.

3. In many cases (such as Monetary Operations, External Sector Statistics, and Cross Border Supervision), CAPTAC-DR provides important TA, which is not easily available from other donors operating in the region.

4. We have noted a number of instances where CAPTAC-DR missions have worked productively in partnership with other donors to provide TA (e.g., Large Taxpayer Units in El Salvador). However, in some cases this collaboration appears reactive, rather than proactive.


20 | P a g e

Effectiveness 1. A number of projects, such as those on National Account Statistics, Liquidity Management or on External Sector

Statistics, were particularly adapted to CAPTAC-DR’s TA delivery approach, as they were served through regular advisory visits.

2. A regional anchor can be a great support for regional projects, as was the case for CCSBSO for Cross Border Supervision.

3. IT upgrades are often necessary alongside technical advice (e.g. customs-tax collaboration in Nicaragua or customs reform in Costa Rica). Since the IMF excludes funding for IT investments, complementary support from donors and/or a major financial commitment from the member countries are required.

4. Generally, hands-on training sessions delivered by the RAs and STXs during missions are highly valued and considered very effective. In many cases though, including Operational Risk & Cross Border Consolidated Supervision, the time dedicated to this type of training seemed low.

5. TA beneficiaries have often suggested an increase in the duration of training and mentoring (e.g. Cross Border Consolidated Supervision).

6. The detailed recommendations provided during CAPTAC-DR missions (and spelled out in the various reports reviewed) are technically sound and well suited for in-country conditions. The RA structure and modus operandi should allow CAPTAC-DR to monitor the implementation and respond to calls for help as and when appropriate.

Efficiency 1. Although impressive progress has already been achieved, long-term support is likely to be required in all the projects

under review (perhaps with the exception of the customs-tax information sharing initiative in Nicaragua). To achieve all the higher outcomes of the TA delivery may require a further three or more years. The planning process is however organized annually; thus obscuring an overall programmatic approach and can lead to fragmented TA efforts that miss the big picture.

2. In some cases, such as for example Liquidity Management, CAPTAC-DR has produced a monitoring matrix with the advancement of project recommendations per country. These documents have been circulated regularly with country counterparts, thus helping to keep the momentum for reform.

3. The generally weak project level accounting and financial management systems do not enable accurate budgeting and monitoring of TA costs, and measuring its cost effectiveness The current budgeting system mostly provides information on mission costs (by person time of RAs or STXs). This process does not easily allow for building in the time of backstoppers to particular projects and other ancillary costs. It is not clear how to track actual versus budgeted costs for individual, multi-year, multi country TA projects. Similarly, for individual workshops, amounts spent by the Center and HQ are accounted for separately.

4. Project design has not been made explicit and sometimes shows gaps. It is not clear which other aspects, beyond the reach of CAPTAC-DR’s TA have to be put in place in order for the project to succeed (e.g. IT upgrades, or institutional infrastructure for cross-border consolidated supervision). A Results Chain approach facilitates this analysis. This tool also allows for a better understanding of all the risks hinging on the achievement of project goals.


21 | P a g e

Sustainability 1. There is limited use of training-of-trainers (‘ToT’) to build local capacity. CAPTAC-DR’s short-term mission based

delivery modality does not lend to this, as more than one cycle of training is usually required for local trainers to take over effectively. ToT also depends on the adequacy of resources at the TA beneficiary institutions (s) to support/justify in-house training capacity. Developing a ToT approach is a highly complex issue and should be analyzed in depth prior to launching any particular initiative.

2. Building in-house training capacity at the various counterpart institutions also constitutes a priority in the majority of the projects reviewed.

3. The pace of development in relation to specific topics varies from country to country, there seems to be scope for more technical support between member countries.

4. Absorptive capacity is relatively high across the region. In the exceptional occasions where it has been inadequate, the Center has taken appropriate adaptive measures. Political commitment in turn, seems to vary according to countries and technical subjects. For example, in the case of Nicaragua and Honduras, renewed political commitment brought a new impetus to the tax-customs and Large Taxpayer Unit projects.

4.3 CASE STUDY: COSTA RICA – CUSTOMS ADMINISTRATION

CAPTAC-DR has provided TA to the Costa Rican Customs Services (‘CCS’) to revise its customs procedure manual as well as the customs IT system (‘TICA’). Both initiatives are expected to contribute towards significant improvements in the performance of customs controls, as well as in strengthening CCS’ ability to raise revenue for the Costa Rican Government. In addition, CCS adopted a list of 11 customs monitoring indicators, which are linked to CCS reform action plan. Another project pillar relates to improving operational procedures. This work has so far focused on Costa Rica’s ports and airports. The first step involves standardizing specific actions at various points of the workflow. Secondly, information channels

between the different actors are strengthened. Within this context, one particularly important aspect has been the introduction of improved customs value measurement systems, thus helping address under-declarations. The final key TA activity has related to introducing risk-based control systems. This work has involved setting up risk profiling mechanisms based on importer segmentation. Within this framework, a pilot initiative has been launched at the Santa Maria customs point. As a result, Costa Rica already boasts: A 5.3 percent increase of its selectivity performance index between 2010 & 2011 (13 percent at the Santa María site); Over 130 percent improvement in the 2011 results of ex-post audits, particularly in customs value audits; and A 23.4 percent increase in customs revenues in 2011 (whereas imports grew by 10.9 percent during the period). We rate Relevance of the project as Excellent (score 3.8), demonstrating its close alignment with CAPTAC-DR, regional, and Costa Rican strategies and plans; strong ownership demonstrated by CCS and broad coordination with development partners.


22 | P a g e

With respect to Effectiveness, our rating at the Outcomes level is also Excellent (3.6) reflecting the substantial achievements of outcomes at a very early stage. However, we note that there is a risk that further achievements may be impaired if the authorities’ focus shifts away from this area. The high quality of advice delivered is reflected in a rating of Good (3.5) for Outputs, which is just one notch below the top category. Efficiency is rated as Good (3.5), reflecting a high ratio of outputs to inputs. Outputs and a number of outcomes have been attained at a remarkable speed. Finally, Sustainability is rated on the upper range of Good (3.3), because the majority of the advice delivered so far is likely to be retained within CCS. As project implementation continues, prospects for sustainability are likely to improve even further. 4.4 CASE STUDY: CROSS BORDER CONSOLIDATED SUPERVISION (REGIONAL)

The project seeks to contribute to the adaptation and implementation across CAPTAC-DR members of best international practices in banking supervision and regulation. Its major thrust has centered on risk-based supervision, cross-border consolidated supervision, and the adoption of prudential criteria based on international accounting standards. Given the relatively important presence of regional and global financial conglomerates, compliance with best practices in this area is expected to bring significant gains in terms of increased financial stability across the region. So far CAPTAC-DR has carried out a number of initiatives, namely: assessments/diagnoses and strategic advice to The Central American Council of

Superintendents of Banks, Insurance and Other Financial Institutions (Consejo Centroamericano de Bancos, de Seguros, y de Otras Instituciones Financieras – ‘CCSBSO’). The Center has also completed a methodology for risk analysis and consolidated cross border supervision of financial conglomerates. CAPTAC-DR additionally held a regional workshop on the issues of cross-border consolidated supervision in April 2012. In the short-term, efforts to implement the methodology are already underway in Costa Rica and Guatemala. We are informed that CAPTAC-DR is providing TA from December 2012 to Panama in drafting changes in legislation that would allow supervision of holding companies that included financial intermediaries. This is very important because Panama hosts most of the financial holding companies in the region. In December 2012, the final version of the methodology, including comments from the IMF headquarters, was presented to the bank regulators of the seven countries. Also in December, CAPTAC-DR set up TA to support Panama. We rate Relevance of the project as Excellent with a score of 3.6. The TA project is fully consistent with the priorities of the CCSBSO and with overall strategic objectives of CAPTAC-DR. The outcome of the TA complements the work of other development partners in the financial sector. The limited awareness of some of the complementary TA provided in the region suggests that donor consultation could be improved. In addition, updating legal and regulatory frameworks need prioritization for the TA to deliver substantive outcomes. With respect to Effectiveness, our rating at the Outcomes level is Good with a score of 3.2. The high quality of advice delivered is reflected in a rating of Excellent at 3.6 for Outputs.


23 | P a g e

Efficiency is rated at the high-end of Good at 3.3. The TA appears to have been delivered close to plan. Our principal concern is whether the TA, which was sufficient for this initial stage, may need to be increased substantially, for example through revisions to the legal and regulatory frameworks, and how much more may be needed to ensure high quality supervision. In turn, Sustainability is rated as Good at 3.0 as it will depend not only on adoption of the methodology and improvement and maintenance of the capacity of the supervisory institutions but also on upgrading the methodology to deal with new developments in financial markets. The capacity of the supervisors in the region is reasonable but would benefit from improvement. Finally, sustainability will depend on the supervisory agencies’ capacity to retain high quality staff, which so far appears likely. 4.5 CASE STUDY: OPERATIONAL RISK SUPERVISION (REGIONAL)

Operational Risk (‘OR’) Supervision has formed an increasingly important element of risk-based supervision (‘RBS’) in the evolution from Basel I to Basel III and its adoption by OECD member countries. Supervision of operational risk management is ultimately aimed at reducing banks’ risks in non-lending and funding risk areas. It is still in its infancy worldwide. Basel III proposes allocation of capital against some of the operational risks once assessed. This TA initiative seeks to increase the stability and robustness of the financial sector by enabling country regulators to supervise banks’ operational risks in a more effective manner. In the short term the project has supported the publication of new/revised regulations on operational risk in El Salvador, Honduras, Panama,

Nicaragua, and the Dominican Republic. In the medium-term, more effective operational risk supervision should result from the interventions of various providers of TA in the region as well as from the continued efforts of both the CCSBSO and the country-level supervisors. The dissemination of best practices and the incidence of actual risk events will also form part of the results expected from the Center’s Consolidated Cross Border Supervision TA. The work was appropriately sequenced with respect to country coverage, beginning in countries without a regulation on operational risk management, then moving to countries where existing regulation needs improvement. CAPTAC-DR has provided high quality TA on new supervisory regulations on operational risk and modification of existing regulations in countries where appropriate. The Center also contributed to the development of training plans for supervisors and staff of banks in operational risk and advised on dissemination. Finally, superintendents (i.e. bank regulators) have been trained on this issue. Similar outputs are expected in 2013 in Costa Rica and Guatemala. Relevance is rated Good (2.8) because it is important for CAPTAC-DR banking regulators to understand the implications of this new area of work. The Center, together with the IMF, constitutes the ideal partner for this work. Nevertheless, in our opinion this particular technical area is less relevant to CAPTAC-DR regional banks than in more developed banking systems, such as USA and Europe, where banks engage in active market trading activities with some well publicized disasters (sub-prime mortgages, rogue traders and so on). Moreover, we believe, that regulations should have been sequenced after identifying specific operational risks in the financial institutions, not the other way around. A solid assessment is also made of the project’s Effectiveness although of limited depth. Its deployment of a range of Outputs of


24 | P a g e

high technical quality warrants a rating as Good (3.5). In addition, short-term Outcomes have been achieved, though achievement of medium- to long-term outcomes is likely to be far more complex. A solid 3.0 points (Good) is assigned to Outcomes. Efficiency is also considered Good (3.0), as activities have been carried out on a timely basis. As the project is still in its early stages, largely limited to improving or developing regulations, its Sustainability could not be properly assessed. The development of in-house, national, and regional capacities is considered critical aspects to watch, as project implementation continues. 4.6 CASE STUDY: EL SALVADOR – EXTERNAL SECTOR STATISTICS

The Salvadorian Government, and more specifically the country’s central bank (Banco Central de Reserva de El Salvador - ‘BCRES’), has pushed forward the reform agenda. Improvement in the way statistics are compiled can already be noticed. In fact, the project is on its way to achieve most of its intended outcomes. Furthermore, prospects for sustainability of this intervention are high. The strategic objective of this initiative is to support El Salvador in compiling both comprehensive and consistent statistics of the country’s external sector. A key concern is to ensure compatibility across the region and to apply international practices, standards, and basic principles of compilation and dissemination. This project ultimately seeks to provide quality information to Salvadorian decision-

makers in order for them to steer economic policy, against the backdrop of high sensitivity to shocks affecting either exports or remittances. On the basis of the initial assessment above, a concrete plan of action was set. Progress was monitored against this plan. The country’s central bank has high absorptive capacity. For this reason, the project strategy has focused on providing advice at regular intervals, while steering the timely implementation of the project’s work plan. When technical difficulties are encountered, CAPTAC-DR experts support to their counterparts at BCRES in order to overcome these problems. This project is carried out within the context of a CAPTAC-DR implemented regional initiative, the Regional Program for the Harmonization of External Sector Statistics (‘PRAESE’ in Spanish). The project’s outputs mostly consist of trained technical staff and new compilation techniques. A third output is a series of recommendations to allow this work to be carried out in a more conducive environment. These outputs contribute towards improving capacity at BCRES to produce sound external sector statistics while also converging towards closer regional harmonization. The legal framework is now in place and over 90 TA recommendations have been implemented, of which 25 were considered top priority. Most of these recommendations deal with improvements in the database and of the data analysis.


25 | P a g e

We rate Relevance of the project as Excellent (score 3.6), reflecting its close alignment with CAPTAC-DR’s, national and regional objectives. Strong ownership has been demonstrated by BCRES. With respect to Effectiveness, our rating at the Outcomes level is also at the top level of the good category (3.4) reflecting the probability of attaining the project’s outcomes. About two thirds of the priority recommendations have been implemented and as a result we consider the progress made sufficient to provide a strong rating on effectiveness of outcomes. The high quality of advice and the seamless flow of technical missions and regional workshops are reflected in a rating of Excellent (3.6) for Outputs. Efficiency is again rated as Good (3.2), reflecting solid organization. There is still room for improvement on cost efficiency in running and organization of workshops. Sustainability is rated as Excellent (3.6), because the caliber of BCRES staff suggests that the majority of the advice delivered so far is likely to be retained. Also, the new techniques introduced do not require additional resources. As project implementation continues, prospects for sustainability are likely to improve further. 4.7 DESKTOP REVIEWS Regional: National Accounts This intervention has helped modernize the implementation of National Accounts across CAPTAC-DR’s member countries. By introducing modern statistical methods, this TA has changed the way business is conducted in a number of central banks and/or statistic bureaus in the region. Through a series of advisory missions and training events, the Center has helped its counterparts produce higher quality National Accounts Statistics (‘NAS’), while also making them more comparable with those of other countries and at different points in time. The Center has made available detailed information to its member countries in this technical area. Furthermore, it has provided hands-on training focused on real-life situations, both through highly practical exercises or learning the use of highly specialized software. Throughout the implementation of these activities, technical advice has consistently followed the prescriptions of the IMF on various account types (for example Quarterly National Statistics). Finally, training events have been further enriched by sharing the experience of other countries in the region and beyond, particularly on current practices and lessons learned. CAPTAC-DR TA has been extremely Relevant with a score of 3.7 (Excellent). The TA has also been Effective (with scores for Outcomes and for Outputs respectively at 3.4 and 3.6). The project is also deemed Efficient (3.3) and Sustainable (3.2), both marks being within the top half of the Good category.


26 | P a g e

Regional (CRI, DOM, & GTM) Support to Monetary Operations and Liquidity Management The Center has provided valuable TA in order to help counterparts in Costa Rica, the Dominican Republic and Guatemala to (i) better manage short-term liquidity; (ii) develop and use macroeconomic models for forecasting purposes; and (iii) strengthen the deepness and the overall functioning of foreign exchange markets. Given the complexity of all three technical aspects addressed, progress hinges on both counterpart institutional capacity and their commitment to carry out the necessary reforms. The Center has made a strong contribution toward maintaining momentum for reform. Achievements are already substantial, but a long-term approach will be necessary in order to attain such a deep and varied set of objectives and to ensure that gains are consolidated in the long-run. Relevance is rated as Excellent (3.8). All other criteria are marked within the top half of the Good category with a score of 3.2 for Sustainability and 3.3 for Efficiency. Effectiveness is also deemed Good with 3.4 and 3.2 points respectively assigned to Outputs and Outcomes. El Salvador & Honduras: Large Taxpayers Units (‘LTUs’) The original large taxpayer list was reduced to 600 in El Salvador (accounting for about 58 percent. of total tax collection) and to 621 in Honduras (12 percent of GDP). In both countries, the volume of information available about these large companies is systematically growing. CAPTAC-DR has also supported both countries in implementing a number of key recommendations. They revolve around the determination of an optimal organizational structure and operational processes for their respective LTUs. In El Salvador the LTU has now been set up, and work processes are currently being optimized. A large taxpayer support center has been established with adequate facilities and equipment, which has been funded through bilateral aid channels with the USA. The Honduran LTU is already operating in a new building with adequate overall facilities (e.g. computers).10 Honduras has also made significant progress in updating its audit manual. The sector files are expected to be prepared in both countries soon. The ratings are: Relevance, Excellent (3.7); Outcomes, Good (3.3); Outputs Good (3.5); and Efficiency Good (3.0). Sustainability is marked at 3.1 (Good). Nicaragua: Tax – Customs Information Sharing Since its inception, CAPTAC-DR has maintained frequent contact with Nicaragua’s tax and customs administrations through a number of channels. The Center’s diagnostic identified the need to achieve concrete synergies between DGI and DGSA, through enhanced information sharing and joint operational collaboration. In line with international best practice, it was decided to operationalize this information exchange by means of virtual private networks (‘VPNs’), in order to allow each institution to keep autonomous information platforms. Isolating evasions is facilitated when comparisons of consistency can be easily made between customs declarations and tax returns.

10 Though they are part of the same team, units are operating out of both San Pedro Sula and Tegucigalpa.


27 | P a g e

CAPTAC-DR’s advice was directed at solving specific issues appearing during the early implementation, such as dealing with extraordinary or anomalous situations, optimizing VPN communications between both institutions and streamlining specific export/import procedures. Inter-institutional information sharing is already underway. The project’s relevance has been rated Excellent (3.6) whereas all other aspects where considered Good. Sustainability, Efficiency, Outputs and Outcomes were respectively rated at 2.9, 3.0, 3.5, and 3.3. Regional: Single Treasury Account This assessment has revealed a solid TA project, providing sound advice on a matter of great importance regionally. The advice provided has been of high quality and is well adapted to the needs of CAPTAC-DR member countries. A similar project was carried out in Costa Rica and the Phase II Program Document cites the success of this project as an example where Costa Rica could provide peer to peer guidance to other countries in the region that are implementing Single Treasury Accounts. At the same time, it is extremely important for local stakeholders to understand that Single Treasury Accounts improve control and transparency over the use of government financial resources. They can also bring large financial gains to the treasury while also improving overall efficiency as regards financial management. As with other CAPTAC-DR TA, results are affected by counterpart capacity and ownership of project activities. Implementation has advanced faster in some countries than in others. TA has been provided at each national level, though following a similar pattern of activities across countries. Delivery has relied on TA missions, which have often included a workshop element. One of the first activities has typically involved making a diagnosis and sensitizing stakeholders (such as individual Ministries). As a next step, the countries have established concrete roadmaps to achieve Single Treasury Account implementation. These roadmaps show various degrees of advancement, according to the reports reviewed. Relevance for this project was rated Excellent (3.8), whereas Sustainability, Efficiency, Outputs and Outcomes were respectively scored at 3.0, 2.9, 3.0, and 3.3. All these scores fall within upper- and mid-range of the Good category. 4.8 HIGHLIGHTS OF RESULTS FROM SURVEY OF TA BENEFICIARIES Responses were received from 35 participants, in respect of seven technical areas/projects. Survey questions covered a number of areas, ranging from CAPTAC-DR’s TA modalities, to TA planning and monitoring, as well as the participants’ views on the effectiveness of the TA. The survey responses need to be analyzed with some caution, as they can sometimes yield contradictory results. Nevertheless, we believe that the responses taken as a whole provide a reasonable overview of TA beneficiaries’ views on CAPTAC-DR’s approach to partnering with them, and the value they place on the TA. We present some of the highlights of our findings in this section.


28 | P a g e

With respect to designing of the TA, the survey found: 19 of the 34 respondents strongly agreed (and 32 agreed or strongly agreed) that there was a comprehensive reform

strategy covering their sector. 17 out of 35 strongly agreed (and all 30 agreed or strongly agreed) that CAPTAC-DR TA is assisting in one of their

country’s/institution’s priority areas. A similar number of respondents, more specifically 18 out of 35 strongly agreed (and all 30 agreed or strongly agreed) that the TA was carefully sequenced to ensure successful results.

21 out of 35 respondents strongly agreed (and all 33 agreed or strongly agreed) that CAPTAC-DR and their institution had worked in partnership to design and prioritize the TA they were receiving. However, a lesser number (15 out of 35) felt strongly that the process had made them feel ownership and commitment for the project (while three disagreed with the latter).

22 out of 35 respondents strongly agreed (and all 32 agreed or strongly agreed) on the importance of regional harmonization in their technical area, whereas a slightly lower number (21 and 29 respectively) considered that regional integration was an important objective for their institution. Roughly similar numbers of respondents (18 and 31) felt that CAPTAC-DR is helping their organization fulfill its regional harmonization agenda, whereas four participants did not agree with that assertion.

With respect to coordination and relationship of CAPTAC-DR TA with HQ: 14 out of 24 respondents mentioned that

their organization is receiving TA from the IMF in addition to CAPTAC-DR’s.

Only five respondents strongly agreed, five agreed, three disagreed, one strongly disagreed and one didn’t know that CAPTAC-DR is a distinct entity from the IMF. This reinforces the idea that TA partners have a mixed perception of CAPTAC-DR in respect to its closeness with the IMF.

Four out of 14 respondents did not know about TTFs.

With respect to CAPTAC-DR TA, Chart 6 opposite highlights the following: All but one respondents strongly agreed,

or agreed that TA advice provided is clear, practical and easily implementable.

There was a lower level of support for adequacy of follow-up after TA delivery (three respondents disagreed, two did so strongly and one didn’t know).

11 respondents out of 31 strongly agreed (and 21 agreed or strongly agreed), that they needed more continuity of advice and at more frequent intervals.

At the same time, 14 out of 31 respondents did not prefer having a longer-term TA advisor.


29 | P a g e

A significant minority of 12 out 28 respondents felt that regional experts had been more effective than international STXs.

Encouragingly, it was generally felt that CAPTAC-DR understands respondents’ countries and institutions (12 out of 30 strongly agreed, while 25 agreed or strongly agreed).

A similar majority feels that the Center does promote regional integration.

Some of the comments made in the survey are presented below11:

About the RAs and the experts: The RA has always been monitoring the execution of the recommendations provided by the experts during each one

of their visits. The consultant has shown adequate follow up to the priority TA subjects. Ensure that experts coming to our country already know our laws. The consultants have a vast experience and a high level of technical knowledge. In a particular case, an STX did not meet the expectations. Experts have assisted us on specific topics, which has been very useful. About CAPTAC-DR: I don’t know much about CAPTAC-DR. I know it is an organization that coordinates with the IMF and that the

resources come from that source. CAPTAC-DR responds quickly to TA demands. About TA projects: The duration of the projects is excessively short. It is complicated to define whether TA advice should be longer or shorter, or at more frequent intervals. This is

defined by the needs of each country. Having adapted well to this partly explains CAPTAC-DR’s success. The TA needs assessment should be updated. There is no appropriate alignment between outputs and outcomes. Some projects (for example in the fiscal area) have been suspended without major progress. There is no continuity within a structured, medium term plan.

On CAPTAC-DR’s training: It would also be an improvement to include better training components, as so far there has been none. The TA could be improved through virtual courses (e-training) specifically directed to each of the sectors involved. There is no evidence that the training workshops are related to, and are designed to strengthen the TA needs and

capacities.

11 In order to make the document easier to read, we have translated and, in some cases slightly edited, the comments without affecting their meaning.


30 | P a g e

About the respondent’s organization: We have adopted a series of management indicators. We understand our strengths and weaknesses but in some cases the solution does not depend on this office, rather

from other factors. Implementing most recommendations is viable, but not easy in all cases. In the case of external sector statistics, it is very important for us to follow the directives set by the international

organizations in charge of their coordination. Miscellaneous: Regional integration is not perceived with similar interest in all the countries. Important misalignments (between and

within countries) still persist. Support IMF’s effort and expand CAPTAC-DR to the whole Latin American continent. Include internships overseas. There is not much evidence of explicit measures to avoid duplication in the TA received. It is very important for developing countries to receive this type of support with specialists … Thank you very much

and I hope to continue receiving your support and to see our project fully implemented.


31 | P a g e

5. EVALUATION RESULTS FOR REGIONAL WORKSHOPS 5.1 OVERVIEW Workshops and seminars are a very important part of CAPTAC-DR’s capacity building efforts in member countries. These are divided broadly into two types: 1. Regional seminars/workshops, often covering broad topics, usually at a

relatively high level. Some of these workshops might be implemented in partnership with the IMF Institute.

2. In-house training workshops, at TA beneficiary institutions, usually to support the implementation of TA advice.

Workshops and seminars directly supporting TA project implementation in a particular country have been evaluated as part of the overall TA project. In this section, we present our evaluation conclusions in respect of a sample of regional workshops. The sample was selected on the basis of their linkage with the TA projects selected for case studies and desktop reviews (i.e., covering the same subject area). For all the workshops selected, where available, their briefing papers, end of workshop reports including participant feedback scoring, budget versus actual costs, and workshop agenda were reviewed. For the latter two workshops, a selection of presentations was reviewed for quality and relevance. In addition, participants in the selected workshops were invited to complete an online survey.

During FY2010-12, CAPTAC-DR conducted 56 regional workshops, training more than 1,438 participants (at an average of almost 26 participants per workshop). From July 2009 until December 2012, total direct cost for workshops and seminars was US$2.1 million. As Chart 7 shows (figures relate to FY2012 only), after the Caribbean Regional Technical Assistance Center (CARTAC) and AFRITAC East (AFE), CAPTAC-DR is the third largest RTAC investor in seminars and workshops as a proportion of its budget (with an outlay of 11.1 percent)

The 2012 Annual Report indicated that most of the regional seminars and workshops had been conducted in collaboration with other regional or international organizations including IADB, CIAT, ECLAC, WCO and World Bank. The IMF TA departments were closely involved in the delivery of 13 of the 28 events in 2012 and the IMF Institute delivered three generic level seminars. Of the 56 regional workshops held over the review period, 29 took place in FY2012; 21 in FY2011; and 6 in FY2010. 15 workshops were dedicated to various statistics-related issues (e.g. NAS, External Sector Statistics, etc.); while eight focused on Monetary Operations. In addition, seven regional seminars were held on each of Financial Supervision and PFM. Also, six and five events were respectively devoted to Customs and to Tax Administration. Finally, eight seminars on related topics were organized together with the IMF’s Institute for Capacity Development (ICD).


32 | P a g e

5.2 REVIEW OF SELECTED WORKSHOPS The regional workshops selected for review were:

Partici-pants No.

Budgeted Cost* US$

Actual Cost* US$

Cost* per Participant

US$

Partici-pant

Rating Tax Administration Mass Audit Processes, San Salvador, 20-21 February 2012 27 21,856 23,993 889 4.9 Customs Administration Risk-Based Customs Management, Guatemala City, 4-6 June 2012

25 33,024 28,485 1,139 4.9

Public Finance Management Public Accounting & Int’l Public Sector Accounting Standards (‘IPSAS’) Implementation, Santo Domingo, Nov. 3-4 2011

23 24,072 21,943 954 4.5

Financial Supervision Regional Experience on Consolidated Supervision of Financial Groups, Tegucigalpa 23-25 April 2012

25 26,910 22,183 887 4.9

Monetary Operations Regional Seminar on Liquidity Management, San José, 28-30 June 2012

13 8,793 3,852

276 N.D.

Regional Seminar on Monetary and Foreign Exchange Operations, Tegucigalpa, 21-25 May 2012

23 30,990 30,017 1,305 4.5

Statistics National Account Statistics, Antigua Guatemala, May 21- June 1 2012

20 65,990 63,568 3,178 4.9

Closing of Phase I of the Regional Harmonization of External Sector Statistics Project, Guatemala, 28-29 Sept. 2011

16 21,209 10,506 657 4.9

Excludes indirect costs (e.g. RA time, backstopping and HQ incurred costs). See comments in this section. Source: CAPTAC-DR Our review of available documentation suggests that the workshops are well designed; are implemented effectively; have focused agendas; and provide high quality knowledge to participants. Direct cost per participant each day for the above seminars was on average US$299, a relatively low figure for high quality training (although this excludes HQ incurred direct costs). The participant ratings of the workshops selected are exceptionally high, averaging almost 4.8 out of 5.0 points. Our assessment of the regional workshops is consistent with the opinions expressed by respondents in relation to TA (see Section 4). Our conclusions are presented in Sections 5.4 (KEQs and Evaluation Ratings) and 5.5 (Findings and Recommendations).


33 | P a g e

5.3 RESULTS OF WORKSHOP PARTICIPANT SURVEY A total of 54 (31.2 percent of attendees) respondents completed the survey. The number of respondents ranged between 10 (37 percent) for Mass Audit Processes, to 1 (8 percent) for the Liquidity Management workshop. For most results responses have been aggregated by technical topic, which means that Liquidity Management has often been analyzed together with Monetary & Exchange Operations. Conversely National Account Statistics is aggregated with the workshop on closing the PRAESE project (which deals with External Sector Statistics). The survey results have indicative value only. Workshop Design 49 percent of the participants felt that there was an explicit needs analysis conducted prior to organizing their

workshop whereas 39 percent did not know. For particular subjects, the positive answers ranged from a high of 66.7 percent for both PFM and Monetary Operations, to a low of 12.5 percent for Financial Supervision. The confirmation rate ranged between 40 and 62 percent for the remaining workshops.

A very high 90 percent of the respondents confirmed that the workshop was pitched at the right level for participants, with a 100 percent response rate for Tax Administration and for Monetary Operations. The main exception to this rating was for the Public Accounting workshop, as half of the respondents considered that the technical level was pitched too high. This is an endorsement of the procedures CAPTAC-DR uses to attract the appropriate level of participant: participants are selected by each of the beneficiaries of the workshop following on from an invitation letter sent by CAPTAC-DR which profiles the ideal candidate.

62 percent of respondents confirmed that they were receiving TA from the CAPTAC-DR or another donor in the area

covered by their workshop whereas 19 percent didn’t know. These responses were highest for PFM and Tax Administration (respectively 75 and 70 percent each), and lowest for Monetary Operations and Financial Supervision (with only half of the respondents confirmed this to be the case). For the latter, the remaining half of the respondents did not know the answer to this question.

Generally, these results suggest that although there is some mixed feedback, overall the workshops selected for our review were well targeted and aligned to TA/reform initiatives in the respondents’ countries, though there might potentially be some overlap with other donors.

Workshop Benefits Table 8 highlights the top three perceived benefits to participants from the reviewed workshops.

Benefit Tax Customs PFM Super- vision

Monetary Ops.

Statis-tics Overall

Theoretical/technical knowledge 2nd 2nd 3rd 1st 2nd 1st 2nd

International best practice 1st 1st 1st 2nd 2nd 2nd 1st

What is happening in the region 2nd 3rd 1st 1st 3rd

Networking

Apply at my workplace 2nd 2nd 3rd 4th

Career progression


34 | P a g e

Respondents’ rankings of the relative benefits of attending a CAPTAC-DR workshop show coherence. Learning about international best practices in their technical area was cited as a top three benefit by 75 percent of the respondents (40 answers). Acquiring theoretical/technical knowledge was cited by 66 percent (35 responses) whereas updates on regional developments commanded 60 percent of the replies (i.e. a total of 32). These results suggest that CAPTAC-DR workshops impart solid and useful knowledge. One might also interpret that there could possibly remain room for improvement on ‘soft’ learning outcomes such as networking. Nevertheless the value of meeting fellow officers from other countries in the region has been systematically highlighted in the respondent’s comments to this question. Workshop Delivery Regarding the delivery of the seminars, 17 out of 51 respondents felt that they covered too much theory and should have been more practical (out of which seven strongly agreed with this assertion). A very high proportion (over 90 percent) agreed or strongly agreed that the presenters and their presentations were of very high quality and that the workshops were well organized. These results are compatible with the participants’ post-workshop evaluation sheets reviewed. Of the individual technical areas, the participants to the workshops on Statistics had the highest proportion of highly satisfactory ratings in respect of both presenter and presentations quality. The Risk-Based Customs Management workshop (75 percent) and the Public Accounting training event (67 percent) were the only two technical areas where a majority agreed to a statement on having developed action plans. Conversely, Monetary Operations (67 percent) and Statistics (62 percent) were the two topics for which a majority felt that they would have most benefited if action plans had been produced as part of the workshop.


35 | P a g e

Implementation of Workshop Lessons 26.5 percent of the survey respondents had been able to implement less than a quarter of the concrete actions suggested in the workshop at their workplace, whereas 40.8 percent had been able to implement almost 75 percent or more. In line with the findings of previous evaluations, the principal barriers to implementation were identified as: Lack of budget/financial resources (38 percent); The suggestions do not relate to my day-to-day job (17 percent); Changes dependent on other peoples’ actions (17 percent); and Not practical to apply lessons given level of institutional development (14

percent). This demonstrates the limitations in relying purely on workshops for delivering TA recommendations, and the need to target senior managers to develop the consensus for reform, which the workshop participants can then implement, based on the workshop contents and supported by TA. A third of participants felt there was follow-up from CAPTAC-DR staff on implementation of lessons learnt, 24 percent hadn’t, and eight percent did not feel it was necessary. 32 percent did not know if there had been follow-up. Of the technical topics, such as Statistics training, 58 percent of respondents strongly agreed that there had been follow up (with a further 17 percent agreeing and no one expressing disagreement). At the other extreme, in the case of Monetary Operations, though half of the respondents agreed that there had been follow-up, none of them strongly agreed with this assertion. General comments from the respondents Below is a selection of general comments from the workshops survey: I have improved my knowledge and established

relationships with colleagues from the region. A follow-up system needs to be established. Give continuity to the subject to assess progress. Complement with videoconferences or on-line. This training is a great benefit for our countries. Best experience, due to what I learned and to the

cordiality of participants, instructors & support staff. All the participants mentioned experiences very

similar to those from our country.

Thanks for the technical support they give to the countries that need it most.

Please improve the translations. Otherwise I congratulate you for an excellent seminar.

Impeccable logistics, excellent preparation. Consider hotel costs until the end, as I had to leave

to empty my luggage from the room. Some instructors shared experiences via e-mail. Excellent + the course should be longer. The organizing committee has done a great job.


36 | P a g e

Respondents’ Overall Assessment of Workshops Overall, over 96 percent of the respondents were satisfied with the workshop (half of them being highly satisfied). Recurrently suggested improvements from survey respondents included: Increase the duration (most common comment). Create a permanent liaison point for follow-up. Ensure that participants have previous preparation. They should remain interactive. More practical (extremely common comment). Evaluation form with more closed-ended questions. Increase frequency. More time to discuss in-country application. Develop practical exercises using spreadsheet. Invite the same participants for a follow-up event. Better sound systems. Ensure that participants work on the subject. Ensure that speakers are proficient in Spanish or

speak clear English. Elaborate cases using examples from the region. Improve simultaneous translation. Go into more detail. That the host country shows its current situation. Provide material before the seminar. Avoid excessively long days.


37 | P a g e

5.4 EVALUATION KEQS AND RATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities

Workshops are generally considered to be an important part of the Center’s strategy and annual work plan and those selected for review were consistent with the Program Document and approved plans. The workshops are generally in subject areas that are considered to be at the core of the national governments’ reform agendas. Positive responses overall suggest ownership and commitment. Our assessment of the TA projects in the same subject area also reinforces his view.

Consistency with IMF Headquarter/other activities This is the case. Workshops generally focus on practical issues, and where policy input is required and in some other instances, IMF HQ participates actively.

Coordination with Development Partners This happens to a reasonable degree, with a number of workshops being held with participation of other TA providers. There is no evidence of proactive efforts to connect with other donors active in areas covered to draw on their experience.

Context of donor/TA landscape of CAPTAC-DR TA Usually the workshops are linked to CAPTAC-DR TA projects and are considered to be practical, participative and of high quality.

Score for relevance 3.7 Excellent EFFECTIVENESS Impact NA Outcomes Intervention’s outcomes achieved or likely to be achieved (as defined in its logical framework): Short, medium and long term outcomes.

Expected through TA projects or implementation of lessons learnt. Workshop topics link with TA programs, although they may not cover all countries. Follow-up of workshops, unless the country already has a project in the subject area, has been highlighted as a weakness, and the CAPTAC-DR has no provision for mentoring.

Significance of CAPTAC-DR’s contribution/likely contribution to developing core economic functions and institution building in the country/region, through the activity.

General capacity building, especially when reinforced through linked TA. There is however evidence of positive change at the participant’s institutions after they participate in the workshops (although survey evidence indicates substantive barriers to implementation of workshop lessons).

Score for outcomes 3.5 Good Outputs Intervention’s outputs achieved or likely to be achieved (as defined in its logical framework), including: o Timeliness of reports/workshop presentations; o Quality of reports/workshop presentations; o Appropriateness/applicability of advice, given

existing capacity/constraints; o Whether outputs are likely to produce intended

outcomes.

Our review of presentations of selected workshops shows these to be of very high quality. This view is reinforced by participant feedback. However, there has been consistent feedback that most workshops are too short to appropriately absorb all the lessons. Moreover, many workshops that do not produce action plans or have no linked/follow up TA are likely to produce less impact. The extensive list of workshops organized in FY2011-12 was wide ranging and of high quality.

Score for outputs 3.7 Excellent EFFICIENCY Process and implementation efficiency Generally well designed and timely; logistics and organization are rated very

highly. Appears that there is emphasis on a participative approach and peer knowledge sharing. The program is well coordinated and planed as between CAPTAC-DR and HQ. Workshops are well pitched to participant level. Lack


38 | P a g e

Key Evaluation Questions Comments of action plans/follow up embedded in workshop design.

Efficient use of resources (human and financial) and attention to cost effectiveness

Solid expertise. Language issues occasionally highlighted. Operating close to budget. Cost effectiveness appeared good. However, all costs (e.g., RA time and HQ incurred costs), are not included in the analysis.

Monitoring and reporting Limited evidence of post workshop monitoring of implementation of lessons learnt. TAIMS is not being used for these purposes. It is difficult to apply RBM to workshops on a stand-alone basis. No evidence of formal extraction/usage of lessons learnt.

Score for efficiency 3.5 Good. SUSTAINABILITY Sustainability of TA activity Sustainability of workshop lessons on their own, without linked reforms/TA,

are difficult to assess. Hard to assess evidence that feedback is widely disseminated or that institutional capacity is sustainably strengthened: the workshops and TA projects are mutually supportive. Nevertheless, there appears to be an eagerness to learn and absorptive capacity seems good.

Contribution to building sustainable regional TA, implementation capability.

No strategy for linking up with local/regional institutions, such as e.g., to build local capacity. Regional presenters are usually staff from beneficiary/other institutions who are presenting their experiences. In our opinion, this is a major weakness.

Score for sustainability Not Demonstrated Rating Based on Previous Evaluation Methodology

DAC

criteria Sub-criteria

Sco-re

Weight %

Weighted Score

Weight %

Total score

Rating

Relevance Consistency with Program Document and Government Priorities

3.8 60% 2.28 32% 1.20

Consistency with IMF Headquarter/other activities

3.8 20% 0.76

Co-ordination with Development Partners 3.6 20% 0.72 3.76 Excellent Effectiveness Impact - 28% 0.99 Outcomes: TA 3.5 30% 1.05 Outcomes: Regional capacity building 3.3 30% 0.99 Outputs 3.7 40% 1.48 3.52 Good Efficiency Process and implementation efficiency 3.8 40% 1.52 22% 0.80 Efficient use of resources (human and

financial) and attention to cost effectiveness 3.8 40% 1.52

Monitoring and reporting 3.0 20% 0.60 3.64 Excellent Sustainability Sustainability of RTAC/TA activity 0 75% (0%) - 18% 0.43 Contribution to building sustainable regional

TA, implementation capability 2.4 25%

(100%) 2.40

2.40 Good

TOTAL 3.42 Good


39 | P a g e

6. ASSESSMENT OF CAPTAC-DR STRATEGY 6.1 INTRODUCTION AND OVERVIEW CAPTAC-DR’s strategic approach, in common with the other RTACs, is a relatively unique TA delivery model in that it seeks to balance: Member countries’ TA needs within its remit; IMF’s strategic priorities in the CAPTAC-DR member countries and the region; and Donors’ development objectives. CAPTAC-DR’s Phase I strategy was succinctly phrased in its strategic logframe12: “The Strategic Objective: Improved institutional capacity of the beneficiary countries and strengthened financial and economic regional integration. The overall objective of the CAPTAC-DR is to help the beneficiary countries achieve their long-term development goals by offering technical assistance and training to improve their institutional capacity in macroeconomic management in the IMF's areas of specialization, and, at the same time, to strengthen economic and financial regional integration, in close cooperation with the other providers of technical assistance to the region”. Some of the other RTACs, in common with many other TA programs, also include growth and poverty reduction as part of the statement, but it should be clear that these objectives are implicit in CAPTAC-DR’s strategy. At a more detailed level CAPTAC-DR has run a very clearly focused operation, possibly helped by clarity of TA demand arising from its careful national consultations but also regionally with such bodies as CAMC and CCSBSO and a strong early cooperation with IADB. CAPTAC-DR plays an important part in the IMF’s wider strategy to deliver demand-driven TA to its member countries. The IMF delivers strategic TA from HQ and CAPTAC-DR uses largely local expertise to deliver on-the ground capacity building. CAPTAC-DR has developed in a short time a track record of being a well-functioning and successful operation whose activities are owned by country authorities.13 We revisit the overall objectives outlined in the Program Document below, and in the remainder of this section review and comment on various aspects of CAPTAC-DR’s strategic approach. 6.2 ALIGNMENT OF CAPTAC-DR AND SC OBJECTIVES FOR THE CENTER A survey to SC members asked them explicitly about their goals and priorities as regards CAPTAC-DR. The responses were somewhat convergent, but still showed some nuances between donors and member country representatives. 80 percent of donors considered the maximization of the Center’s developmental impact as its very important priority (40 percent for member countries). In addition all donors considered very important (40 percent) or important (60 percent) to focus both on countries most in need and on countries with less access to other TA. All member countries either agree with these assertions (80 percent) or strongly agree (remainder).

12 2012 CAPTAC-DR Annual Report. 13 Much of the above has been summarized from the 2009 Program Document.


40 | P a g e

CAPTAC-DR member countries also consider that equal distribution of TA is an important issue (and more so than donor representatives). All country officials agreed (60 percent) or strongly agreed (40 percent) that resources should be distributed equally among technical topics. At the same time, 80 percent of them felt that support should be provided equally to all countries. These results indicate that the SC has a broad range of preoccupations when considering the Center’s goals, though the developmental impact of its work does remain an overall priority. Chart 10 below depicts the distribution of SC members answering about particular issues that they consider are ‘very important’.

At the time the survey was held, the majority (60 percent) deemed the Center had sufficient resources to attain its goals across member countries. All member country representatives and half of the donor respondents considered that priorities needed to be refined in order to ensure that strategic goals are achieved. In general, opinions about

resource availability and its implication were diverse both between and within survey groups. Because of the latest developments regarding the discontinuation of the Spanish contribution to CAPTAC-DR (20 percent), the above perceptions may have changed already. Member country officials most frequently cite access to another source of training and TA in general, and from the IMF in particular as the main reason to participate in CAPTAC-DR. Donor respondents, in turn, have a wide set of reasons, without any of them gaining preeminence. Finally, a large majority of respondents felt that the Center is delivering its outcomes as expected or better. 6.3 ACHIEVING THE ADVANTAGES OF CAPTAC-DR’S TA DELIVERY APPROACH Our survey results, our findings from case studies, project desk top reviews and interviews point to the proven advantages of the CAPTAC-DR approach. It is responsive, focused on capacity building, flexible to changing circumstances, and able to develop close and continuous relations with TA recipients which engender trust in and ownership of projects. CAPTAC-DR is not the only TA resource targeting Central America on a medium to long-term basis, but uniquely operates on a completely regional basis, engaged only in provision of TA unrelated to lending, from a regionally located office, covering a wide range of administrative capacity building on macroeconomic and financial sector issues. CAPTAC-DR is also not the only TA provider in the topical areas it operates but does bring into one structure a whole range of macroeconomic expertise: public finance, revenue administration, financial sector supervision, and macroeconomic statistics. Collectively all these topical areas form part of the core expertise of the IMF (and CAPTAC-DR). CAPTAC-DR, like other TA providers makes use of local expertise, though the Center is the only provider of TA in this area with a permanent team of RAs based in the region, each covering one of the topical areas. CAPTAC-DR does not have a monopoly on any particular topical area of expertise. CAPTAC-DR does however have a significant experience edge on all of them, which is leveraged by collaborating with other donors/TA providers, as listed below.


41 | P a g e

Macroeconomic Statistics – IMF is the leader in collating and disseminating macroeconomic statistics but also

CAPTAC-DR cooperates actively with the Economic Commission for Latin America (‘ECLAC’) and with the Central American Monetary Council on regional seminars and avoids any risk of duplication with IADB.

Tax Administration – on a global basis IMF/CAPTAC-DR is one of the most important sources of tax administration TA while also cooperating with other TA providers, especially IADB, GiZ and in regional seminars with the Inter-American Center of Tax Administrations (‘CIAT’) and the World Bank.

Customs administration – similarly the IMF is not only a global leader in customs administration TA but also cooperates with other donors and organizations including the World Customs Organization and IADB and to some extent with the World Bank and USAID.

Public Finance Management – on a global basis IMF only contributes about 5 percent of all TA in this area, but is well focused in Central America. In addition, this work is complemented by closely cooperating with donors such as US Treasury, GIZ, USAID, IADB and the EU.

Supervision of Financial Sector (mainly Banking) – mostly cooperating with IADB, World Bank and CABEI and with the regional supervisors council, CCSBSO.

The RA role is a unique modality provided by CAPTAC-DR and the RTACs. There are long-term advisors, as distinct from CAPTAC-DR’s resident advisors, operating in some areas and countries in the region whether employed by IMF, World Bank or other organizations, but these tend to be anchored in a particular area and country. The long-term in-country resident advisor role is often unsuitable for certain TA purposes for a variety of reasons including the absorptive capacity of the beneficiary. CAPTAC-DR’s RAs step in at intervals appropriate to the case and maintain a role and brief over an extended period. 6.4 PROGRAM DELIVERY COMPARED WITH ANNUAL WORK PLAN We have summarized the strategy in section 6.1 above. The general direction of the work plans and the project selection follows the broad areas set out in the Program Document. Table 9 below is constructed by taking the customs area as an example for FY2011:

Country Program

document All countries

Work plan 2010-11 Input weeks Actual work 2011-12 Input

weeks

Dominican Republic

Reform customs control procedures Develop post release audits Develop risk management techniques

Strengthen customs supervision with emphasis on post-shipment oversight, use of risk management techniques, and enhanced integration with domestic taxes. Develop mechanisms to strengthen supervision and facilitate trade at the port of La Haina.

7 Strengthen IT systems; Update Audit Plan.

7

Costa Rica Strengthen post-shipment oversight with a comprehensive, strategic view of supervision to be used as a basis for the development of the annual oversight plan and the risk management model. Broaden use of risk management techniques and tools to enhance selectivity and results of operations or oversight activities.

8 Air/ocean freight controls to lessen smuggling; strengthen IT systems; monitor new performance indicators.

12


42 | P a g e

Country Program

document All countries

Work plan 2010-11 Input weeks Actual work 2011-12 Input

weeks

El Salvador

Reorganize the customs oversight service with a comprehensive, strategic approach to supervision, focusing on risk management and enhanced integration with domestic taxes. Develop mechanisms to strengthen supervision and facilitate trade in connection with land border clearing.

9 Improve selection criteria.

Progress negotiations with Honduras; strengthen IT system.

9

Guate-mala

Reorganize the customs oversight service with a comprehensive, strategic approach to supervision, focusing on risk management and enhance integration of domestic taxes. Develop mechanisms to strengthen supervision and facilitate trade in connection with land border clearing.

8 Selection methodology in ex-post audits; strengthen planning and management; customs controls and facilitation.

8

Honduras Review customs processes subject to a comprehensive, strategic view to strengthen primary supervision, to be used as a basis for the preparation of the annual a posteriori oversight plan and the risk management model based on more effective integration with domestic tax activities. Increase use of risk management techniques to reduce selectivity indices in customs clearance and improve assessment results.

9 Risk Department organize selectivity profiles; Strengthen planning and management, customs controls and facilitation.

9

Nicaragua Strengthen supervision based on analysis of processes with a comprehensive overview and greater integration with domestic taxation, use of risk-management techniques, and a posteriori inspection.

8 Review technical circulars and publish in web site.

9

Panama Support the self-assessment process of the national customs authority in connection with the application of international best practices shared at the regional level and the requirements deriving from this process. Review the customs procedures and processes based on the functionalities of the new computerized customs management system, with a comprehensive, strategic view to strengthen primary customs supervision and facilitate trade.

4 Strengthen IT systems; identify various improvements for prevention /mitigation of risks. Strengthen planning and management, customs controls and facilitation.

4

Regional Support Customs Union through CACU

Support COSEFIN action plan. Develop action plan for administrations in line with Council of Ministers of Economy & Trade (COMIECO) integration objectives. Seminars.

16

11

22

Total 80 80

The table above illustrates a few points: The Program Document is very broad and allows considerable flexibility to adapt work plans year by year in response

to needs and demands. There is no financial budget analysis for the five years by topical area and country to provide some context for

measuring against annual TA delivered and annual work plans in financial terms.


43 | P a g e

The annual work plan for FY2011 is fairly specific and planned by RA or STX weeks required in a logframe format. The description of actual TA delivered is much less detailed, is contained in the narrative as opposed to the logical

framework part of the Annual Report, and (in customs administration in this case) it is difficult to compare the actual TA delivered with that planned a year earlier.

The total number of weeks on customs administration ended up as originally budgeted with five weeks less on regional TA (customs union initiative faltering) and five weeks more in two of the countries.

Whilst the process of annual work planning helps to adjust the implementation of strategy in a flexible manner when faced with changing circumstances in the region, the lack of a five-year implementation plan expressed in terms of timetable and cost makes it difficult to track progress against the strategy, except perhaps once the five years is completed. A similar difficulty of comparison arises when comparing the annual work plan with actual given that the latter is more vaguely described than the former. Monitoring the strategy would be easier if a five-year plan of interventions, outputs and outcomes had been projected year by year, topic by topic and country by country, rather than only a very broad logframe for the five years taken as a whole. In practice there will be perhaps substantial change between actual and planned over a five-year term. Whilst the annual work plans provide good detail on activities, outputs and outcomes the time frame is not the same as for the strategy. At the moment the strategic logframe is too broad to be of value. The quality of TA delivered appears high from our assessment of case studies and desktop reviews, as well as survey responses. Nonetheless, there could be some lessons to be learned from the experience so far in Phase I. For example, has the momentum on regional integration slackened to some degree? This appears to be the case with customs union. Despite apparent TA demand should other areas of financial sector risk based supervision than operational risk have been allocated greater priority? The fact that CAPTAC-DR has been able to deliver high quality TA despite an apparently lower level of backstopping and project management than budgeted in Program Document begs the question as to what the level of these functions should be in the future. As a percentage of RA and STX costs backstopping and project management were estimated in the Program Document budget at 21 percent. The percentages for FYs 2010, 2011 and 2012 were 21, 14 and 14. The volatility of backstopping and project management may be unavoidable because there are some less predictable variables in any one year, such as quality of RAs and STXs, turnover of RAs, complexity of issues and probably some procedural issues on charging back and so on. Nonetheless, and because quality control is such a strong and important feature of the IMF’s TA operations, at the least these swings in performance against budget need explaining when they arise. With better RBM procedures in the future it could be worth considering more decentralization of project monitoring to the CAPTAC-DR level, especially given that the annual work plans are largely constructed from the ground up. The introduction of the multi-year projects was a big step forward in AFRITAC West (AFW) accompanied as they have been with a results focus. A similar approach could be effective in CAPTAC-DR because many of its projects also span more than one year. This would seem a natural follow-on from the plan to provide longer-term capacity building TA. Overall, we believe that CAPTAC-DR is well on track with its strategy. Budget areas should be reviewed and balances struck between central versus local controls, and projects should be differentiated more clearly as between multi-year and short-term (less than one year).


44 | P a g e

6.5 IMPLEMENTATION CHALLENGES The main challenges relate to issues of sustainable capacity, funding of CAPTAC-DR and funding and provision of TA in areas not covered by CAPTAC-DR on particular topics. Under the heading of training, there have been two principal modes: regional seminars and workshops, and mentoring (on-the-job training). Not achieving a sustained strengthening of capacities is probably the largest risk because to some extent it is predictable and potentially manageable. Generally the institutional capacity is stronger among the CAPTAC-DR members than in some of the other RTACs but there remain divergences between members and the issues to address will vary. It is no coincidence that El Salvador is cited twice (tax and PFM) as a success story in the 2012 Annual Report and Costa Rica once (customs); other success stories in Supervision and Monetary Operations are for regional TA. In some of the desktop and case study interviews the issue of sustainable capacity related to human resources and management was raised. Much of the work in Monetary Operations, Supervision and Statistics has the central banks and supervision agencies as beneficiaries of the TA. By and large the HR issue is less of a challenge in these institutions. The main risk areas could therefore be with Tax, Customs and PFM and, according to the Center Coordinator, with National Accounts Statistics in some cases. HR policies and management need to be addressed in order to mitigate high staff turnover, settle appropriate remuneration structures, implement staff appraisal schemes, career paths, and development of institutional training capacity. Guidelines and manuals are useful tools for institutional memory and guidance of staff but not enough. There also seems to be a tendency to allocate TA fairly evenly among the member countries especially in areas that cover the same, or a similar topic as part of the regional harmonization effort. In some cases members with less absorptive capacity could be disadvantaged by this approach. The discussions of April 2010 in the region, which were mentioned in the Steering Committee minutes of June 2010, indicate that equal distribution is a desirable policy: “A more or less uniform distribution of available resources among all members”. Despite this last comment, the work plan for FY2014 does indicate a wider range of TA delivery than in the past especially in the fiscal area with Costa Rica at 37 planned weeks versus Panama at 19, reflecting the strength of Costa Rican demand and reform commitment. Both the monetary/financial and statistics areas are evenly spread. Funding of CAPTAC-DR has been only a minor risk affecting operations so far in Phase I. There is a clear risk to the adequacy of funding of Phase II now that Spain, which contributed 20 percent of the Phase I pledged funds, has announced its decision to withdraw. Continuing success of CAPTAC-DR should otherwise encourage the remaining team of donors to continue with at least a similar level of funding for Phase II. Provision of TA in topics not covered by CAPTAC-DR is important because some of the necessary interventions affect the sustainability of the Center’s contributions. As an example of a TA gap, a major host of financial holding companies in the region (and even globally) is Panama. The regulators there have been enthusiastic supporters of the objectives of the TA on Cross Border Consolidated Supervision but, for full cooperation, legislation is needed to provide them with appropriate powers (CAPTAC-DR is now providing some assistance, although this was not originally in the work plan for FY2013). Donor coordination and cooperation is essential to mitigate this type of risk. Other examples include projects where the achievement of higher-level outcomes requires automated systems, for which often CAPTAC-DR/IMF is unable to provide TA funding.


45 | P a g e

The logframes in use do have very broad risks sections. They tend to be repeated topic by topic e.g. “weak political commitment”. Whilst some of these risks are obvious they tend to be too generic to be of much value for project design and monitoring. There is little evidence that RAs conduct formal risk analysis before embarking on a TA initiative. It should help early identification of risks such as absorptive capacity, need for legislation, need for automation and identification of other gaps, if a formal risk analysis section was included in the logframes. CAPTAC-DR has been providing regional seminars and workshops at the rate of about 20 a year with a spike to 28 in FY2012 and a plan for about 20 again in FY2013. Mostly these have been well received for content and coverage, as well as the networking opportunities provided. We believe that CAPTAC-DR may be able to provide more mentoring TA where implementation capacity is weak, but the motivation for reform exists; short-term RA and STX missions should be, in our opinion, supplemented by more extended TA, either from CAPTAC-DR or in partnership with other donors. 6.6 REGIONAL VS COUNTRY-FOCUSED TA AND TRAINING There is a role for regional seminar/workshop training and country level TA. Both these interventions, if well designed, are mutually reinforcing. Much of the country-specific TA has also been set in a regional context albeit delivered country by country. For example, the medium term expenditure framework projects in PFM, the cross border consolidated supervision of the banking sector, the work on taxpayer segmentation, risk-based controls in customs, and external statistics under the PRAESE project. The country by country approach to deliver TA in the same or similar areas helps to build regional harmonization of standards, data and comparable statistics which prepare the ground for more regional integration when the political climate permits. It seems that regional integration is high on the agenda at rhetoric level but still facing major obstacles at implementation level.


46 | P a g e

Seminar/workshop topic Days In collaboration with

Participant Rating (out

of five)

Related country TA: Multi-year

Related country TA- by country

Customs Risk analysis systems 26 OMA, AFIP,

AEAT, SAT-MEX 4.8 Yes All 7 countries

Risk modeling, ethics & transparency 23 IADB 4.9 Yes CR, ESP, HON, PAN, Controls management 25 OMA, SAT-

Guatemala 4.9 Yes All 7 countries

Tax Overview on controls management and planning

25 - 4.9 Yes All 7 countries

Taxpayer risk profiles 27 - 4.9 Yes CR, ES, HON, PAN, DR PFM Public accounting and IPSAS 23 IMF HQ 4.5 Yes CR Public debt management 32 IMF HQ 4.7 No Program topic but not yet

delivered outside of public debt stats (external & domestic)

Working group on medium term expenditure framework (MTEF)

26 IMF HQ 4.8 Yes All 7 countries

Total 207

20 others in Monetary Ops, Supervision, Macroeconomic statistics, Institute courses

462

Total: 7 669 Source: CAPTAC-DR The regional seminars and workshops have been designed to relate to the country delivered TA as illustrated in Table 10 above for the period May 2011 to April 2012. CAPTAC-DR also runs three generic courses every year delivered by the IMF Institute: for FY2012 these were on basics of bank supervision, macroeconomic forecasting and financial stability/economic policy. 6.7 ROLE OF NATIONAL AUTHORITIES The role of the national authorities is critical to the work of CAPTAC-DR and is exercised at various levels. Each of the countries participates as members of the CAPTAC-DR Steering Committee. Seven of the countries are represented at very high level: Minister of Finance and Central Bank Governor. We have commented on governance in Section 10 of this report. The SC provides a forum for the national authorities to influence CAPTAC-DR strategy, to share experiences with their counterparts and to monitor CAPTAC-DR performance. All of the countries have committed to contribute US$250,000 towards the cost of CAPTAC-DR, which indicates good commitment. Guatemala is additionally providing a substantial commitment through hosting the Center’s offices in the central bank building and providing local staff. The determination of TA needs and priorities is an active consultative process between various IMF departments (WHD and TA departments at HQ plus input from CAPTAC-DR in the region) and the country authorities. The main counterpart authorities at policy level in the countries are usually the Ministries of Economy and Finance and the central banks.


47 | P a g e

Regional organizations are also very important in influencing the determination of TA needs especially those who attend the SC meetings: CAMC, CCSBSO and COSEFIN. More detailed project design, work planning and TA delivery involves active participation with the beneficiary organizations concerned which are typically below Ministry level e.g. customs agencies, taxation authorities, central banks, and statistics departments in central banks. At this level the beneficiaries’ role is to engage with the planned TA by allocating appropriate counterpart staff and carry out implementation of TA recommendations. The recommendations will usually be geared to their absorptive capacity and formalized in Missions Reports. After each stage of TA delivery the beneficiaries have the opportunity to comment on the reports and recommendations before they are finalized. CAPTAC-DR missions are commonly about two weeks of expert input (STX and/or RA). Implementation rates of agreed actions by the beneficiaries prior to follow on missions vary. Variation in performance highlights an absorptive capacity (or an ownership) issue in some institutions and or countries that requires flexibility in TA delivery. The national authorities and the particular beneficiaries of TA also have a role to play in donor coordination and cooperation in order to optimize the TA received, avoid duplication and identify gaps that are not being served. Performance in this role also varies between CAPTAC-DR countries; a potentially important tool to help countries to perform this role is the Aid Management Platform developed by the Development Gateway (formerly part of the World Bank and now an independent not-for-profit organization). Regional authorities also play an important role. CCSBSO are a strong counterpart to the TA on Cross Border Consolidated Supervision and Operational Risk and the CAMC similarly on topics such as Liquidity Management and Inflation Targeting. One of CAPTAC-DR’s roles is to provide TA that at the least helps to harmonize standards and procedures in the region. In most instances, ownership for a TA project is generated through the interaction between CAPTAC-DR (in particular, the responsible RA), and the counterpart institution. The latter’s role in requesting and prioritizing the TA is an important contributor to the ownership. The RAs are, within their constraints, also responsive to ad hoc TA needs and show flexibility in the timing of missions. 6.8 COORDINATION WITH OTHER IMF ACTIVITIES AND DONORS/IFIS The framework for CAPTAC-DR’s coordination with the rest of the IMF is provided by the RSN/RAP. The backstopper at HQ plays a key role in ensuring that TA activities are effectively organized in this regard. There have been several comments from individuals interviewed that coordination with other TA providers by CAPTAC-DR needs to be improved. Whilst there is always room for improvement, we believe that CAPTAC-DR has done a good job so far. We understand that for Phase II CAPTAC-DR plans to strengthen collaboration with other TA providers by building strategic working partnerships. The survey responses indicate that it continues to be regarded as a weak area by the donor members of the SC. One particular concern has been expressed over the timely release of mission reports especially to donors active in the same areas of TA. This process requires time to prepare mission reports, build in feedback from backstoppers and then allow for an internal review process, followed by a 60-day period, during which the beneficiaries may object to release reports to those who have signed confidentiality agreements. This time-consuming arrangement is seen as too protracted some times to be of benefit to complementary TA provided by third parties.


48 | P a g e

Coordination with other donors is managed through several channels: The CAPTAC-DR SC includes many of the important donors in the region. Meetings take place annually and whilst

the donor representatives are often from head office as opposed to from regional offices, the extensive reporting on project activity provides an opportunity for the donors to share that information locally.

CAPTAC-DR maintains a comprehensive website, which also summarizes project activity and, with the exception of tax and customs, the mission reports (we understand that tax will start posting its reports from FY2013).

CAPTAC-DR cooperates with other donors on several of its TA projects: in customs with the World Customs Organization (‘WCO’) and IADB; in PFM with US Treasury on Medium Term Expenditure Framework (‘MTEF’) in Dominican Republic, with GiZ, and USAID on MTEF and Single Treasury Account (‘STA’) in El Salvador; and in consultation with the EU and IADB in Nicaragua, Honduras and Dominican Republic; in Supervision mainly with IADB, World Bank and CABEI, with World Bank on RBS in El Salvador and generally on RBS with CCSBSO and CABEI; and on macro-economic statistics with SECMA, ECLAC and IADB.

Resident advisors communicate regularly with other donors in their areas. The Center Coordinator regularly visits key counterparts throughout the region, especially in the work planning stage,

and maintains close contact with key regional organizations, attending many of their meetings. Every SC report provides a section on donor coordination activities undertaken during the reporting period. Although the CC participates in donor coordination meetings, his activities are not specifically reported to the Steering Committee. The CC’s available time for coordination with beneficiary countries and donors is limited. We have recommended lightening his burden in more administrative areas in order to free up time for a fuller coordination activity. A range of options is possible for donor coordination. First, this work can start with information sharing then it can extend to making an effort to avoid overlaps with other TA providers; to ensure complementarity of TA on ad hoc basis; and then progress to reaching out proactively to other donors on specific interventions where TA is needed say to implement CAPTAC-DR recommendations. Finally, strategic partnerships could be formed with one or more TA providers to work together on a number of projects from the design stage. CAPTAC-DR has already collaborated actively with some other TA providers and plans to build more strategic working partnerships in Phase II. It should be a priority task for the beneficiaries of TA to drive the donor coordination process: in practice this does not always happen systematically. It may be worth considering introducing some of the central agencies in CAPTAC-DR’s member countries to the Development Gateway’s Aid Management Platform as a tool for systematically recording and disseminating TA from other sources. There is of course a trade-off between optimizing the benefits that could be achieved through perfect donor coordination and the complexity, time and risks involved. This will vary between donors and projects and the CC needs to make the appropriate assessment in arriving at the right balance. Partnerships with Regional Organizations CAPTAC-DR already works closely with important regional organizations. This work has yielded promising results.


49 | P a g e

Leveraging success stories There are examples of successful implementation of projects with good outcomes already achieved in certain countries. Sharing this experience with other countries in the region through presentations by the recipients of the TA to regional workshops or even by hosting secondments, could be explored as a way to leverage off local expertise developed through CAPTAC-DR TA. Lessons from the work of other RTACs should also be shared. 6.9 CAPACITY BUILDING The CAPTAC-DR Program Document highlights a number of initiatives that the Center undertakes to support capacity building in the region. We review the activity relating to some of these below: Training of Trainers Our case studies and desktop reviews suggest this tool is used in a limited way. In any event, we believe that CAPTAC-DR’s mission-based TA approach is not ideal to achieving this objective. In order to be effective, a minimum of two-to-three cycles of training of prospective trainers and mentored delivery is required, which is difficult within one mission. Nevertheless CAPTAC-DR has enough flexibility to be able to accommodate a Training of Trainers initiative, perhaps initially on a pilot basis, e.g. through greater use of STXs for this purpose. This approach could be a critical component towards achieving sustainability. 6.10 COMMENTS ON OVERALL PORTFOLIO APPROACH The Center’s TA and training portfolio has evolved in line with the Program Document as demonstrated earlier. The one exception has been in the capital markets area: there has been some TA in public debt but only very limited so far. The TA effort has been fairly consistently delivered by topical area in Phase I with little variation between years.

Statistics has maintained a robust percentage of TA in CAPTAC-DR as shown in Table 11 opposite. This area is a critical building block for effective macroeconomic management, complex but relatively precise and easy to deliver and monitor. Results can be very tangible. There can be resource issues to manage the data updating and collection through surveys.

Revenue Administration attracts the support of many donors but these tend to be more fragmented in their approach, serving special niches and often only in one or two countries as opposed to the entire region that CAPTAC-DR serves. The most difficult sector to make progress in is Customs, which is said to be particularly politicized. CAPTAC-DR has the presence and probably the tenacity to make the difference in this crucial sector (for eventual regional integration). At the country and regional level the consistency of TA allocation is slightly more mixed. The TA is demand driven but once allocated the prevailing policy of distributing TA in roughly equal proportions runs the risk that not enough is delivered to the weaker countries or where there may be a disproportionate temporary need. The other RTACs we have reviewed

Topical area Phase I average

(2010-2012)

Statistics 25%

PFM 18%

Revenue Administration 29%

Monetary Operations 28%

Source: CAPTAC-DR


50 | P a g e

appear more responsive to the needs of the weaker beneficiaries and their SC country members also appear sympathetic to an unequal distribution of TA. As financial sectors, especially in the leading countries in the region, diversify and consolidate, a more systematic approach is likely to be needed for ensuring sectors such as insurance and capital markets are more effectively served. It may be impractical to engage RAs for each of these, and consideration should be given to engaging STXs under call-down contracts to meet demands in these and other areas. 6.11 STRATEGY DEVELOPMENT FOR THE NEXT PHASE We believe that the next phase of CAPTAC-DR would provide an opportunity to continue with a robust and coherent strategy, the implementation of which could be monitored more effectively under the IMF’s RBM framework as strengthened by our recommendations. Our detailed recommendations are covered in Section 11. We believe that the following issues need to be considered, which have implications for both strategy development and the RBM framework, given the close inter-dependencies between them: Develop a costed five year strategic plan: budgeted by project and by year. Plan, monitor and report longer-term projects (more than one year) on a multi-year basis. Consider increasing the percentage of longer-term projects to about 50 percent. Consider allowing in where needed additional time for mentoring and on-the-job training in for countries with weaker

capacity. Consideration should be given either by CAPTAC-DR or, through the Center and its Steering Committee, to other

donors to provide specific TA to mitigate weak absorptive capacity and high staff turnover where it occurs. Use Results Chain for project design to establish TA sequencing or other pre-conditions (e.g. legislation); other TA

required not delivered by CAPTAC-DR and identification of other TA delivery sources; and any areas where donor coordination is required and project specific risks.

Assist where practical in the development of country level or sector level medium term development strategies. The IMF should consider developing a number of Key Performance Indicators (KPIs) (examples of some have been

presented in the Report) to monitor the relative performance among the RTACs.


51 | P a g e

7. CAPTAC-DR FINANCIAL REPORTING 7.1 INTRODUCTION AND SUMMARY This section reviews the costs incurred by CAPTAC-DR to-date, compares them with budget, and seeks to identify reasons for any variances. We also highlight some of the issues relating to the Center’s financial reporting matters. The separation of CAPTAC-DR’s systems from those at HQ creates a very complex system for recording relatively simple entries. Previous evaluations and reviews of other RTACs, using the same financial systems, have highlighted weaknesses in IMF’s management financial reporting system as an area that requires substantial strengthening. 7.2 DISTRIBUTION OF TRUST FUND EXPENSES

Chart 11 above highlights the main component of CAPTAC-DR’s actual costs incurred during Phase I from Trust Fund resources (as of December 2012) and, for comparison purposes, those of AFW during first half of its Phase III (i.e. FY2010 until end of FY2012). The totals were US$20.4 million and US$10.7 million respectively. The broad distribution of costs differs significantly between both RTACs. The share of RAs costs slightly exceeds a quarter of total outlays for CAPTAC-DR, whereas it reaches close to half for AFW. The latter makes in turn much less relative use of STXs (23 percent), against CAPTAC-DR’s 33 percent. Analyzed by ratio of RA time to STX time the difference is even more marked: CAPTAC-DR leverages off STX time at about twice the level of AFW. Workshops (and miscellaneous expenditure) also account for a larger proportion of total expenditure in CAPTAC-DR, with 11 percent (versus only eight percent). CAPTAC-DR’s office and start-up costs have to date been substantially lower than that budgeted.


52 | P a g e

7.3 ANALYSIS OF ACTUAL VS BUDGET COSTS Chart 12 below shows a breakdown of CAPTAC-DR’s actual versus budgeted costs until the end of FY 2012. Of the total budgeted costs (including trust fund fee and office costs) of US$20.7 million up to that point, the Center had incurred actual cost of US$16.2 million, which entails an execution rate of 78.2 percent. The largest positive variance was on office center costs (which include IMF and host country contributions) and if these are taken out in the comparison the execution rate rises to 83 percent. In dollar terms, under-expenditure was greatest for the following items: (i) Office Operations/Startup Costs (US$1.6 million); (ii) Backstopping (US$0.7 million); (iii) RAs (US$0.5 million); (iv) for Seminars and Training (US$0.5 million); and (v) project management (US$0.2 million).

In percentage terms the greatest variances were in Backstopping (59 percent); Office Operations/Startup Costs (48 percent); and Regional Travel (35 percent). Conversely, STX, Staff Travel, and RAs were the most balanced in percentage terms (respectively at six, eight, and 11 percent). None of these variances are very critical when considering that reaching full operating capacity took longer than expected. There were considerable recruitment delays which lasted to various degrees until June 2010 when the last RA was

hired. In fact, six months after CAPTAC’s inception, only two RAs had already been appointed. This delay mainly resulted from the difficulties in finding professionals with the appropriate profile, despite a sustained search effort. Of the aggregate under spend of about US$4.5 million, 97 percent of it arose in the first two years. During FY2012 the Center was operating at almost 100 percent of that year’s annual budget.

Backstopping and Project Management Budget vs Actual for HQ Departments

in US$’000 Budget Actual Difference Percentage Backstopping

FAD 798 350 448 56% MCM 187 75 112 60% STA 262 86 176 67%

Project Management FAD 201 160 41 20% MCM 101 33 68 67% STA 89 250 (161) (280%) ICDGP 167 46 120 72% FIN 56 34 22.0 39% WHD 71 0 71 NA Institute 22 20 2 91% Source: CAPTAC-DR


53 | P a g e

In general the IMF’s introduction of actual cost charging (as opposed to standard cost) in 2009 added an element of accuracy when calculating expenditure. If the project management and backstopping budget lines were excluded, expenditure variances would be significantly lower. CAPTAC-DR is the first of the IMF’s RTACs to introduce this costing model. As such, the learning curve for all of the IMF has been steep. It is expected that in the next CAPTAC-DR phase, budgeted amounts will more accurately forecast actual expenditure for both backstopping and project management. More specifically, Table 12 above shows the gap between budget and reported actual costs for HQ backstopping and project management for the different HQ departments during FY2010-12 (to end April 2012). Other than Statistics and the IMF Institute significant portions of the allocated budgets have not been utilized. In part this is explained by the slower than expected build up to full capacity. As stated elsewhere, this could be due to several reasons or a combination thereof: over-budgeting, under- charging, the Center RAs and STXs not needing or receiving as much oversight, or resources were stretched at IMF HQ. We do recognize that the backstopping process is a very important part of the IMF’s quality control of TA delivery and provides the IMF with a strong comparative advantage as compared to some other TA providers. The budget for backstopping for the period May 2009 – December 2012 was 26.7 percent of RA costs, given that STXs are mainly back stopped by the RAs, this seems on the high side; the same ratio for actual costs during that period was 11.4 percent. Certainly in our view the TA quality remained of a very high standard. In reviewing the CAPTAC-DR financial information, we came across the following additional issues that are worth highlighting (figures referred to are for FYs 2010-2012, unless otherwise stated): Spending on STXs has increased significantly in FY2012 (US$2.9 million) from FY2011 (US$1.6 million). In FY2012,

spending on STX was roughly similar to outlays on RAs. During the following fiscal year the proportion had risen to 1.5:1. This suggests ever more effective leveraging of RA time which should impact favorably on efficiency.

We could not easily relate the connection of the financial results to the work plan (which is shown in units of time) against the financial budgets. For example, in FY2012 we know that the use of STXs in financial terms was about 16 percent under budget. But we don’t know whether this was due to a lower deployment of days/weeks or due to lower STX unit costs. We did not analyze this at ICDGP level but in any event it would be helpful to have this analysis automatically provided in CAPTAC-DR reports.

The current management reporting system applies a combination of accrual accounting (for some items, at the year-end) and cash flow accounting (during the fiscal year. Most accrual entries are passed only at the year end. The combination of these factors could make accounting information produced during the year unreliable.

The process of budget monitoring appears to be focused on ensuring that costs charged are within the annual rolling budget. CAPTAC-DR has detailed analysis of the costs it incurs, relating to workshops, attachments, local travel and disbursements. CAPTAC-DR does not however, possess any details of the basis on which HQ charges costs relating to RAs, Institute courses and HQ backstopping and project management, which constitute the largest element of CAPTAC-DR’s costs.

ICDGP monitors the overall budget, but does not similarly have ready access to details of the underlying breakdown of the costs being charged by the TA Departments (for example, rates, number of days, etc.). The individual TA Departments appear to monitor these, with varying degrees of intensity.


54 | P a g e

There is nevertheless considerable attention paid both by CAPTAC-DR, Finance Department and ICDGP on ensuring costs being charged to the CAPTAC-DR line items are appropriate with respect to their classification. We believe that monitoring of the CAPTAC-DR budget needs to be elevated to more of a ‘business’ level from its focus on total costs in budget line items. The CC should be responsible for this, and information barriers, such as staff costs, STX rates, rates of recharges by TA Departments need to be removed to enable this. Over time, for this to happen effectively, the IMF’s information systems need to generate useful cost breakdowns on a more regular, timely, and consistent basis.

7.4 OPERATIONAL AND FUNDING ISSUES RELATING TO FINANCIAL REPORTING There are a number of issues relating to financial reporting at HQ that lead to weaknesses in management-related financial reporting. They have been exhaustively dealt with in our report relating to the evaluation of the AML/CFT Topical Trust Fund, previous evaluations of the AFRITACs and other RTACs, and HQ-specific reviews. For example, the 2010 CARTAC Independent External Evaluation stated: ‘The IMF uses a surprisingly crude financial management and cost attribution system that is generally not fit for purpose and this has an impact on CARTAC.’14 We have not repeated all the shortcomings relating to financial accounting and reporting at the HQ level here and a detailed assessment of HQ systems was beyond the scope of the Evaluation. Nevertheless, these problems have a direct impact on CAPTAC-DR’s ability to produce financial information that can be used to assess performance at TA intervention and at RTAC level. We understand that steps are being taken to address them. The IMF has highlighted that reporting requirements may differ between RTACs, TTFs and Bilateral accounts. From ICDGP’s standpoint it is desirable to make some trade-offs in reporting for the sake of ensuring maximum consistency between these initiatives. The IMF has also rightfully pointed out that there are resource implications associated with developing new management reporting systems. Responsibility for specifying the requirements, and the continued maintenance of such systems is borne by the ICDGP. These costs are not eligible to be charged to external donors (although we believe that the RTAC-specific costs should be chargeable). Institutionally, capital funds available for development of new systems and reporting are limited to a limited pool, and are subject to competition against other administrative projects in the Fund.

14 Caribbean Regional Technical Assistance Centre CARTAC: Independent External Evaluation. February 2010.


55 | P a g e

8. ASSESSMENT OF CAPTAC-DR OPERATIONS 8.1 OVERVIEW CAPTAC-DR plans, delivers and monitors TA to its seven member countries. Its operations are unique and relatively complex for a small unit. CAPTAC-DR’s advantages and operational drawbacks derive from its integration with IMF HQ and its TA departments. The moves toward medium-term planning and increased focus on TA results are resulting in changes in operations within CAPTAC-DR. Generally, progress has been greater in areas where CAPTAC-DR can take decisions and implement initiatives autonomously. In the more critical areas, however, CAPTAC-DR is dependent on changes or decision-making at HQ. A key issue affecting our review of CAPTAC-DR’s operations relates to the implementation of an effective Results-Based Management system (‘RBM’), and a greater focus on defining, delivering, monitoring and reporting results. CAPTAC-DR has made substantial progress in this area but more needs to be done to develop RBM at a project level and to develop ‘smarter’ indicators. We have covered this in more detail in Section 9. Other areas affecting operations are reviewed in the sections on Financial Reporting (Section 7) and Strategy (Section 6). 8.2 DEVELOPMENT OF THE WORK PLAN The Operational Guidance Note for RTACs15 outlines the following process for development of the Center’s annual work plan: 1. An initial draft of the work plan is prepared by the RTAC Coordinator,

based on early inputs and requests of assistance from the beneficiary countries’ authorities, as well as on early and close consultation between the RAs and their backstoppers.

2. The initial draft work plan is then submitted simultaneously to the area (WHD) and functional departments (and copied to ICDGP) as early as possible for comments, so that it can be coordinated with, and integrated in, the functional departments’ Resource Allocation Plan and provide input for area departments’ overall country strategies.

3. Based on the inputs from area and functional departments, the Center

Coordinator prepares a revised draft work plan for final approval by the area and functional departments.

4. The final draft work plan is sent to the RTAC SC for review in advance of each meeting. The work plan is discussed and endorsed by the SC at the meeting.

5. The work plan is ready for implementation. 15 IMF Regional Technical Assistance Centers: Operational Guidance Note for Staff. January 11, 2006.


56 | P a g e

Thus, the work plan sees input from a wide range of constituencies, the overall objective being to ensure that it: Reflects the TA needs of member countries; Is consistent with WHD’s strategic objectives for the region; Is consistent with the objectives and capacity of the relevant TA Departments; Is consistent with CAPTAC-DR’s delivery expertise; and There is strategic coherence between the RTAC’s work and that of other Fund HQ-led activities in the member

countries and the region. The process is similar in the other two RTACs we have reviewed, but in their case the process seems more symbiotic (backwards and forwards between the various parties) than in CAPTAC-DR’s case. The CAPTAC-DR approach seems to have more emphasis on the first stage above in building a bottoms-up tide of demand from the beneficiary country members. The Center Coordinator and/or Resident Advisors launch a fairly intensive consultative process about three months ahead of producing a final work plan. They visit each of the directors of Customs and Tax and attend CIAT meetings, they have attended the third regional working group on medium term expenditure frameworks (for PFM), the CAMC for Monetary Policy and Operations, relevant central banks and the CCSBSO for Banking and Financial Sector Supervision and relevant authorities (usually central banks) for Statistics. WHD, ICDGP and TA departments do review CAPTAC-DR’s submissions in the light of their overview of priorities and needs for the region and in determining allocation of resources. RAs may not be experts on all aspects of their expertise but supplement their skills by leveraging off employment of STXs. Beneficiaries need to agree formally with the TA proposed because all TA is demand driven. Some of the TA in CAPTAC-DR’s case will flow through from prior periods i.e. multi-year projects. IMF HQ may see the big picture at policy and strategy level whereas the Center Coordinator and RAs are closer to what may be achievable on the ground. The WHD now produce an excellent and user friendly Regional Strategy Note (‘RSN’) which summarizes recent developments in each country; some of the specific challenges; a self- assessment rating on absorption of TA overall and by sub-sector (e.g. fiscal); a summary of TA delivered in the prior year or so whether by HQ or CAPTAC-DR and sets out various projects or programs for the year ahead with, in most cases, an estimate of the number of weeks of TA delivery needed and by which entity (HQ, CAPTAC-DR or TTF). The overall WHD program and the CAPTAC-DR work plan are not synonomous, given the demand driven imperative for CAPTAC-DR. However as Table 13 below illustrates for Customs, there has been close collaboration between CAPTAC-DR and WHD over the past two years.

Country CAPTAC-DR work plan 2013 Summary of activities

WHD RSN 2013 Summary of activities

Costa Rica Primary and ex-post controls New controls Honduras Controls strategy, risk profiling etc. Computerisation Nicaragua ditto Selectivity for audit El Salvador ditto Nothing proposed Guatemala Selectivity and ex-post controls; coordination

with tax authorities Focus on improving control strategy

Panama Ex-post controls Nothing proposed Dominican Republic Strengthen risk controls and audit planning;

special customs regimes Post release audit, risk-based on import clearance, control of free trade zone


57 | P a g e

As regards resource allocation there is an approximate correlation between the RSN and the CAPTAC-DR work plan for FY2013 in person years:

Summary plan PFM Revenue Adminis-tration

Monetary operations

Financial sector

supervision

External statistics

National accounts statistics

WHD –allocated to HQ TA departments

1.2 1.6 0.8 1.1 0.7 2.1

CAPTAC-DR 1.7 3.3 1.5 1.4 1.0 1.0 Total 2.9 4.9 2.3 2.5 1.7 3.1 Source: CAPTAC-DR

The RSN also estimates weeks of RA and STX needed by country from CAPTAC-DR in each of the main TA areas. Assuming that a person year equates to 52.2 weeks then the WHD estimates for FY2013 are for about 543 weeks (total person years in the RSN are 10.4 although they only cross cast to 9.9 in the table included in the RSN). CAPTAC-DR’s work plan estimates (bearing in mind FY2013 will be a 10 month year due toaligning of CAPTAC-DR’s fiscal year with IMF fiscal year) come in at 591 weeks so reasonably well aligned with the WHD estimates. Overall, subject to our comments on implementation of a wider strategic framework with a longer horizon, an effective RBM system to monitor it, and better linkage between the work plan and CAPTAC-DR’s financial resources, we feel that the work plan development process is working well. CAPTAC-DR might also consider using a similar system of beneficiary TA request forms as AFE since these are well designed and request much relevant information including strategic objectves, longer-term as well as more immediate TA needs and identification of other TA providers working with the relevant institutions in simlar or related areas. Delivery of TA and Administrative Support The TA is delivered in several ways or a combination of ways (i) through the resident advisor; (ii) through the resident advisor and STXs supervised by the resident advisors;(iii) through joint delivery with other donors (forming strategic partnerships with other donors is a key strategy for Phase II); through joint delivery with IMF HQ (usually limited to diagnostic missions) and through regional or national workshops and seminars (some of these in collaboration with other organizations such as ECLAC and many with participation of IFIs or donors such as IADB, World Bank and GiZ). The CAPTAC-DR office is located in Guatemala City in the same building as the central bank. This is convenient for ease of access to other countries in the region but with a substantial disadvantage compared to some of the other countries because of the security risk. This can make recruitment or retention of RAs more difficult and adds to costs because of security issues. The office is run by a CC supported by an office manager, four administrative assistants and two drivers/security staff. The seven RAs are based in the office (about 44 percent of the time on average) when not on mission. The Center Coordinator is IMF staff and is recruited by WHD and the RAs and STXs by the IMF TA departments. The STXs report (in practice) to the RAs; the RAs report to the TA departments but are also coordinated by the CC. The latter reports to the SC as well as to WHD.


58 | P a g e

Budgeting and accounting are collated by ICDGP with data input on CAPTAC-DR office costs, seminar and workshop costs and resident advisor or STX out of pocket costs fed in from CAPTAC-DR’s office manager. CAPTAC-DR has no control over allocation of HQ costs for project management and backstopping. The CAPTAC-DR office manager has no access to HQ general ledgers. Expenses in the field are covered by IMF replenishment funds that are made periodically through the fiscal year. Receipts are sent to HQ for ICDPGP review and approval and are further processed by the Finance Department. The CAPTAC-DR office-related costs are fixed overheads. RA costs tend to be treated as a fixed overhead also, even though they are on rolling one-year contracts. During the FYs 2010-2011 the largest savings as compared to the Program Document budget were in backstopping and project management. The STX costs were in line with the Program budget. CAPTAC-DR maintains a high quality public web site (and restricted access site for pre-qualified parties that sign the confidentiality agreement (members of the CAPTAC-DR Steering Committee: donors, country members and regional organizations), which covers organization, personnel, activities, results, workshop presentations and TA reports. The latter are on the members only site and are available if the beneficiaries do not object. There is also a regular quarterly newsletter. Both these media channels should facilitate donor coordination. Although, through the ‘members only’ window, SC members can obtain more detailed project information and mission reports, there is often a delay due to the drafting and an internal review time taken plus a 60-day period, during which the beneficiary may object to release reports. Some of the donors have complained about delays. We cover reporting and monitoring in Sections 10 and 11. 8.3 SCALE AND RESOURCE MIX

CAPTAC-DR really only got started from about mid-way through the 2010 fiscal year, started to accelerate in FY2011 and caught up almost completely with the original Program Document budget during FY2012. The most representative year to demonstrate the resource mix is therefore FY2012 (the results for FY2013 will be out shortly, but they only covered 10 months as the year end was changed to April 30 to align with the IMF’s fiscal year). As compared to annualization of the Program Document budget for FYs 2010-2014 the input from RAs was back on track as measured by cost whereas backstopping and project

management were still about 44 percent behind, and, most notably, STX expenditure was about two thirds higher. The substantially increased leveraging of RA time (1.3 of STX time to 1 of RA time) through deployment of STXs is in our view very positive and should enhance the TA delivery efficacy. The two AFRITACs we have assessed, AFE and AFW have been far less able to leverage in this manner; their ratios are below 1:1).


59 | P a g e

8.4 REVIEW OF THE CAPTAC-DR CENTER COORDINATOR’S ROLE

In common with all other RTACs the CAPTAC-DR Coordinator is a Fund staff member of the area department (WHD). The Center Coordinator is selected by, and reports to, WHD, which defines the terms of reference, in consultation with the functional departments and ICDGP. After the selection process has been completed, WHD informs the CAPTAC-DR Steering Committee of the selected candidate. The CC plays a critical role in CAPTAC-DR’s governance, and his role is consistent with those of the Coordinators of the other RTACs. The role is complicated by the fact that integration between the Center’s and IMF HQ activities produces a web of complex reporting, management and budget control responsibilities. The following are the main CC responsibilities as extracted from ToRs of RTAC CC, along with our comments on those responsibilities:16 1. Managing the Center; 2. Managing the work program; 3. Managing the budget; 4. Coordinating the work of the Steering Committee (Secretary to the SC); 5. Managing communications; and 6. Managing contacts with external stakeholders. In practice, in our opinion the word ‘management’ is used loosely. The Coordinator has the authority to hire and review the performance of only the Center’s local administrative staff. The RAs are appointed by the TA Departments and their performance reviews and decisions on contract renewal are also the latters’ responsibility. The RAs report to their backstoppers in relation to all technical aspects of their work. The CC only has control over a small portion of the budget, relating to local staff and miscellaneous costs, and those relating to workshops. Costs relating to RAs and STXs are decided on between the head office (TA Departments) and, to a limited extent in respect of STXs, the RAs. In effect, the CC has direct control over less than 10 percent of the CAPTAC-DR budget. A significant proportion of costs are charged directly to CAPTAC-DR by the various HQ departments, without substantive breakdowns being provided to the CC. The budgets for the year are agreed as between the TA departments and the Center (with agreement from WHD) but from that point on during the year the actual costs charged are not routinely explained to the CC. The budgeting and financial reporting system is suboptimal (see our comments in Section 7). The Coordinator does ‘coordinate’ the work program, which is the key strategic tool for the Center. It seems that the CAPTAC-DR Coordinator exercises a much greater influence over the shape of the work plan than in the two AFRITACs we have reviewed. Our understanding is that the role of the Center Coordinator has evolved from that of the traditional IMF Resident Representative. However, in our view the nature of the role should be substantially different. The Coordinator can exercise influence on different aspects of CAPTAC-DR’s strategy and operations, and we understand that the degree of such influence varies between CCs depending on their personalities, relationships and management styles. Nevertheless, in our view, in the absence of direct control and decision-making powers over substantive elements of CAPTAC-DR’s strategy and operations, it would be difficult to hold the Coordinator responsible for CAPTAC-DR’s performance. 16 Report of the Interdepartmental Working Group on Regional Technical Assistance Center Governance. IMF. January 2011.


60 | P a g e

We present below the recommendations relating to the Center Coordinators’ role, from a recent African Department Resident Representative workshop in which CCs from AFRITACs were present: Center Coordinators managerial effectiveness is constrained because of (i) lack of effective control of the center’s consolidated budget; and (ii) their limited impact on the selection and performance appraisal of resident advisors. The greater empowerment of CCs necessitates a concurrent rethinking and overhaul of the integration of the centers’ administration and management with the headquarters’ systems. There may be a lack of clarity on the division of labor between RTACs and HQs, in particular when HQs undertake TA activities beyond usual diagnostic/strategic planning work in the same area where an RTAC is involved.

Chart 15 provides estimates of the CCs’ time allocation for CAPTAC-DR, AFW17 and AFE, the three RTACs we are currently evaluating. Unlike the relative time allocation of the RAs (see Chart 16 below) there is divergence among the CCs in how they prioritize their work in terms of time allocation. The CAPTAC-DR Center Coordinator’s highest estimated time allocation is on the work plan (27 percent) and reporting (27 percent) and tasks related to the SC (9 percent). The chart for CAPTAC-DR probably under-estimates the time spent on ‘stakeholder liaison’ because much of this is probably

covered in the work plan development. These time distributions highlight to us the significantly administrative nature of the CC role in all the three RTACs. Given the high level of qualifications and experience of CCs some consideration should be given to lightening the administrative burden so that more time can be devoted to activities such as stakeholder liaison (including donor coordination) and oversight of project monitoring using RBM techniques. With the introduction of RBM in the past two years (introduced into CAPTAC-DR during FY2012) effective use of RBM tools such as results chains and logframes for monitoring project results probably requires a central focal point. This should be the CC supported by a research and monitoring assistant, if time permitted and would also involve closer cooperation with the RAs. The 2005 Independent External Evaluation recommendation of the AFRITACs was that CCs should participate in the performance evaluation of RAs even though they report to TA departments. This recommendation could be explored again in light of the apparently reduced level of project management by HQ. RTAC Coordinators go through the standard Annual Performance Review process conducted by area departments.

17 AFW information is from the previous CC, the current one having started only recently.


61 | P a g e

8.5 RESIDENT ADVISORS CAPTAC-DR has seven RAs, comprising one in PFM, two in Revenue Administration (Customs and Taxation respectively), two in Statistics (External Sector and National Accounts), one in Banking Supervision, and one in Monetary Operations. RAs are selected and hired by the relevant HQ TA Departments, which define their terms of reference, in consultation with the area department. The functional departments are responsible for annually assessing the technical competence of the RAs (usually conducted through review of outputs and a joint diagnostic mission with a member of HQ staff) and for deciding on whether to extend their contracts. Five of the CAPTAC-DR RAs come from the region, one from South America and one from France. They are highly experienced in their field, usually with practical experience at senior levels in implementing institutions. The RA has a number of responsibilities; the main ones are as follows: To deliver TA to beneficiaries; To design projects; To identify needs for an STX and then monitor their performance; To report back to the TA departments at IMF HQ and to Steering Committee; To assist in determining TA needs; and To assist in determining priorities.

The RA has the advantage of being within the region and close to the potential beneficiaries of TA and as a result is better placed to judge absorptive capacity, develop sound working relationships and coordinate with other providers of TA in a similar or related topical area. The RA is able to develop trust and good communications with potential beneficiaries. The RA spends about seven to eight weeks a year on missions (including preparation and reporting time) to the CAPTAC-DR member countries, depending on the needs and nature of the projects being delivered. Overall no more than 30 percent of an RA’s time is in the field delivering TA and about 11 percent on average (the range is 8 percent to 15 percent) is taken up with reporting to IMF TA departments post mission. Mission preparation which includes initial briefing papers takes on average about a further 10 percent. The level and frequency of reporting/coordination between CAPTAC-DR and HQ is a heavy burden, accounting for up to 20 percent of the RA’s time (derived from the spread sheets we asked all RAs to complete, which they did). Especially for those projects that are likely to be spread over more than one year and several missions there could be scope for reducing this burden by agreeing the entire project plan up front so that reporting can focus on exceptions rather than the routine. This leaves relatively little time for monitoring TA implementation remotely, handholding TA beneficiaries between missions, and research or professional development. Chart 16 opposite compares the distribution of RA time allocation in CAPTAC-DR, AFW and AFE. They are very similar, with generally over 65 percent of RA time being allocated to missions and related work.

Source: RA estimates


62 | P a g e

Typically reports are required after each mission even if it is only for one week. The findings from the mission will then be analyzed and commented upon by the relevant HQ department before release to the beneficiary before dissemination. Table 15 above was developed as part of the Evaluation from estimates provided by the CAPTAC-DR RAs. Detailed timesheets are not maintained, but the broad consistency between the time allocations suggests that the top-level estimates are reasonably accurate. The RAs from time to time also work between 10 and 20 percent of the official hours on unpaid overtime and, for example, travelling over weekends. 8.6 SHORT-TERM EXPERTS The role of STXs is twofold: Deliver expertise in a part of the specialized topic that the RA is less familiar with; and Augment the delivery of TA through allowing the RA to leverage their time. Formally, it is the HQ TA departments’ responsibility to identify and engage the STXs, and to backstop their work. The TA Departments have discretion to delegate these responsibilities to the RAs. RAs usually provide the first line of backstopping to STXs, with HQ providing input subsequently. The STX will be recruited through the HQ TA department but will be briefed and monitored by the RA. The STX will submit reports to the RA, who will in turn submit reports (amended/edited if necessary) to HQ. It seems from our review of reports and work plans that the majority of STX missions are two weeks or very occasionally longer. Often missions are only one week, especially when they are being jointly carried out with a RA. In our view, such short STX stays may not be economic. Sufficient financial information is not available to reflect the disproportionate impact of recruitment, travel time and travel costs on per day TA delivery costs. As indicated in our findings and recommendations for case studies and desktop reviews, we believe that more flexibility may be required in deploying STXs in order to improve efficiency and TA effectiveness. In particular, for capacity building TA (the majority of CAPTAC-DR’s work), a wider range of periods of STX deployment may be needed to ensure TA lessons are effectively implemented and the projects deliver results more efficiently.

CAPTAC-DR Resident Advisor Allocation of Time (%)

Source: RA estimates

Tax Customs PFM Supervision Monetary

Ops. External

stats

National Account

Stats. Avg

Own mission related 56 58 54 52 54 50 58 55 (of which client facing) (37) (39) (27) (27) (25) (25) (31) 30 Managing (backstopping) STX-only missions 10 8 10 8 10

10 8 9

Workshops/Seminars 8 8 13 17 13 11 12 12 Administration (incl. meetings, reporting, SC) 10 10 10 10 11

14

11 11

Vacation/Leave/Holidays/other 16 16 13 13 12 15 11 13 TOTAL 100 100 100 100 100 100 100 100 Source: RA estimates


63 | P a g e

The nature and extent of use of STXs also has an impact on the ability of RAs to leverage their time, which is considered in more detail in the next section. 8.7 LEVERAGING RA TIME The extent to which RAs leverage their time is, in our opinion, an important issue for CAPTAC-DR. The comparison of the STX/RA cost ratios for the different RTACs18 in Chart 17 opposite shows that CAPTAC-DR has the highest rank with respect to leveraging STXs and is also above the 113 percent level anticipated in the 2009 Program Document. It seems that the availability and ease of recruitment of suitable qualified STXs is perhaps much better than in other RTAC regions.

RAs have been mostly retained as Chart 18 opposite demonstrates. Most categories have held RAs from the start: Tax, Monetary Operations, Supervision and National Accounts. The External Sector Statistics and Customs RAs only left in FY2012 and were replaced in October. The only high turnover area has been PFM, with the third RA arriving in July 2012.

8.8 ROLE OF IMF HQ DEPARTMENTS Each of the various HQ departments plays a distinctive role in relation to CAPTAC-DR. TA Departments The TA departments play the largest role once TA priorities have been decided they periodically diagnose the needs in a particular topical area in a particular country and help determine priorities; they procure the RAs and the STXs; they oversee the performance of the RAs; and quality control all mission reports before distribution to beneficiaries. They also consult and submit their own recommendations for TA delivery in a particular country.

18 In a small number of cases budgeted costs were used in the absence of ready availability of actual costs. For CARTAC, 11 months’ actual costs were extrapolated for the full year.


64 | P a g e

Western Hemisphere Department WHD is responsible for coordinating all inputs on TA needs and proposed projects and, through an extensive consultation process regionally and with TA departments, produces a Regional Strategy Note for the Western Hemisphere, including CAPTAC-DR region. The RSN then forms the basis for a resource allocation plan. CAPTAC-DR will be allocated projects that are largely aimed at strengthening administrative and implementation capacity whereas HQ will focus more on diagnosis and strategy. WHD conducts regular surveillance missions and brings a regional as well as specific country experience and context to portfolio design for CAPTAC-DR members. It is worth noting that the process seems less symbiotic that we have seen with the African Department. The TA project selection process seems much more bottom up and actual annual work plans appear to diverge in many respects from those summarized on a country by country basis in the WHD RSN. Global Partnership Division of the Institute for Capacity Development ICDGP is responsible for coordinating TA activities; fundraising; putting together and monitoring the Program budget for the funding cycle of CAPTAC-DR (now five years) and annual budgets; communicating with and briefing the donors to CAPTAC-DR, assembling the budgets and reporting on financial performance; monitoring the timing and amounts of contribution to the funding by donors; and procuring and coordinating the periodic appointment of independent mid-term evaluations of CAPTAC-DR performance. Project Management and Backstopping Two of the key activities through which the TA Departments provide support and exercise quality control over CAPTAC-DR are through project management and backstopping. Project management relates to administration attributable to specific projects, including administrative support, human resources, travel and resource planning, and budget monitoring. Backstopping includes activities such as guidance to the RAs and in some cases STXs on mission objectives and timing, acting as a bridge between WHD and the experts with discussions on TA priorities and specific needs, providing analytical input on briefings and mission reports, drafting official letters accompanying reports to the authorities; the RAs tend in practice to provide the backstopping functions for the STXs. There are also some joint missions, other than diagnostic ones, where HQ TA departments and CAPTAC-DR collaborate.

Given the fact that CAPTAC-DR has managed to continue a stream of high quality projects and is very results oriented in its reporting, it could be worth analyzing why it appears to be apparently needing more backstopping and project management than other RTACs. FAD PFM’s division delivered high levels of backstopping because of the difficulty in recruiting RAs for the region. The region is conveniently located from HQ as compared to other RTAC operations and perhaps this is a partial explanation. There may be a point at which a reduced mission role from HQ TA departments could be cost effective in

the future without diluting the quality of TA program design.


65 | P a g e

Chart 19 above shows the ratio of HQ backstopping and project management costs over RA and STX staff costs for all the RTACs for FY2012 (CAPTAC-DR figures are for the period July 2011 to June 2012 as per the FY2013 Annual Report).19 There is a wide variation in reported backstopping and project management charges between the TA Departments. On average to date, FAD has recharged 4.5 percent of RA and STX costs for backstopping and 1.5 percent for project management; MCM has recharged 3.6 percent and one percent respectively; and STA has recharged 8.8 percent and 4.3 percent respectively. We understand that part of this may also be due to differences between TA departments in their internal administrative processes that may lead to not all costs being recharged. The extent and quality of backstopping on individual projects has been difficult to evaluate in detail within the scope of this project, as most of it is carried on through email exchanges. 8.9 INTERNAL REPORTING CAPTAC-DR RAs and the Center Coordinator need to produce monthly, quarterly, half yearly and annual reports on their activities and results for various audiences, often repeating the same information. RAs estimate that it takes them on average 1.4 person weeks (10 weeks for seven RAs), and the Center Coordinator spends approximately 14 person weeks for preparing reports. In total this comes to about half a person year. We have commented on the Steering Committee reports in Section 10.5. We believe that in addition, TA departments should consider the need for the quarterly reports and if they are necessary explore the scope for streamlining them. We believe that elements of the Center’s senior staff reporting responsibilities could be delegated to more junior levels, along with other functions, which we discuss in more detail below. 8.10 CAPTAC-DR’S OVERALL PROJECT AND PROGRAM MANAGEMENT Overall the project and program management is strong for several reasons: RAs are usually highly experienced in their areas of expertise and are known and respected in the region by

beneficiary clients; and IMF HQ exercises quality control by requiring consultation and approval of terms of reference for each mission (the

briefing papers) and then quality and peer review after the end of each mission (the mission reports).

Mission reporting seems to consume significant proportions of RA time (about 20 percent). Some RAs complained about the onerous level of reporting adding that often comments and feedback are delayed and even then sometimes lacking in substance. Long-term resident experts employed by the IMF tend to report quarterly (as opposed to after each mission, which are often only two or less weeks long). Maybe there is now sufficient experience of CAPTAC-DR’s performance to consider piloting more delegation of power from HQ, namely entailing less frequent reporting, and more oversight delegation to CCs. The overriding caveat would be preservation of work quality.

19 In a small number of cases budgeted costs were used in the absence of ready availability of actual costs. For CARTAC, 11 months’ actual

costs were extrapolated for the full year.


66 | P a g e

The reporting may be too fragmented if mission by mission or for too short periods. A well designed project, especially one spanning more than two years, could have a comprehensive terms of reference at the outset, even phase by phase, and this might avoid spending time on briefing papers for each short one or two-week mission. This also applies to post mission reports. CAPTAC-DR has enthusiastically embraced the process of launching logframes and reporting of results and is one of the more advanced RTACs in embracing RBM concepts. Program and project management could be further strengthened by making a clear distinction between activities, outputs and outcomes and then focusing on outputs and outcomes in reporting. Secondly, logframes should be developed at project level, country by country, especially for multi-year projects (as pioneered by West AFRITAC), because, if they are really to be used as a monitoring tool, this is the only effective way to do it. Thirdly, results chains could be more effectively used to identify all the interventions needed to secure the overall objective, so that the role of CAPTAC-DR is clear and in context. The results chain would also identify any inputs from other sources (facilitating coordination and collaboration with other donors, regional organizations and TA deliverers) and help identify any gaps that need attention. Utilizing RBM to the full will become routine once the practices are embedded with RAs, STXs and HQ. In the first year or so it may facilitate better monitoring to have RBM overseen and coordinated at CAPTAC-DR level. We comment in more depth on project management in Section 9 where we address suggested improvements in the project design process, assessment of risks, designing for specific results, and systematic capture of lessons learned. Designing for Capacity Building Many of CAPTAC-DR projects span more than one year and we believe that the projects could be better presented by estimating the delivery over the span of those years rather than only for part by part on an annual basis. This may demonstrate the longer term nature of the capacity building more clearly. It may also make it easier to plan for allocating more resources for the same topic in some countries rather than in others where differences in absorptive capacity exist. The level of TA required may vary significantly e.g. some beneficiaries are very capable of absorbing peripatetic TA whereas some may need more mentoring during and even after the project has finished. In countries that suffer from low capacity, extended periods of on-the-ground presence may be needed for knowledge transferred to be embedded. This may require STXs spending more time in the field or LTX mentoring. Many of the short-term projects are not aimed at complex or, by definition, long-term capacity building, but some of these projects are catalytic for longer-term follow on projects. Some of the design considerations appropriate to longer-term projects could be considered for these e.g. some of the pilot projects clearly have that potential.


67 | P a g e

8.11 PROFESSIONAL SUPPORT FOR CENTER COORDINATOR & RESIDENT ADVISORS We believe that the efficiency and effectiveness of the CAPTAC-DR Center Coordinator would be substantially enhanced if more professional support was available, given the proportion of time that is spent on activities that could be delegated to such a person. We believe that RAs would also benefit from similar support from, say, junior economists/professionals from the region, who could act as dedicated research assistants. In addition to taking on some of the RA’s existing responsibilities, they could support them on matters relating to more effective implementation of RBM. Some of the functions they could take on would be: Drafting periodic internal and SC reports; Researching country/regional sectors; Supporting the monitoring of TA projects in between missions; Over time, drafting notes such as Briefing Papers; and Improving communications with RAs in other RTACs. The research assistants could potentially be seconded from beneficiary countries, as an additional contribution in kind, and in turn individuals would be benefiting from the capacity building. However, we believe that the positions need to be for the medium-term, say for a minimum period of two years, to accommodate the learning curve needed to familiarize themselves with IMF quality and processes. This may only be practical if the research assistants are recruited directly by CAPTAC-DR. Should the suggestion be taken up, we would recommend that the position is initially piloted with one or two research assistants being shared by, say, two RAs each. The RAs would need to volunteer for the pilot, as for the pilots to succeed it is important that the RAs are accustomed to delegating and working in a partnership with and mentoring more junior staff.


68 | P a g e

9. REVIEW OF RBM IMPLEMENTATION AND CAPTAC-DR LOGICAL FRAMEWORKS

9.1 INTRODUCTION The need to report on the results of the IMF’s TA interventions has been highlighted in independent evaluations for a number of years. This has been reinforced by external donors, who have been funding an increasing proportion of the IMF’s TA. In this section we provide an overview and our comments on RBM introduction in the IMF and the RTACs to date. In addition, we comment on CAPTAC-DR’s logframes. In our view, the present organization structure, resource mix, and TA delivery approach of CAPTAC-DR and RTACs is not well suited to deliver RBM effectively. Given CAPTAC-DR’s organization structure, the main burden for implementing RBM would fall on the RAs. Currently their main focus is on delivering TA. Shifting their focus to project management and reporting within the prevailing modus operandi would, in our opinion, detract significantly from CAPTAC-DR’s strengths. The weakness in the IMF’s financial reporting systems and, in particular, lack of breakdowns in cost analysis also renders effective designing and monitoring of projects very difficult. In our opinion the IMF is implementing what could be defined as Results Based Reporting (‘RBR’), rather than a framework for managing, monitoring and reporting on projects to deliver better results. CAPTAC-DR and CC have taken considerable ownership of this initiative and have spent substantial effort in developing a results reporting system. The implementation of RBM was an objective referred to in the Program Document but CAPTAC-DR had to wait until 2012 when the IMF had designed a logframe system that would be common to all RTACs. In the spirit of RBM CAPTAC-DR developed comprehensive results reporting from the outset and from 2013 had identified indicators and milestones for its logframes. Substantial progress has been made to date, and our assessment is that the Center is providing meaningful information on the results being achieved on its TA activities. However, as we highlight in this section, substantial work remains to be done to complete the process, and a thorough reassessment of its objectives needs to be made. Experience at other donors and IFIs suggests that this entire process can take a number of years. 9.2 STATUS OF IMF RBM INTRODUCTION The IMF has used results based reporting for some time, with varying degrees of success. In 2010, an external consulting firm was engaged to develop a basic framework as a reference point for the introduction of RBM reporting into RTACs. The resulting report, presented in August 2010, made some general recommendations for approaching the tasks. The IMF felt that the recommendations fell short of expectations. Nonetheless, the IMF found that the study highlighted some important institutional barriers to the introduction of RBM and managed to generate some buy-in of resident advisors and the Steering Committees in selected RTACs. 20 The formalization of the process to introduce RBM IMF-wide, including the RTACs, was initiated with the commissioning of an Interdepartmental Working Group on RBM in late 2010. It drew on the results reporting pilots at AFW and PFTAC. 20 Interdepartmental Working Group on Results Based Management in the IMF: Terms of Reference. 21 October 2010.


69 | P a g e

An Interdepartmental RBM Implementation working group has been established to support the roll out and implementation of the new RBM framework. Full roll-out was expected to take approximately three years. We understand that the new RSN format/process referred to in the Strategy section of the Report has replaced the umbrella logframes. The RBM implementation has been dormant, but has recently returned to work. Funding for IT development, which is for part of the overall requirement, has been secured. However, we understand that the proposed IT system does not provide tools for project management, or for integrating all project related cost information. A number of critical decisions relating to implementation are still to be made. In the interim, the RTACs, including CAPTAC-DR, have started implementing RBM, within the overall strategic framework presented above. AFW has gone one step further by producing project level logframes for its multi-year projects. Section 9.3 below provides our comments on the RBM working group’s recommendations above; Section 9.4 presents an illustrative reporting framework for CAPTAC-DR; and Section 9.6 provides our comments on the RBM approach currently being implemented by CAPTAC-DR. 9.3 COMMENTS ON IMF RBM FRAMEWORK Our principal comments in respect of the current IMF approach to RBM implementation as outlined in the above recommendations are as follows: 1. We concur with the need to anchor RBM in the RSN/RAP process. However, as recognized by the RBM working

group, embedding the RTAC level logframes will be complicated, especially at early stages of RBM implementation, and there is need for flexibility in the degree to which CAPTAC-DR’s strategic logframes and objectives are required to be integrated into the RSN.

2. The integrity of the logical framework lies in the assumption

that all CAPTAC-DR level outcomes would draw on outcomes produced by all RAs engaged in the Center. Any outcomes produced by the RAs not contributing to the CAPTAC-DR level outcomes break this integrity. Similarly, all RA-level outcomes should be directly linked to outcomes produced by the countries in his/her portfolio. This principle extends to project level, and within projects for outcomes, outputs and inputs. If any country is excluded from the logframe chain, for example by not being an ‘intensive user’ as suggested by the IMF RBM framework, the logical framework system would break down. The RBM framework suggests that TA interventions in countries that are excluded from the logframe chain should be monitored by reporting ex post through work plans. We are unclear how such monitoring and reporting would differ from that within a logframe.


70 | P a g e

3. The RBM framework provides that RAs should produce topic-specific logframes using a format used by HQ. We understand from ICDGP that country-level logframes are intended to be the lowest level ones produced at RTACs, although there is some uncertainty in this regard. We believe that should this be the case, there is a risk that consolidating the cluster of heterogeneous projects in a country that would contribute to country-level outputs and outcomes within the same logframe would make monitoring the performance of individual TA interventions very difficult. The RBM framework suggests that the RTAC work plan should instead be used to monitor and report on project progress. We believe that this could lead to inconsistencies and duplication of efforts.

4. The issue relating to maintaining the logical chain extends to that within a project. For example, the proposed

definitions of outputs (what the IMF is responsible for, e.g., drafting a manual) and outcomes (measurable change in recipient institution), leaves a gap in between (i.e. the implementation of the manual in the working practices of the beneficiary institution). Implementation of advice is a critical link in the chain between the delivery of TA and the results being achieved and being monitored through indicators. DAC defines implementation within outputs; we have defined them for the purposes of the Evaluation as Intermediary Outcomes (see Section 1); and the RBM working group defines these as milestones. The proposal that milestones should only be established and monitored for projects spanning more than 18 months risks, in our view, the link between outputs (as defined by the RBM working group) and outcomes being broken.

5. Following on from the above point, considerable emphasis is placed in the IMF RBM framework on establishing and

monitoring outcomes (as defined by the working group), and for developing preferably standardized indicators for them. Whilst these outcomes are important, and in some areas of CAPTAC-DR activity such as targeted interventions in Revenue Administration these would be relatively measurable and within a relatively short period, in areas such as PFM, Financial Sector Supervision and Monetary Policy, these may take a number of years to be realized and even when they are, it may be very difficult to isolate CAPTAC-DR’s contribution to achieving them. We therefore believe that whilst ensuring that outcomes are indeed monitored and reported on, considerably more emphasis needs to be placed on monitoring and reporting on outputs and milestones.

6. We believe the suggestions on improving Briefing Papers and Back to Office Reports are positive steps.

7. We welcome the emphasis that the RBM framework places on correctly identifying the project’s underlying

assumptions and risks. Whilst we believe there is no need for CAPTAC-DR to undergo detailed risk analysis for every TA intervention, as suggested in the Strategy section we feel that the process should be formalized by developing sector results chains at the time country-level strategies are developed by the RAs.

8. In our view, results of projects are best judged in the context of resources that have been applied in delivering them.

The lack of emphasis on accurately identifying project costs could detract from the meaningfulness of the results. As indicated previously, our overall impression from the IMF’s approach to RBM is that what is intended is more of a results based ‘reporting’ rather than a results based ’management’ system. The focus on reporting is perhaps understandable, given the RBM working group’s ToRs and report. Our discussions suggest that the impetus for it has come principally from the need to report on TA results to an external audience. The framework as implemented cannot effectively be used as a management tool, as it does not address the planning, monitoring and implementation of the entire project management cycle. Moreover, by separating reporting from project management, the risk is that any weaknesses in the latter will continue, and separation of logframe related administration will be seen as an additional burden by project managers, rather than a tool for them to more effectively manage projects.


71 | P a g e

The IMF has made the following comments in response to our observations relating to RBM: Organizing log frames at project level makes sense, but only where an appropriately integrated and supportive IT

system exists. A Project Officer to take forward the IT part of RBM has now been recruited by the Strategy and Evaluation (SE) Division.

The issue of improving project management systems to monitor resource utilization at project level will also be taken forward by the SE project officer.

The decision was made to use topical log frames in order to ensure ownership by the RAs of ‘their’ log frames. Regarding country level log frames (covering all topical areas by country), the initial RBM roll out considered these, but noted that this risks an attribution gap between the country level log frame, which would be quite ‘high level’ and the topical level which is more operationally useful for the RAs.

The RBM working group ruled out including detailed inputs/financials in the log frame in the early stages of the RBM roll out, in order to focus attention on the results (outcomes) to be achieved, and to meet donor request to capture and report results, as well as to reduce the burden on RA. An important but intangible success of RBM to date has been an increased awareness of the log frame methodology and terminology among staff, whose main concern is on the technical delivery, rather than on project cycle management. To a large extent the RSNs and the country pages serve this function.

9.4 ILLUSTRATIVE REPORTING

FRAMEWORK FOR CAPTAC-DR

The role of CAPTAC-DR’s reporting system is to codify in measurable terms the Center’s objectives at various levels, and inform readers of its success in achieving them. To the extent that the CAPTAC-DR is part of a bigger institution, in this case the Western Hemisphere Department of the IMF in the first instance, it also needs to demonstrate how it is responsible for, and is contributing to achieving, the latter’s strategic objectives. The approach illustrated in Chart 21 opposite broadly follows the IMF RBM approach, with the following exceptions/ areas of emphasis: Budgeting and reporting unit to be TA

projects, rather than missions. For reference please see our case studies and desktop reviews in Volume II for how we define projects.


72 | P a g e

The need to integrate CAPTAC-DR and country level TA strategy and work plan with reporting framework. The need to integrate CAPTAC-DR’s operational (through work plans and logframes) reporting system with its

financial reporting. Logframes would need to start at the TA project level. Ad hoc and smaller interventions could be incorporated through

a consolidated approach. Logframes would include activities, and more detailed presentation of inputs. A clearer linkage between project-level, country-level and CAPTAC-DR-level logframes (principally outcomes). More detailed budgeting for projects (these are already done in adequate detail in some IMF units).21 Periodic analysis of budgets versus actual costs at project and CAPTAC-DR level. Logframes and financial information to be used for monitoring and reacting to project implementation.

9.5 THE PROJECT MANAGEMENT CYCLE In our view, in an environment such as the IMF/RTACs where hitherto the principal focus has been on delivering high quality outputs and advice, with the responsibility for implementation being left to the beneficiary, the move to being judged on results should involve more than simply developing a more elaborate reporting framework. Once the TA deliverer’s focus shifts from outputs to outcomes, the entire project management cycle needs to be adapted to delivering more successful outcomes, rather than focusing principally on outputs. Chart 22 opposite presents a generic simplified project design, implementation and monitoring system. To the extent that results in terms of realization of outputs, outcomes and objectives are to be achieved, each stage and component of the system makes an important contribution. In this section we utilize this illustration to highlight some of our observations relating to CAPTAC-DR’s project management and monitoring. CAPTAC-DR is not typical of other donors and IFIs in that the RAs are integrally involved in delivering projects. That leads to lighter touch monitoring and formalized management of projects. Based on our review of the projects and CAPTAC-DR’s operations, we feel that some of the areas highlighted below warrant attention. The challenge will be to sufficiently formalize many of the informal practices already taking place, whilst enabling the TA team to remain client and TA focused, flexible and responsive. As highlighted in Section 8, we believe that RAs will need additional junior level professional support and have made some recommendations in this regard.

21 AML/CFT Topical Trust Fund.


73 | P a g e

9.6 COMMENTS ON RBM IMPLEMENTATION AND LOGFRAMES AT CAPTAC-DR CAPTAC-DR has started implementing the IMF RBM from the FY2012. Substantial progress has been made to date, and our assessment is that the Center is providing meaningful information on the results being achieved on its TA activities. In this section we provide an overview of CAPTAC’s approach and some observations thereon. CAPTAC-DR’s logframes are structured at two levels: 1. At the Center/Program strategic level (the medium-

term strategic framework for the current phase). This provides the overall program strategic objective for the current phase: ‘Improved institutional capacity for beneficiary countries and strengthened financial and economic regional integration.’

This is further elaborated as: ‘The overall objective of the CAPTAC-DR is to help the beneficiary countries achieve their long-term development goals by offering technical assistance and training to improve their institutional capacity in macroeconomic management in the IMF's areas of specialization, and, at the same time, to strengthen economic and financial regional integration, in close cooperation with the other providers of technical assistance to the region.’ The program-level objective is supported by six topical objectives and one training objective. Outcome indicators for each of these topical areas are provided in the separate ‘Topic Area Logframes’.

2. There are seven topical area logframes (Statistics has two, for External Sector and National Accounts, respectively). They have the following levels: i. Topical area objective (for the phase). ii. Medium term outcomes and related indicators, for the phase. There are usually four outcomes per topical area

(Tax Administration has three and External Sector Statistics has two) and one indicator for each outcome. There is provision for reporting on progress towards these indicators.

iii. Milestones for each outcome, for each fiscal year. Typically there is one milestone per outcome, although in some instances there can be two or three. There is provision for reporting on milestones achieved.

iv. There is provision for highlighting specific risks affecting each outcome. v. Outputs, in terms of number of expected TA reports and regional seminars in the fiscal year are reported on. vi. Inputs, in terms of person weeks of RA and STX TA, and for seminars, inputs in terms of RA and STX person

weeks and participant costs are also reported on. vii. There are no separate logframes for training seminars.


74 | P a g e

Our assessment is that the logframe methodology being implemented by CAPTAC-DR is well structured and logically coherent between the RTAC and topical levels. We present below some comments in respect to its application: 1. In our view, it should be possible to attach verifiable indicators to the CAPTAC-DR level strategic objective (as is the

case for outcomes, etc.), so that progress toward achieving this objective can be monitored over time. CAPTAC-DR should seek to define such indicators for the Centre-level objective and how they are to be measured at the end of Phase II.

2. The logframe structure is topic-led, rather than country-led. This is adequate to the extent that CAPTAC-DR’s logframe is viewed on a stand-alone basis, especially as it can be configured to a country dimension. However, the RSN process is country-led and sets objectives by country. Any integration into regional (WHD)-level logframes would be facilitated if the logframes were principally structured along country, rather than topic lines. We do not believe that this is a critical issue that needs to be addressed immediately, but should be borne in mind as the logframes evolve.

3. More care needs to be taken in defining outcomes, and distinguishing them from milestones. In addition, they need to be defined in a way that facilitates measuring progress in and achievement of such outcomes. For example, the following outcomes and indicators are difficult to verify on the basis of information available: a. ‘Tax administration officials…better trained’ as an outcome, and ‘improved technical and management capacity

of tax officials’ is difficult to distinguish between and verify without more specific indicators for assessing such processes and capacity.

b. ‘Strengthened mechanisms to provide liquidity to commercial banks…’ as an outcome and ‘…improve mechanisms to provide liquidity to the financial system’ as an indicator has essentially no difference between the outcome and indicator and is difficult to verify easily and assess if the level of ‘improvement’ achieved.

More importantly, in our view, many of the outcomes listed at topic level are more appropriately classified as milestones, as per the IMF RBM definition. OECD DAC defines outcomes as: ‘The likely or achieved short-term and medium-term effects of an intervention’s outputs.’ It also defines outputs as: ‘The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.’ The indirect outputs, i.e. changes resulting from the intervention, are in our opinion defined under the IMF RBM as milestones. It follows that in our opinion, ‘effects’ of an intervention’s outputs are the intended results of that intervention. For example, in Customs Administration this could be an increase in revenues in the targeted area, or a reduction in costs per unit of revenue, or indicators of better risk-based targeting, etc. For PFM, depending on the areas targeted, this would include indicators of more improved budget forecasting, lower cash management costs, etc. For Statistics, outcomes could include promotion to SDDS, etc. In some areas, such as Financial Sector Supervision, immediate results are more difficult to define, although for example improvements in financial stability indicators could be used. Where results are difficult to verify, independent process-oriented indicators such as relevant FSAP or independently assessed BCPs, PEFA ratings, RA-FIT ratings, World Bank CPIA indicators, could also be used as a substitute.


75 | P a g e

Although indicators of outcomes have been identified in a number of instances in the topical logframes, a significant number of indicators are, in our opinion, more appropriately classified as milestones. These principally relate to changes in processes and practices in the TA beneficiary institution, and should be seen as interim results that contribute to the achievement of the outcomes. Examples of these include: a. ‘…financial supervision Superintendencies in the seven member countries adopt measures to reduce gaps…’ b. ‘central banks develop and adopt new methodologies and/or procedures….’ c. ‘officials adopt new methodologies according to MBP6 to compile external statistics….’.

4. In certain instances, milestones indicated are more appropriately classified as activities or outputs, e.g., regional

workshops/training. Moreover, milestones defined with words such as ‘progress in’, ‘improvement in’, ‘enhancement of’ etc. are difficult to assess without more specific definition of what constitutes such improvement.

5. We believe that having three/four outcomes per topic covering the entire CAPTAC-DR membership, and one milestone per outcome, may be limiting. We appreciate that this restriction is needed to present a coherent logframe in a summary form. The situation should be monitored and if necessary, more relevant outcomes and outputs included that more fully define the CAPTAC-DR topical work programs.

6. For each indicator, a credible source of verification should be listed. We believe that this would add to the discipline of

suggesting indicators that can be verified in practice.

7. As indicated above, we believe that the absence of project level logframes that build up to the topical level ones constitute a structural weakness. However, to the extent that the work plan constitutes a proxy for project level logframes, and is clearly interlinked with the topical logframes, mitigates this weakness. Care must be taken to ensure that there is logical coherence between the work plans and topical logframes, along the lines highlighted in Section 9.3 above.

8. There is a need to integrate the annual budget and actual expenses with the work plan and its implementation,

including at the project level. At present, CAPTAC-DR ensures this linkage at the budgeting stage, but limitations in cost allocation to projects etc. prevents adequate analysis of variances between actual and budgeted costs.

9. We believe that at the topical level, risks identified are necessarily generic and to the extent that they vary between

countries, these cannot be identified in the summary format. Nevertheless, identification of all the specific TA implementation risks associated with a project is an important contributor to designing more robust projects, and this should be made an integral part of the diagnostic/early phase terms of reference for a project.

10. CAPTAC-DR provides analyses of plan vs actual implementation of various aspects of its work plan. We believe this

should continue to be an important component of its reporting to its stakeholders. More explanations need to be given in some instances of the reasons for the variance.


76 | P a g e

10. COMMENTS ON CAPTAC-DR GOVERNANCE 10.1 INTRODUCTION AND OVERVIEW 18 entities (not counting the fact that member countries may be represented by both central banks and ministries of finance) are entitled to attend SC Meetings: seven countries from the region, seven donors, three regional organizations and the IMF. Of those, 15 commit funds, including the IMF with in-kind contributions. At SC meetings, the IMF team is represented by at least one person from WHD, ICD, FAD, MCM and STA and the CC as SC Secretary. The CAPTAC-DR RAs and the IMF Resident Representative based in Guatemala also attend regularly. Some SC members send more than one person to a meeting. The average turnout so far in Phase I amounts to about 29 individuals. The Center Coordinator acts as the Secretary to the SC and the RAs make short presentations of about 15 minutes each on their activities and results for the period under review.

Chart 24 shows the relative funding contributions by different donors and member countries for the currrent program phase as at April 2012. Funding-related issues are covered in more detail in Section 10.6 below. Cumulative expenses as at 31 December 2012 were US$18.2 million, approximately 50 percent of pledged funds with 22 months to go to the end of Phase I. Our assessment of CAPTAC-DR governance has relied on reviewing (i) minutes of Steering Committee meetings; (ii) CAPTAC-DR reports to the Steering Committee; (iii) an online

survey of Steering Committee members; as well as (iv) interviews with selected member country and donor representatives during field visits. 10.2 CAPTAC-DR AND STEERING COMMITTEE SURVEY RESPONSES/OBJECTIVES In order to assess the degree to which Steering Committee member objectives and priorities with respect to CAPTAC-DR were aligned, the SC survey respondents were asked to rank various potential goals of the Center in order of importance. A majority of respondents among donors (80 percent) and beneficiaries (40 percent) considered that the top priority is to maximize the developmental impact of CAPTAC-DR’s intervention. We have summarized the SC survey responses in greater detail in Section 6. Finally, all member country officials and 80 percent of donor representative considered that in the current phase of the program CAPTAC-DR is delivering its results either as expected or higher.


77 | P a g e

10.3 STEERING COMMITTEE ROLE All of respondents to our survey (except for two donors who didn’t know) feel that CAPTAC-DR discharges its roles and responsibilities effectively. The same proportion of respondents is also satisfied that CAPTAC-DR responds quickly to SC requests. However, 43 percent of respondents consider that the excessively large size of the SC hinders its effectiveness. In general, there seems to be some lack of clarity on the definition of the SC’s role, despite the fact that the Program Document defined it and further clarification was provided by the IMF to the July 2012 SC meeting in response to a request from the SC Chairman. In fact 37.5 percent of all respondents felt that the SC’s role and responsibilities are not clear (or expressed that they didn’t know). Nevertheless, the survey shows a broad coincidence in views from both donors and member country representatives. Both groups single out the provision of strategic guidance, along with endorsement of the Center’s work plan and monitoring of its implementation as top priorities (though in different orders). In contrast, member country representatives felt more strongly than donor respondents about their duty to ensure that their countries’ interests are properly represented, and to coordinate donors’ work.

As underlined by a respondent’s comments, there is no formal Terms of Reference or Charter. This is not the case with some other Trust Funds managed by IFIs. The 2009 Program Document describes the role of the Steering Committee as: ”to provide strategic guidance and contribute to the setting of the Center’s priorities, including the endorsement of the Center’s work plans. It is expected that the Committee will look at broad country priorities and provide its views in assisting beneficiary countries in the core areas of its expertise”. The Program Document does not mention the role of assessment of achievements or the monitoring of implementation although this role might be implicit in so far as CAPTAC-DR goes to great lengths to report on results. The ICGDP expanded to some extent on the Program document on issues such as decisions by consensus, and meetings to be annual, but did not mention monitoring of results as a SC role. The “endorsement” of the work plan at an SC meeting, which only takes place once a year and where the work plan proposed is distributed six weeks beforehand for the next operating year starting only a few weeks after the meeting, seems to allocate almost a rubber stamping remit to the SC. One of the donors interviewed stressed the key importance of RBM and even hinted at the idea of a working sub-committee to analyze results arising from RBM. In one of the other RTACs a survey respondent’s comment illustrates this interpretation further by mentioning that the SC is: “presented with a final product to be endorsed. [Members] can express whether we like something or not. But … there is very little room for change. Also, it would be helpful if a manual on rules and procedures would actually clarify the roles and powers of the Committee“. Table 16 below provides an overview of few of the issues raised and comments/suggestions made by members at SC meetings (which are non-attributable to any specific member) during the current Program phase.

SC Member 2nd SC meeting 3rd SC meeting 4th SC meeting

Country 1 Broaden access to Center/IMF training through online courses

Country 2 Increase systemic risk training Country 3 Continue government finance

statistics at country level Donor 1 Work plan should be in RBM format Donor 2 As above Observer 1 A sub-committee to identify TA priorities

with specific proposals on its complement of members. No consensus on this idea.

Reduce level of backstopping costs and Trust Fund mgm’t fees- relocate savings to TA

Source: SC minutes


78 | P a g e

So far the minutes of SC meetings don’t record much comment on strategic issues, TA priorities or achievement of results. Apart from themes arising from the SC survey summarized we also interviewed three of the SC donor members in some depth. One of the strong themes was on donor dialogue and dissemination of reports. Two of the donors wanted to see some improvement in this area and suggestions included: monthly donor video conference meetings; STXs to routinely touch base with relevant local donors in country; and, speed up access to TA reports. The point was also made that governments should take the lead role in setting TA priorities (implying a possible suspicion about supply driven TA?). These three interviews powerfully confirmed however strong support for CAPTAC-DR and a very high degree of praise for the quality of the RAs. One of the donors was emphatic that what was needed was more CAPTAC-DR and not less, but due to the importance of some topics, more focus should be on topics such as Revenue Administration even if it meant dropping some other areas, such as Statistics. Role of SC Members Representing Regional Bodies and Recipient Countries In the CAPTAC-DR context there is an important distinction between regional organizations and country members. Only the Central Bank of Economic Integration (BCIE), which is a donor, is a voting member of the SC but several others play an active role as observers. All seven of the countries are members of the CAMC, and CCSBSO and all except Panama of the Council of Treasury and Finance Ministers of Central America and the Dominican Republic (‘COSEFIN’). These three organizations are official observers at SC meetings. They formally have no votes, which is of little import in a consensus decision driven forum, but do have strong influence. CAPTAC-DR works closely with both CAMC and CCSBSO. The country members to some extent have more than one role: all seven have contributed US$250,000 each to the funding cycle and Guatemala has contributed an additional US$1.4 million in kind through hosting the Center’s offices and providing local staff in its central bank building. The country members therefore exercise the same rights as any other donor. As such, they can and should hold CAPTAC-DR accountable for its efficiency and effectiveness in the delivery of TA. The country members are also beneficiaries of TA although we have not seen much evidence of their reporting back on issues such as TA quality delivered, donor coordination or TA demands. Role of External Donor SC Members Essentially the external donor role should be the same as for the Steering Committee members as a whole, with the addition of a contribution on donor coordination. Just as country members can contribute beyond their role in governance, so can external donors bring expertise and knowledge to CAPTAC-DR, to the IMF and to fellow SC members. Two areas in particular stand out: firstly, in donor coordination many of the external donors have regional initiatives of their own, some of which overlap with those of CAPTAC-DR. The IDB probably has the largest contribution to make in this area. Germany, through GiZ, has maintained an active dialogue with CAPTAC-DR on some projects, notably in Tax Administration and PFM in El Salvador. CAPTAC-DR plans to strengthen donor collaboration with SC and non-SC members (such as the World Bank) in Phase II through forming strategic partnerships. Secondly, some of the donors have sound experience in RBM and project monitoring, the benefits of which could be passed on to the IMF and CAPTAC-DR. In fact, this is an area of particular interest for most of them.


79 | P a g e

Finally, SC members, and donors in particular, should play a crucial role in ensuring that resources are spent efficiently. The IMF has highlighted that the role of the SC in the governance of the Center needs to be considered against the fact that the RTAC is an IMF office. This allows for integration of the work of the RTAC with HQ work, as well as quality control by HQ and complementarity with TA delivered by HQ (upstream in nature, compared to the downstream/implementation focus of RTAC TA). While this ensures the consistently high quality of relevant TA and training, it also places some limitations on the extent to which the SC acting collectively can set priorities for the Fund. 10.4 OBSERVATIONS ON STEERING COMMITTEE MEETINGS The SC meetings are annual and until now have been held in one of the member countries close to the reporting year-end. CAPTAC-DR has now changed its fiscal year end to 30 April (to align with the IMF fiscal year) and is holding the next SC meeting in Washington DC to follow on from the IMF’s Spring Meetings, which will be attended by Ministers of Finance and Central Bank Governors from Central America, Panama and the Dominican Republic. SC meetings are chaired by the President of the CAMC; the Vice Chair is the President of COSEFIN; Members bear their own costs for attendance. Each meeting lasts about 4 to 5 hours (less than a full day) and the dates (till 2012) were usually close to the IMF’s regional conference thereby saving costs (although this year the SC meeting will be in DC following the change of CAPTAC-DR’s fiscal year end). We believe that a key contributor to an effective SC is the continuity of its membership. In CAPTAC-DR’s case the attendance record is excellent: Representatives from each of the countries have attended all four SC meetings so far with the following exceptions: Costa Rica for the second and Panama for the first. The external donors also have a good attendance record with only the IDB and the BCIE missing one meeting each out of the four so far. The three observers (CAMC, CCSBSO and COSEFIN) have not missed any meetings. There is a more serious issue with continuity though. Only the Dominican Republic and Nicaragua have been represented by the same individual at three of the four meetings. There are no countries where the same individual was present at all four meetings spanning the past three years. A slightly worse picture arises with the external donors: none of these members have managed continuity beyond two meetings and the IADB and BCIE have been represented by a different individual every time. The observer members have the best record with CAMC having the same representative at each of the last three meetings. The other issue worth highlighting is the number of attendees from CAPTAC-DR and the IMF, who typically number as many as the country members present and constitute a considerable bloc. This is because of the perceived need for having representatives from CAPTAC-DR (the CC and each of the RAs presenting their work) and from HQ’s WHD, TA and ICGDP departments, plus usually a representative from the IMF Institute. Given the time for preparation, travel and attendance this represents a substantial cost. We acknowledge that there are many benefits in having high-level, broad and deep IMF/CAPTAC-DR representation, but would recommend an assessment of the cost-benefit of this approach.


80 | P a g e

10.5 CAPTAC-DR REPORTING TO THE STEERING COMMITTEE CAPTAC-DR provides comprehensive reports to the Steering Committee at each meeting. Following year-end the reports include financial reports for the prior year (actual against budget) and a detailed work plan for the following year. The mid-year reports provide financial reports for the prior six months but do not provide detailed work plans. The annual report package is usually sent out to SC members about six weeks in advance. The agenda is usually sent out eight weeks in advance and affords SC members the opportunity to raise issues for the meeting, although in practice so far no requests for agenda amendment have been made. The reporting package typically comprises: The agenda followed by minutes of any presentations made to the meeting (Chair, CC, RAs and ICDGP

representative) and comments, questions and issues raised by SC members; a report in narrative form of TA activities and results by topic e.g. Tax Administration;

Report in detail and tabular form of TA activities and results by country; An annex which shows in chart form the distribution of TA by topic and by country; An annex showing the delivery of TA by topic through regional seminars and workshops; The work plan in narrative form for the following year; The work plan in logframe form of planned activities and results by country.

These reports in the aggregate are long, approaching one hundred pages in total (although still only half the length of those for, e.g., West AFRITAC).

A typical SC meeting lasts half of a day. The start times are usually at 9.30 am and finish times at 4.00 pm (including a meeting between donors and CAPTAC in the morning, outside the SC meeting). Two coffee breaks and lunch consume two hours. Four and half hours are left for the agenda of which about 2.5 hours are spent receiving short presentations from the RAs and the Center Coordinator. Reviewing the Minutes of the verbal proceedings it is clear that time is limited for discussion on any important issue. The SC is expected to endorse work plans and so presumably some time has already been spent studying the detailed reports of the work plans which run for about 40 pages (narrative about 25 percent and the balance in the form of topical logframes by country). Chart 25 opposite presents the responses relating to CAPTAC-DR’s reporting from the SC member survey. Generally, there is moderate and sometimes slightly negative

satisfaction with the adequacy of the reports, though consistently with dissenting opinions. Negative views outnumber positive ones as regards information on the coordination with the IMF. There is also a relatively large proportion of respondents expressing that they ‘don’t know’. In general, these results indicate that this appears to be an area requiring further improvement.


81 | P a g e

The reports provided by CAPTAC-DR are not only comprehensive and extensive, but they are also highly informative and of high quality. The topical logical frameworks are also analyzed on a country by country basis as well as for the region and are a very useful step forward for showing how the Center addresses long-term capacity building as well as a vehicle for deploying the developing skills in RBM. We believe there may be some drawbacks in the reporting because of the length of the reports, the shortage of time at SC meetings and the impact these factors could have on the SC’s governance priorities. There does not seem adequate time to discuss results as opposed to merely receiving a report. The results themselves are still presented in most cases in too generic a manner. There seems little time to discuss strategic issues. Discussion on either of these topics is hampered by a lack of continuity of representation on the SC. The sheer pagination of the annual work plans may require a disproportionate amount of SC time for a body that only endorses it. The shortage of meeting time may be a constraint with an agenda that allows limited space for strategic issues. A strategic issue might for example be how to devise a framework for more effective TA coordination by CAPTAC-DR. The following comment from a donor representative at the SC summarizes many of the problems with the current reporting: “Reports are often too long. And although they contain a lot of information, the information is not yet presented in a strategic and user-friendly manner. As mentioned above, reporting by country might facilitate the overall understanding of the context and the niche that IMF TA occupies relative to other providers. Also, there should be a clearer link to other IMF TA modalities. CAPTAC has started reporting on results. Especially the short country examples are a good step in this direction and should be built upon. TA is also often presented as a one-off exercise. It is not clear whether there is a concrete road map followed by the center (e.g. how will various TA missions contribute to a specific reform agenda? how about the implementation phase of reforms?). As to financial reporting: as far as I can see, CAPTAC does not report by project... Rather, the overall use of funds by the center is being presented. This is exactly what makes it difficult to judge the cost effectiveness of CAPTAC TA. I also don't get the impression that CAPTAC is really presenting the work in project logic yet.” 10.6 SUSTAINABILITY OF THE CAPTAC-DR FUNDING MODEL

CAPTAC-DR achieved full funding of its Program document targets. It was several months before the Center could become fully operational. By the 2012 year CAPTAC-DR was in full flow and tracking the level of expenditure forecast by the Program Document. The main risk to sustaining more or less the same level of funding in Phase II lies in retaining the commitment of the existing donors especially as some of the larger traditional donors in Europe, such as DfID (Great Britain), AFD (France) and SECO (Switzerland) are focused on other areas of the globe and not likely future donors for CAPTAC-DR. Spain has decided that it will not pledge funds for Phase II, which in Phase I accounted for 20 percent of the total.


82 | P a g e

It may not be expeditious, but the most grounded way to budget for a five-year phase would be to derive a medium-term macroeconomic and financial sector management plan for each of the CAPTAC-DR member countries and cost these out. CAPTAC-DR’s contribution would probably be parts of each component and these could be costed in turn. CAPTAC-DR countries do not have PRSPs. Very few of CAPTAC-DR’s countries have produced a more detailed development plan in the areas covered by the IMF. The Evaluation’s Terms of Reference require us to comment on CAPTAC-DR’s sustainability, with respect to its ability to financially sustain itself without donor contributions. Our survey suggests that donors and member country representatives have slightly different views. As depicted in the chart in the above, the majority (55 percent) of respondents expect that this sustainability will be reached either in more than 15 years from now or never. However, half of the CAPTAC-DR country representatives and 40 percent of donors responding expected to reach this stage in five to 10 years. In fact, there is no evidence from SC meeting minutes that financial sustainability has been discussed as an objective, although we understand that there has been some discussion concerning member country contributions. Should this be desired, the SC should agree a target horizon and formulate a strategy for gradual takeover of financial commitments by member countries.22

22 This idea is supported by comments made by both donor and member country representatives.


83 | P a g e

11. RECOMMENDATIONS 11.1 INTRODUCTION AND OVERVIEW In this section, we essentially collate and summarize the principal recommendations and suggestions provided in the preceding sections of the Report. These are not exhaustive, and you should review the rest of the Report for a complete set of recommendations. You are also referred to Volume II, which provides specific recommendations relating to the case studies and desktop reviews of TA projects selected for detailed evaluation. Overall, as the evaluation of CAPTAC-DR suggests, the TA advice being provided by the Center is very strong. The team is responsive and quick, and considerable care is taken to ensure that the advice provided is tailored to local circumstances. The main focus of our recommendations revolves around two issues: How does CAPTAC-DR leverage its resources to maximize its developmental impact? How does CAPTAC-DR ensure that the potential of results being achieved is realized, from the high quality advice it is

providing? This is a particularly important area because: o As the Center is focusing on outcome-level results, which are inherently more difficult to deliver than outputs, the

need for more systematic risk assessment, monitoring and project cycle management is critical. o How can RBM provide an effective framework for managing and reporting on CAPTAC-DR’s TA and workshop

activities? We recognize that a number of the recommendations in the Report are beyond the remit of CAPTAC-DR and need consideration at an IMF institution-wide level. Nevertheless, we have sought to concentrate on recommendations that can mostly be implemented by CAPTAC-DR autonomously. Development of a detailed costed strategy with verifiable outcomes should be a priority for Phase II. Effective implementation of RBM and improvements in project cycle management will require more attention being given to project management. We believe that it is important that the CAPTAC-DR preserves one of the key strengths of its TA delivery model: that its TA is delivered to a significant degree by its RAs. We therefore believe that processes need to be streamlined, whilst not leaving significant gaps in monitoring and reporting. Additionally, we believe that RAs need more support to increase their effectiveness. We have not repeated the recommendations on improving RTAC governance, operations and strategy produced by the various IMF working groups, except so far as they have arisen directly as a result of our observations during the Evaluation. We acknowledge that implementing many of the recommendations, especially in relation to the effective implementation of RBM, require allocation of financial and human resources. Some budgetary allocation has already been made in this regard. This should be an important matter of discussion between the IMF and the donors. A summary of the main recommendations is provided at the beginning of this section. The subsequent texts provide more detail, including rationale and context. In the latter sections we have indicated with each one of them (or groups), who we believe should take responsibility for following up, should they be adopted.


84 | P a g e

11.2 SUMMARY OF RECOMMENDATIONS The recommendations below are listed broadly in order of importance, both between and within each headline recommendation. Recommendation 1: A more detailed and costed strategy should be developed for the next phase, the implementation of which should be easy to monitor. This should be done through: Drafting a coherent, costed strategy for each sector in each country, rather than some illustrative projects. Using a

streamlined, country/sectoral results chain approach, they should reflect interventions that are appropriately sequenced; take into account other reform needs; provide assessment of other complementary donor programs and consideration for partnerships; and incorporate objectively verifiable milestones and indicators. To the extent that CAPTAC-DR needs to retain flexibility to address member country needs as they arise, say, 25 percent of the budgets and resources could be allocated for such needs.

Having a strategy that whilst consistent with the RSN, is capable of standing on its own. The Program and annual budgets being clearly built up from and linked to expected interventions and the work plans. Clearer linkages with country/sector strategies in member countries, especially in areas such as PFM. More detailed research into partner regional and donor institutions and their medium-term plans and incorporating

these into the strategy. Also, incorporating in the CAPTAC-DR strategy an action plan for donor coordination for the Center based on clearly defined expectations and objectives from the Steering Committee and member countries.

Differentiating more clearly between multi-year and short-term (less than one year) projects and monitoring the mix between them with the view of maximizing the use of the former.

Avoiding the possible temptation to cut back on difficult areas in favor of easier pickings in others. Developing a financial sustainability plan for CAPTAC-DR as part of the next phase strategy, should it be a medium or

long-term objective of the Steering Committee.

Recommendation 2: Strengthen CAPTAC-DR’s monitoring and reporting through the entire project and program cycle by the: Introduction of a robust accounting package, tailored to and suitable for the use of all RTACs, to enable standardized

and comparable financial information, which can be consolidated easily. It should cover all financial information, including those generated by HQ. Ultimately, it should also be integrated with the RTACs’ operational work plans and activities.

Elevation of budget monitoring from the current accounting level to more of a business level. The CC should take overall charge of this, and information barriers need to be removed to ensure that the Coordinator has adequate breakdown of underlying costs to ensure he can discharge this role effectively.

Identifying information needs for each key user group and designing reports to meet their needs, with a focus on analyses and explanations, deviations from expectations, and exception reports. Appropriate systems to produce such reports should be developed.

Integrating the work plan and annual budget Until IMF systems are capable of producing them, development of a CAPTAC-DR-level system to budget and track

actual project costs. Providing in annual reports more detailed and qualitative financial analysis of actual spends and variations from

budgets and explanations thereof.


85 | P a g e

Streamlining internal reporting. Implementing more robust processes in ICDGP, AFR and TA Departments to ensure all backstopping and project

management costs are charged fully. Recommendation 3: CAPTAC-DR’s operations should be streamlined and the efficiency of resource use improved through: Recruiting research assistants to support RAs. Extending the contract term of RAs beyond the current one-year. Removing barriers to IMF staff taking up RA positions in RTACs, such as the requirement to take unpaid leave. Improving efficiency of STX recruitment and use through inter alia using them for longer missions; targets for minimum

use of STXs; contracting them for projects or periods (rather than missions only); widening their recruitment pool; and giving RAs access to the STX database.

Freeing up more time of the CC on internal reporting, budgeting and the compilation of work plans to allow him to focus more on areas such as RBM, liaison with regional organizations, other stakeholders and donor coordination.

Recommendation 4: TA delivery should be improved and made more results-oriented through a combination of: Complementing the short-term mission based TA delivery approach with medium-term assistance. Incorporating implementation plans in all TA reports with clear milestones and clearly defined, verifiable outcomes. Establishing a system for ensuring that outcomes and milestones can be monitored effectively and timely action taken

to address slippages, outside the mission windows and after the project’s delivery has been completed. Encouraging the national authorities to play an active role in donor coordination and disseminate what is being done

and by whom. Installation of tools such as Aid Management Platform software created by the Development Gateway should be considered.

Recommendation 5: Enhance sustainability of CAPTAC-DR’s TA interventions through: Undertaking staff capacity assessment (numbers, ability) as part of the diagnostic phase of capacity building projects

and tailoring nature and length of TA appropriately. Where needed, improving TA absorptive capacity and sustainability in client institutions through implementation of HR

strategies (by CAPTAC-DR or partners) that improve staff quality and retention. Embedding medium and long-term projects into the host institution’s annual or long-term institutional plan or strategy. Taking steps to partner with regional and country training institutions to host/organize workshops, in order to build

sustainable local training capacity. Reviewing the current usage of capacity building tools such as attachments, training of trainers, use of regional STXs,

etc., the scope for increasing their use considered, and clear annual targets set for each tool.


86 | P a g e

Recommendation 6: Improve CAPTAC-DR’s cost-effectiveness through: More accurately tracking budgets and costs (all costs, including overheads and HQ, particularly for projects);

identifying underlying causes of variances; and implementing a system to take quick corrective action. Extending mission durations and workshop lengths. Reassessing the numbers of CAPTAC-DR/IMF staff attending SC meetings. Ensuring the CC and RAs are focused on higher value-added roles by engaging appropriate support staff. Recommendation 7: The effectiveness of the implementation of Results Based Management should be enhanced by extending it from primarily a reporting to a management tool through: Implementing the use of sector/country results chains as highlighted under Recommendation 1 to provide underlying

information for developing logframes. (i) extending the logical framework approach to the project level within each country; (ii) incorporating project level

input/cost information; (iii) improving the technical robustness of the logframes by ensuring that all projects/countries that are contributing to an outcome are accommodated in a logframe; (iv) incorporating relatively small interventions in a simplified format, rather than being excluded; and (v) placing more emphasis on monitoring milestones.

Drafting Briefing Papers on a project (rather than mission) basis, incorporating full project logframes including outcomes. Once information is available, such logframes should be augmented with: (i) TA advice implementation milestones and dates; (ii) TA timetable, including an outline of future missions in relation to milestones; (iii) project budget, in sufficient detail, broken down by mission; (iv) TA implementation risks identified during the mission and steps necessary by client (in implementation plan) or CAPTAC-DR (e.g. timing of mission, increasing STX support) to address them; and (v) proposed monitoring approach for the TA.

Systematically gathering lessons learnt and incorporating them in future interventions. Using the RBM framework as a TA management tool, rather than simply a reporting tool. Engaging a logframe specialist, either for CAPTAC-DR or say across the RTACs, to review logframe entries and

provide technical feedback to improve users’ practical understanding of their use. Recommendation 8: Improve standardization of practices, reporting and knowledge sharing between RTACs by: Carrying out a study of best practices across all RTACs, especially focused on RBM implementation, reporting, cost

effective approaches, and different components of project and program management. Establishing and a number of Key Performance Indicators to monitor the relative performance between the RTACs, in

addition to them serving as useful monitoring tools for each RTAC. Sharing TA reports and workshop presentations. Circulating to SC members executive summaries of the evaluation of other RTACs.


87 | P a g e

Recommendation 9: The effectiveness of CAPTAC-DR’s governance arrangements should be enhanced through: Re-examining the RTAC organization and management structure again in the light of experience. The reporting lines

are in several different directions depending on whether considering a CC, an RA, an STX or local staff. Financial controls are equally diffuse. The role of the Steering Committee is diffused. In these circumstances it is difficult to pin down accountability, transparency or responsibility for success or failure.

Streamlining the size and reviewing the seniority of participants in the SC or exploring alternative models. Improving continuity of membership among SC members. Streamlining reporting, with a focus on analysis and explanation of variances from plan, rather than mainly raw data. Devoting more space in the meeting agendas the SC to discuss and query results monitoring, discuss project results

and strategic matters. Introducing SC sub-committees (e.g. results and specific strategic issues) to enable the SC to assess results and

contribute to CAPTAC-DR strategy more effectively. Recommendation 10: The effectiveness of workshops and training should be improved by further integrating them with CAPTAC-DR’s TA work, through: Budgeting adequate preparation time for, in particular, in-country workshops to develop tailored case studies. Develop a more structured approach to post-workshop follow-ups including where appropriate: (i) setting up a virtual

network of participants; (ii) providing follow-up training to the same participants on the same topic; (iii) setting up mentorship arrangements; (iv) providing on-the-job training; and (iv) organizing study tours.

Recommendation 11: To ensure the Evaluation’s recommendations are implemented: CAPTAC-DR management should provide a formal response on them. Other IMF units should also comment. An implementation plan with quarterly milestones should be developed and presented to the Steering Committee. A Steering Committee sub-committee to guide and monitor implementation should be considered.

11.3 BACKGROUND TO AND DETAILS OF RECOMMENDATIONS 11.3.1 DEVEL OPMENT AND IMPL EMENTATION OF CAPTAC -DR’S STRATEGY 1. We recommend that for the next phase, CAPTAC-DR explores providing a coherent, costed, medium-term strategy

(and budget per annum) for each sector in each country for as many of the projects as possible. This approach will define priorities to guide CAPTAC-DR for the next five years and narrow the TA selection process to projects that fall within the range defined by the program. It would mean heavy consultation with beneficiary institutions at the preparatory stage but then somewhat less annually thereafter. We recognize that this would be a major task but it seems that inputs to cover much of the material exists already: WHD prepares medium-term RSNs; many of CAPTAC-DR’s existing projects are multi-year; and the TA departments have often identified medium-term priorities in their diagnostic reports. (CAPTAC-DR)


88 | P a g e

2. Whilst it is important that CAPTAC-DR’s strategy is closely integrated into the WHD’s regional and country strategies, it should be capable of standing on its own, and its implementation monitored and reported on autonomously. CAPTAC-DR needs to be accountable to the Center’s stakeholders, in particular its SC, to deliver its strategy. (WHD, ICDGP, CAPTAC-DR)

3. A more defined strategy would also enable CAPTAC-DR’s budget to be integrated with its strategy. This will require strengthening of the IMF’s financial reporting systems, but in our view the current absence of such linkage leads to inefficiencies. For example, a more transparent budgeting of projects and comparisons between results and costs of implementation could lead to RAs engaging STXs for longer missions, once they take into account the full costs of travel, procurement, etc. Moreover, we believe a stronger case could be made to donors and member countries, and within the approval systems in their respective institutions, if a better budgeted program is presented. (CAPTAC-DR, ICDGP)

4. One of the stated aims of CAPTAC-DR is to support countries in implementing their macroeconomic and financial

sector strategies. The CAPTAC-DR member countries do not have PRSPs and as far as we know the completeness of relevant strategies varies quite a lot from country to country (e.g. Honduras does have a National Plan). Some of the regional organizations such as the CCSBSO and CAMC have a clear regional overview in their particular areas of expertise and the IMF TA department missions have made recommendations for each of the topical areas they cover. What seems to be lacking in some cases is an overall macroeconomic management strategy for each country that highlights priorities, even though some will have benefited from FSAPs that are not too out of date. We suggest consideration is given to assistance to the development of an overall country-level and/or sector level reform strategies, which would help countries to see how particular TA (from the IMF and other donors) fits their priorities. We suggest that this area needs addressing and would help firm up the strength of demand for TA if it can be seen within the context of a prioritized strategy and action plan. (CAPTAC-DR, WHD)

5. In order to ensure complementarity with the activities of other donors, that no important barriers to achieving outcomes of CAPTAC-DR projects exist, and to pro-actively partner with other donors to provide supporting TA, the strategy process should ensure that the TA plans of other donors/IFI are adequately researched. In addition, the expectations on CAPTAC-DR from the Steering Committee on donor coordination and its objectives should be precisely defined. A formal strategy and action plan should then be agreed to deliver those objectives. (CAPTAC-DR, Steering Committee)

6. Short-term and multi-year project logframes should be defined, with clear output/outcome indicators which are measurable and verifiable (with defined verification means) and perhaps after the first mission, verifiable milestones. West AFRITAC has established a system of multi-year project planning and monitoring which, with some improvement, could be emulated. Spinning out and separately identifying the multi-year projects would be a positive step forward, recognizing the need for longer-term capacity building and providing a better framework for RBM. We have two suggestions for consideration: firstly, the proportioning of projects as between multi-year and short term and secondly, on the resources dedicated to multi-year projects. Firstly, we recommend increasing the proportion of multi-year projects (excluding statistics which are already multi-year in practice) to about 50 per cent. Secondly, the amount of TA delivered through multi-years projects by resident advisors and STXs appears to be in the range of about fifteen weeks to thirty weeks per country per project, based on the assumption that projects may be spread over about two years of elapsed time (as is the case with the other two RTACs we have examined). We don’t think this will always allow enough time for mentoring or on the job training or for beneficiaries whose available resources are below average in a particular area. The apparent attempt to spread TA equally by topic in each country may not always be


89 | P a g e

equally effective. Whilst it may be ideal for the beneficiaries to implement largely unaided CAPTAC-DR and or IMF recommendations, it is sometimes not practical without further assistance. As a result we recommend that TA delivery plans should be driven by both demand and absorptive capacity and in some instances TA should include hands-on learning using real cases. At the same time, the ability to remain flexible and react to member country needs as they arise requires preservation. Perhaps a proportion of the resources, say around 25 percent of a sector’s budget could be set aside for these needs. The above framework would enable program, country and intervention level indicators to be more credibly defined, and the success of the program more credibly assessed. Additionally, the CAPTAC-DR, country and project-level logframes would be more robustly integrated. (CAPTAC-DR)

7. CAPTAC-DR should continue with its highly focused approach i.e. within each topical area a few key strands of TA such as in Phase I tax segmentation or in PFM TA on medium-term expenditure frameworks. There may be new themes in Phase II but the principle of focusing on just a few priorities in each area and seeing them through to completion, rather than spreading too wide should be retained. At the same time, we would advise against the possible temptation to cut back on difficult areas in favor of easier pickings in others. For example, it seems generally accepted that the fiscal area is difficult and under that heading perhaps especially customs. Increasing revenue is critical in the region; revenue levels are below the averages for Latin America; almost 50 percent of all VAT is collected through customs. CAPTAC-DR with its regional presence has probably the best chance of any TA provider to eventually achieve some successes in customs reform. There is already tangible progress in Costa Rica, for example. (CAPTAC-DR)

8. Some SC members have indicated the need for CAPTAC-DR to achieve financial sustainability. A strategy for realizing this objective needs to be developed. (CAPTAC-DR, SC, ICDGP)

11.3.2 STRENGTHENING CAPTAC-DR’S MONITORING AND REPORTING 1. We recommend the introduction of a robust accounting package, tailored to and suitable for the use of all RTACs,

which will enable standardized and comparable financial information to be produced, and that can be consolidated easily. It should cover all financial information, including those generated by HQ. Ultimately, it should also be integrated with the RTACs’ operational work plans and activities. (CAPTAC-DR, ICDGP, FIN)

2. Budget monitoring needs to be elevated from the current accounting level to more of a business level. The CC should take overall charge of this, and information barriers need to be removed to ensure that the Coordinator has adequate breakdown of underlying costs to ensure he can discharge this role effectively. In particular the analysis of cost allocations from HQ for RAs, STXs, Institute courses, backstopping and project management need to be provided. (CAPTAC-DR, ICDGP, FIN, HR, Other IMF Departments)

3. For the next stage of development of CAPTAC-DR’s RBM database, we believe that the Center should consider the points we have made in relation to the IMF’s guidance on RBM. In addition, we recommend: i. Defining the various target audiences for its TA performance reports (e.g., SC members, WHD, TA Departments,

ICDGP, CC, RAs, etc.) ii. Designing reports that meet the information needs of these various constituencies, with appropriate analyses and

highlights, comparing actual vs plan, and extending over the planning horizon (which for RSNs, is extending beyond one year).

iii. Identifying the input information needed to produce the above reports. iv. Identifying/developing the processes/systems needed for capturing such input information.


90 | P a g e

v. Adapting the current systems and processes to capture the information and produce the reports. vi. Assessing human resource implications for implementing the above and securing adequate budget. (CAPTAC-

DR) 4. The work plan and annual budget needs to be much more integrated. To the extent that the IMF financial and other

information systems do not allow this easily, they can be done manually, in an Excel spreadsheet. (CAPTAC-DR)

5. The absence of project-level cost information is, in our opinion, a major shortcoming. Inter alia, without knowing the true cost of an intervention potentially leads to inefficient use of resources. CAPTAC-DR should explore alternative options for developing budgets and recording costs at a project level (linked to the program budget) until IMF systems can enable automation. Information from HQ Departments would still be necessary, e.g. on project/workshop costs directly recharged by the latter. Donors should approve costs for resources, which we do not believe will be significant, to enable this to happen. (CAPTAC-DR, SC, HQ Departments)

6. Annual reports should include more detailed and qualitative financial analysis of actual spends and variations from budgets. Moreover, underlying assumptions for budgets should be more clearly set out. In general, accounting reports should be more analytical in nature. (CAPTAC-DR, ICDGP)

7. Explore options for streamlining internal reporting. FAD reports are quarterly, MCM and STA’s are after each mission whether for one week or longer. Especially for longer-term projects this level of mission reporting to MCM and STA needs re-consideration. There should also be some consistency on dissemination of TA reports: until now FAD has declined to post such reports on CAPTAC-DR’s web site. (CAPTAC-DR, TA Departments, HR, ICDGP)

8. More reliable processes are needed in ICDGP, WHD and TA Departments to ensure all backstopping costs and project management are charged fully as incurred and, if there is real underspend, the reason explained. (CAPTAC-DR, ICDGP, WHD, TA Departments)

11.3.3 IMPROVING CAPTAC-DR’S OPERATIONS 1. We recommend piloting the recruitment of research assistants to support RAs. These assistants could be recruited

from within the region and could help with such tasks as reporting, RBM and other forms of project performance monitoring. (CAPTAC-DR, Steering Committee, ICDGP)

2. IMF should extend the contract term of RAs beyond the current one year. This short horizon creates, in our opinion,

uncertainty and is a likely barrier to RA recruitment and retention. (CAPTAC-DR, ICDGP, HR, TA Departments)

3. Barriers to IMF staff taking up RA positions in RTACs need to be removed. Recruiting IMF staff to RA position brings various efficiencies and enables them to be effective much more quickly. (IMF Departments)


91 | P a g e

4. With respect to the use of STXs: i. Establish targets for reducing the proportion of one-week only STX missions. ii. More flexibility is needed in deploying STXs, including for periods extending to a number of weeks and months,

where needed. iii. Extend STX terms and contracts for one or more projects, covering to more than one mission, and call-down

contracts, where possible. Set targets for each RA for such contracts. iv. Recruit STXs into the HQ experts pool more proactively, including advertising in appropriate publications/

websites. v. Give RAs access to the HQ STX database. (CAPTAC-DR, IMF Departments)

5. Free up more time of the CC on internal reporting, budgeting and the compilation of work plans to allow him to focus

more on areas such as RBM, liaison with regional organizations, other stakeholders and donor coordination. The CC does need to spend time in helping to determine priorities for TA that inform the work plans but would benefit from help in their actual compilation. The CC’s broad skills and experience in the areas of IMF TA are better used on tasks other than basic administration. An assistant for these purposes could be recruited from within the region without adding substantially to the costs. (CAPTAC-DR, HR, ICDGP)

11.3.4 DEVEL OPING A MORE RESUL TS-ORIENTED TA APPROACH

1. Consideration needs to be given to complementing the short-term mission based TA delivery approach with medium-

term assistance. For example, we noted that missions are rarely longer than two weeks even for an STX. Building on the recommendation above, there will be situations where more mentoring on some topics (for example, assistance in supervision inspections of banks) may require a little longer: this could help cement advice already delivered into routine working practices and should also achieve some efficiency gains. Where CAPTAC-DR resources do not permit this, other donors should be proactively approached. (CAPTAC-DR)

2. As a matter of course, all TA reports should incorporate implementation plans with clear milestones. In developing

such implementation plans, bottlenecks and capacity constraints at the host institution should be considered, and follow-up TA planned. Where such milestones do not correspond with the RA’s work plan, STXs should be engaged to provide support. Where further TA is envisaged, there should be a parallel CAPTAC-DR or other donor plan to facilitate implementation. These plans should form the basis of monitoring the achievement of milestones and outcomes. (CAPTAC-DR)

3. Projects should be defined with clear verifiable outcomes, and perhaps after the first mission, verifiable milestones.

Where this is likely to extend beyond one year, multi-year planning would be necessary, with future years’ targets more flexibly defined if required. This plan needs to be monitored and updated regularly, perhaps with every mission. A key need is to monitor implementation of advice after a project is completed. (CAPTAC-DR)

4. The rate of progress is variable in the region. There could be scope for enhanced collaboration among CAPTAC-DR

members. The Center could play a role as a facilitator of intra-regional technical cooperation (including secondments, study tours, etc.). It is often valuable to understand the neighbors’ solution to similar problems, for example Nicaragua would have benefited from knowing how other tax and customs administrations in Central America share their information. More specifically, encouraging preliminary results in one country (e.g. savings in El Salvador or in the Dominican Republic due to STA implementation) could be leveraged for sensitizing other governments in the region. (CAPTAC-DR)


92 | P a g e

11.3.5 ENHANCING CAPTAC-DR TA’S SUSTAINABILITY

1. Staff capacity assessment (numbers, ability) should be formally incorporated as part of the diagnostic phase of capacity building projects and appropriate action or conditions incorporated as part of the TA advice and implementation plan to address any weaknesses. (CAPTAC-DR)

2. Absorptive capacity: Very few of the CAPTAC-DR projects include missions aimed at strengthening the beneficiaries’

HR management in order to address quality and stability of staff. Some consideration should be given either by CAPTAC-DR or, through them and its Steering Committee, for other donors to provide specific TA to mitigate the limited absorptive capacity and high staff turnover where it occurs. Efforts need to be made to address (i) competitive remuneration structures, (ii) career paths, (iii) meritocratic development; (iv) working conditions generally; (v) staff appraisals; and so on. An effective HR system needs to be built in beneficiary institutions in many cases together with an in-house training capacity where scale permits. (CAPTAC-DR)

3. Embedding medium and long-term projects into the host institution’s annual or long-term institutional plan or strategy is likely to reinforce sustainability and should be encouraged by CAPTAC-DR RAs. This will have the additional benefit of incorporating the monitoring and reporting of implementation of advice in the host institution’s own systems, which can be relied upon by CAPTAC-DR. (CAPTAC-DR)

4. It is critical that to fulfill its regional sustainable capacity building mandate, CAPTAC-DR takes proactive steps to partner with regional and country training institutions to host/organize workshops. Partnerships need to be identified, deepened and widened. (CAPTAC-DR)

5. The current usage of capacity building tools such as attachments, training of trainers, use of regional STXs, etc. should be reviewed. To the extent that these tools are still considered important, targets should be set for each sector and reported against. (CAPTAC-DR, Steering Committee)

11.3.6 IMPROVING CAPTAC-DR’S COST-EFFECTIVENESS 1. A key first step in improving cost-effectiveness is the accurate determination of costs. Our previous recommendations

relating to the need to develop systems and processes to determine project, workshop and program-level costs need to be implemented. Actual costs need to be compared with budgets regularly, the reasons for differences identified and appropriate action needs to be taken systematically to reduce costs. (CAPTAC-DR, HQ Departments)

2. Missions and workshops incorporate various fixed costs in preparation, travel, reporting, etc. Where appropriate, the lengths of such activities should be extended. In addition to improving cost-effectiveness, they enhance effectiveness of TA delivery. (CAPTAC-DR)

3. At present, a substantial proportion of the CC’s and RAs’ time is spent on administrative and reporting activities that

could be carried out by others who are substantially less expensive. Project management and reporting responsibilities are likely to increase if RBM is to be implemented effectively. Appropriately qualified junior staff, e.g. for the CC and research assistants for RAs should be appointed to take on such responsibilities. (CAPTAC-DR, ICDGP)


93 | P a g e

4. As highlighted in the Report, the cost of all RAs, the CC and support staff from CAPTAC-DR and representatives of the different HQ Departments attending the SC meetings is considerable. Steps are taken by HQ Departments to reduce the associated costs, e.g., by connecting attendance with missions and not all the associated HQ travel and time is charged to CAPTACT-DR (although we recommend this should be done to improve accounting accuracy). This is a cost to the system and the IMF/CAPTAC-DR should consider, perhaps in discussion with the SC, whether this level of physical attendance is necessary to discharge the IMF’s accountability and donor relationship responsibilities. (CAPTAC-DR, HQ Departments, SC)

11.3.7 ENHANCING THE EFFECTIVE NESS OF RBM IMPL EMENTATION 1. The Results Chain: There needs to be substantially more consideration given to the context within which CAPTAC-

DR’s interventions, at the sector level within each country and at individual project level, are intended to be delivered. It is highly unlikely that CAPTAC-DR’s interventions will be able to deliver important outcomes on their own. In the absence of a country sector development strategy IMF HQ/CAPTAC will help define such a strategy through its diagnostic TA. The best tool for incorporating all of the above elements is the Results Chain which we have illustrated in each of the case studies. Following on from the Results Chain will be a CAPTAC project specific logframe that may have in its risks section any failure to deliver the other interventions identified in the former. Where project logframe milestones do not correspond with the RA’s work plan or expertise, STXs should be engaged to provide support. Where further TA is envisaged, there should be a parallel CAPTAC-DR or other donor plan to facilitate implementation. These plans should form the basis of monitoring the achievement of milestones and outcomes at both Results Chain and project logframe levels. The relevant dependencies, and associated risks need to be clearly assessed, documented and their potential impact on achieving outcomes analysed. These include: i. Sequencing – e.g., whether necessary preconditions, such as enabling legislation and regulation for

implementing the actions that are the subject of the TA are in place, and if not, what the implications for achieving outcomes are, in their absence.

ii. Wider reform needs (‘assumptions’ in RBM / logframe terminology). Many TA projects, particularly in the areas of PFM and revenue administration, are likely to be dependent on other supporting reforms that may be beyond CAPTAC-DR’s capacity, or its remit (e.g., IT). An absence of, or delays in, such reforms would have a direct impact on the success of CAPTAC-DR’s projects. This will also require a good understanding and methodical documentation of the other IMF activities and those of other donors acting in such areas. The role of other donors could be specifically reported on at the project design stage.

iii. The use of the results chain at the design stage is particularly useful for this purpose because it should show the contribution of CAPTAC-DR in the context of other interventions (supported by other donors or implemented by the national authorities) needed to achieve the high level outcomes. It may also throw up gaps and thereby stimulate raising attention to these with other donors. Secondly, the national authorities need to be encouraged to play an active role in donor coordination and disseminate what is being done and by whom.

iv. Risks – Specific risks to the TA achieving outcomes, and on time. These could include weak staff capacity, high staff turnover or inadequate staffing in beneficiary institutions, poor track record of implementation, etc.


94 | P a g e

These issues could be summarized into a sector results chain, as outlined in our Evaluation Methodology section (Section 1). More importantly, the results chains provide a valuable framework for analyzing the issues highlighted above and could be constructed, for example, at the IMF country level. Information from the results chain would be fed into country and project level logical frameworks and monitored. Much of this work is done already, albeit informally, for example during diagnostic missions, but in our opinion, a more systematic TA oriented approach is required. (CAPTAC-DR, ICDGP)

2. With respect to its RBM framework, the IMF should:

i. Extend the logframe to project level within each country; ii. Incorporate project level input/cost information; iii. Ensure that all projects/countries that are contributing to an outcome are accommodated in a logframe. Relatively

small interventions should be incorporated in a simplified format, rather than be excluded; iv. Place more emphasis on monitoring milestones.

(ICDGP)

3. Briefing Papers should be drafted on a project basis, which could be multi-year, rather than for each mission that is part of a project. In addition to the background, activities and expected outputs, it should include: a. The full project logframe, including outcomes. For the first/diagnostic mission, broad outlines where possible. b. The country level outcomes and objectives and how the project contributes to it in a verifiable way. c. After the first/diagnostic mission, the logframe should include (perhaps elaborated in the BTO):

i. TA advice implementation milestones and dates. ii. TA timetable, including an outline of future missions in relation to milestones. iii. Project budget, in sufficient detail, broken down by mission. iv. TA implementation risks identified during the mission and steps necessary by client (in implementation

plan) or CAPTAC-DR (e.g. timing of mission, increasing STX support) to address it. v. Proposed monitoring approach for the TA.

(CAPTAC-DR)

4. CAPTAC-DR gathers key lessons from its projects on an ad hoc basis. It should formalize this by providing in the BTO a dedicated section for listing lessons learnt, and having a process to periodically collate these and ensure dissemination and incorporation in TA practice. (CAPTAC-DR)

5. We believe that the RBM framework’s effectiveness and usage will be improved if it is implemented as a TA management tool, rather than its current focus on reporting. We note that some RAs have developed monitoring matrices and perhaps these could be analyzed and adapted to other topical areas. (CAPTAC-DR, ICDGP)

6. More training is required for the RAs to appropriately define and input information in the logframe. In our experience it inevitably takes time for users to get familiarized with logframe terminology. One option for accelerating the learning process is to engage a logframe specialist, either for CAPTAC-DR or say across the RTACs, to review logframe entries and provide technical feedback to improve users’ practical understanding of their use. (CAPTAC-DR, ICDGP)


95 | P a g e

11.3.8 INCREASING STANDARDI ZATION OF PRACTICES AND INFORMATION SHARING ACROSS RTACS 1. We recommend a study of the practices carried out by the different RTACs, covering RBM-related and wider

SC/management reporting, and also the approaches being adopted by each for different stages of the entire project management cycle, including, for example, recruitment and management of STXs, in order to identify and disseminate ‘best practice’ information applicable for all RTACs. (ICDGP)

2. The RTACs are relatively unique and it is difficult to compare their performance against other donor or IFI projects. There are currently eight RTACs operating in various regions of the globe, four of which are in sub-Saharan Africa. We believe that the IMF should consider developing a number of Key Performance Indicators (examples of some have been presented in the Report) to monitor the relative performance between them, in addition to them serving as useful monitoring tools for each RTAC. Some key indicators should be easy to develop as high-level budget and actual information is generally standardized. Over time, as IMF’s information systems become more useful for managing projects, the KPIs could become more in-depth and their effectiveness as management tools further enhanced. (CAPTAC-DR)

3. Workshop experience should be shared with other RTACs (this should also apply to TA reports). Many of the topics

are similar and there could be benefits from sharing presentations and ideas. One vehicle for this could be an internet portal for all RTACs, although language issues could be a barrier for some RTACs. (ICDGP)

4. IMF should consider circulating to SC members the executive summaries from the independent evaluations of other

RTACs. There are various common lessons that can be shared across the Centers. (ICDGP)

11.3.9 IMPROVING CAPTAC-DR’S GOVERNANCE

Steering Committee meetings have a mass of information to review in advance and limited time to discuss strategic or other issues during the meetings due to their brevity. Boards of Directors of major companies use sub-committees frequently to cover specific issues e.g. audit committees. There are other models that could be considered to render the present governance structure more a deliberative body than largely, though importantly, a body that CAPTAC-DR reports and is accountable to.

1. We believe the RTAC organization and management structure needs to be examined again in the light of experience.

The present structure may have been appropriate when RTACs were first launched but now could be a time to take a fresh look. The reporting lines are in several different directions depending on whether considering a CC, an RA, an STX or local staff. Financial controls are equally diffuse. In these circumstances it is difficult to pin down accountability, transparency or responsibility for success or failure. Perhaps RTACs should be run more in a typical corporate form i.e. with a CEO (the CC), professional staff (the RAs and STXs) initially accountable to the CC for many project management tasks with back up quality control on technical issues from TA departments and overview of regional priorities from WHD. (HQ Departments)

2. The CAPTAC-DR Steering Committee is very large and includes very high level participants from the seven countries

(Ministers and Central Bank Governors). This needs to be reviewed. (SC)

3. There is also a very low level of representative continuity among donor members with many meetings being attended by a different individual each time. This detracts from the SC’s effectiveness. (SC)


96 | P a g e

4. The annual reports should be reduced in some areas. The logframes are very useful but it may be sufficient to

highlight the outcomes in the reports and these should be of interest if they are more tangible and analysis-focused than at present. (CAPTAC-DR)

5. The meeting agendas should leave space for the SC to discuss and query results monitoring, have the time to discuss

project results and strategic matters. (CAPTAC-DR, ICDGP, SC)

6. The SC should consider establishing a Sub-Committee (perhaps comprising representatives from no more than two or three beneficiary countries, two or three external donors and one regional organization) to guide the implementation of the RBM/logframe related recommendations, so far as they can be carried out by CAPTAC-DR. The Sub-Committee could also guide the implementation of recommendations relating to Steering Committee Reporting. (Steering Committee)

11.3.1 0 ENHANCING EFFECTIVEN ESS OF CAP TAC-DR’S WORKSHOPS AND TRAINING

1. Adequate preparation time should be budgeted to develop tailored case studies. Consideration should be given to extending training sessions, e.g., where appropriate allowing STXs to stay on and provide follow-on training after the RAs have left, especially for in-country workshops. (CAPTAC-DR)

2. Develop a more structured approach to post-workshop follow-ups including where appropriate: (i) setting up a virtual

network of participants; (ii) providing follow up training to the same participants on the same topic; (iii) setting up mentorship arrangements; and (iv) providing on-the-job training; and (iv) organizing study tours. (CAPTAC-DR)

11.3.11 IMPL EMENTATION OF RECOMMENDATIONS 1. To formalize the process of acceptance and implementation of recommendations from the Evaluation, we recommend

the following: i. For each recommendation, relevant CAPTAC-DR/IMF units indicate whether they accept it. ii. The relevant CAPTAC-DR/IMF unit provides any additional comments on each recommendation, especially if

they do not consider a recommendation not to be practical for implementation. iii. With input from all responsible units, CAPTAC-DR produces an implementation plan for recommendations that

have been accepted for implementation, with quarterly benchmarks. iv. A SC Sub-Committee is established to guide the implementation plan. (CAPTAC-DR, Steering Committee, Other IMF Departments).


97 | P a g e

ABBREVIATIONS AND ACRONYMS AFE East AFRITAC AFRITAC African Regional Technical Assistance Center AFW West AFRITAC AML/CFT Anti-Money Laundering/Combating the Financing of Terrorism BCEI Central American Bank for Economic Integration BCPs Basel Core Principles FOR Effective Banking Supervision BCRES Banco Central de Reserva de El Salvador BP Briefing Paper BTO Back to Office Reports (IMF) CAPTAC-DR Central America, Panama and Dominican Republic Technical Assistance

Center CARTAC Caribbean Regional Technical Assistance Center CC CAPTAC-DR Center Coordinator CCS Costa Rican Customs Services CCSBSO Central American Council of Superintendents of Banks, Insurance and

other Financial Institutions CIAT Inter-American Center of Tax Administration CAMC Central American Monetary Council COSEFIN Council of Treasury and Finance Ministers of Central America and

Dominican Republic CPIA Country Policy and Institutional Assessment CRI Costa Rica DAC Development Assistance Committee of the OECD DOM Dominican Republic DfID Department for International Development (UK) ECLAC Economic Commission for Latin America and the Caribbean EC European Commission EU European Union FAD Fiscal Affairs Department (IMF) FSAP Financial Sector Assessment Program FIN Finance Department (IMF) GDP Gross Domestic Product GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit, a German donor

organization GTM Guatemala HQ Headquarters (IMF) HON Honduras HR Human Resources IADB Inter-American Development Bank ICD Institute of Capacity Development (IMF) ICDGP Institute for Capacity Development Global Partnership Division (IMF) IFI International Financial Institution IMF The International Monetary Fund KEQs Key Evaluation Questions


98 | P a g e

LCN Latin American and Caribbean LoU Letter of Understanding LTU Large Taxpayer Unit LTX Long-Term Expert (IMF) MCM Monetary and Capital Markets Department (IMF) NAS National Accounts Statistics NIC Nicaragua OECD Organization for Economic Cooperation and Development OR Operational Risk PAN Panama PEFA Public Expenditure Framework Assessment PFM Public Finance Management PRSP Poverty Reduction Strategy Paper RA Resident Advisor (CAPTAC-DR) RBM Results-Based Management RBR Results-Based Reporting RBS Risk-Based Supervision RSN Regional Strategy Note (IMF) RTAC Regional Technical Assistance Center (IMF) SC Steering committee of CAPTAC-DR SECMA Executive Secretariat of Central American Monetary Council SLV El Salvador STA Single Treasury Account STX Short-Term Expert (IMF) TA Technical Assistance ToRs Terms of Reference/Statement of Work for the Evaluation ToT Training of Trainers TTF Topical Trust Fund USAID United States Agency for International Development US The United States of America US$ United States Dollars WCO World Customs Organization WHD Western Hemisphere Department (IMF)


99 | P a g e

INDEX OF TABLES AND CHARTS

TABLES 1 CAPTAC-DR Evaluation Rating Criteria 2 2 List of IMF RTACS 8 3 CAPTAC-DR Member Country Development Indicators 10 4 Summary of Evaluation Ratings for TA Projects 12 5 Number of Projects Flowing From the Same TA Design and Objective 13 6 Analysis of Projects by Type Including Short and Long-term 14 7 Regional Workshops Selected for Review 32 8 Top Three Benefits of Workshops (Survey Response) 33 9 Actual TA Delivery Compared to Work Plan 41 10 Regional Seminars/Workshops Links to TA Projects 46 11 Percentage of TA by Topical Area 2010-12 49 12 Backstopping & Project Management Budget vs Actual for HQ Departments 52 13 CAPTAC-DR Work Plan Compared to WHD RSNs 56 14 Person Years in Work Plan Compared to WHD Estimates 57 15 CAPTAC-DR: Resident Advisor Allocation of Time 62 16 Overview of issues raised by SC 77


100 | P a g e

CHARTS 1 Illustrative Sector-Level Results Chain 5 2 CAPTAC-DR Member Countries 8 3 IMF Field TA Delivery by Modality 9 4 CAPTAC-DR Resource Allocation by Country and Topic 2010-12 9 5 CAPTAC-DR TA Desk Top Reviews: Evaluation Ratings 25 6 Survey Responses on CAPTAC-DR TA 28 7 Workshops & Seminars/Total Budget 31 8 Survey Responses on CAPTAC-DR Workshop Delivery 34 9 Percentage of Workshop Lessons Implemented (survey responses) 35 10 Where CAPTAC Should Focus TA (survey responses) 40 11 Distribution of Trust Fund Expenses: AFW versus CAPTAC-DR 51 12 CAPTAC-DR: Budget versus Actual Costs FYs 2010-12 52 13 CAPTAC-DR Work Planning Process 55 14 CAPTAC-DR: Actual Expenditure by Type (FY2012) 58 15 Center Coordinator Time Allocation 60 16 Resident Advisor Time Allocation 61 17 RTACs: STX/RA Cost Ratios 63 18 CAPTAC-DR: Tenure of RAs 63 19 Backstopping-Project Management/RA and STX Cost Ratios 64 20 The ‘Logical’ Framework 69 21 CAPTAC-DR: Illustrative Results Reporting Framework 71 22 Illustrative Project Management/Monitoring Cycle 72 23 CAPTAC-DR: Logframe Approach 73 24 CAPTAC-DR: Donor and Member Commitments and Pledges Phase I 76 25 Views on CAPTAC-DR reporting to SC (SC survey responses) 80 26 When Will CAPTAC-DR Financial Sustainability be Achieved? (SC survey responses) 81


101 | P a g e

APPENDICES


102 | P a g e

I. TERMS OF REFERENCE FOR THE EVALUATION

Contents

I. Background and Objectives of the Evaluation ...................................................................... A. Background ...................................................................................................................... B. Objectives of the Mid-Term Evaluation........................................................................... C. Steering Arrangements for the Mid-Term Evaluation .....................................................

TERMS OF REFERENCE FOR

THE INDEPENDENT MID-TERM EVALUATION OF

CAPTAC–DR

August 2012

II. Key Evaluation Questions ....................................................................................................

III. Methodology ....................................................................................................................... A. Evaluation Criteria and Ratings ....................................................................................... B. Information Sources .........................................................................................................

IV. Timing and Key Deliverables .............................................................................................

A. Timing .............................................................................................................................. B. Key Deliverables ..............................................................................................................

V. Evaluator Qualifications ....................................................................................................

APPENDICES ........................................................................................................................

A. Evaluation Sub-Criteria and Weights ............................................................................ B. List of Documents to be Provided to the Evaluator(s) ...................................................


103 | P a g e

I. BACKGROUND AND OBJECTIVES OF THE EVALUATION

A. Background

1. The Central America–Panama–Dominican Republic Regional Technical Assistance Center (CAPTAC–DR) was established in June 2009 in Guatemala City (Guatemala) with the overarching goal of assisting countries in the Central American region to strengthen their capacity for effective macroeconomic management and to support the region's integration into the world economy. Specifically, CAPTAC–DR has the objective of strengthening the institutional and human capacities of its members to modernize tax and customs administrations, undertake effective public financial management (PFM), manage robust monetary and financial systems, and produce high-quality macroeconomic statistics. Initially, the Center’s activities focused on five key areas representing common policy challenges to member countries, including those related to increased integration: tax and customs administration, medium-term expenditure frameworks and PFM-related issues, money markets, financial sector supervision, and macroeconomic statistics.

2. CAPTAC–DR provides technical assistance (TA) and training to seven beneficiary countries1 over a five-year cycle (June 2009 to April 2014).2 CAPTAC–DR operations are funded by contributions from the host country, the IMF, and fourteen bilateral and multilateral donors3, including all of the CAPTAC–DR beneficiary countries. The total budget of CAPTAC–DR over the current five-year funding cycle, including the IMF contribution and the host country in-kind contribution, is USD 35.4 million.

3. CAPTAC–DR operations are guided by a rolling annual work plan which is developed on the basis of beneficiary countries’ needs, is complementary to other forms of Fund TA, and is an integral part of the Fund's overall TA program. CAPTAC–DR is guided by a Steering Committee (SC), acting as an advisory body, and composed of representatives of the authorities of the countries served by the Center, the donors, and the IMF. It meets annually to review progress in the implementation of the work plan, discuss and endorse the work plan for the period ahead and to discuss strategic directions for the Center.

4. CAPTAC–DR assistance to beneficiary countries is currently provided by seven resident advisors and short-term experts and is based on assessment of and feedback from

1 Costa Rica, the Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua, and Panama.

2 Starting FY2013, CAPTAC–DR will align with the IMF fiscal year (May 1–April 30), therefore, it is expected that the first cycle will finish in April 2014.

3 CAPTAC-DR multilateral donors include: European Commission, Central American Bank for Economic Integration (CABEI), and Inter-American Development Bank (IADB). CAPTAC-DR bilateral donors include: Spain, Mexico, Canada (CIDA), Germany (GiZ), and all of the CAPTAC-DR beneficiary countries.

member countries on their TA and training needs, and Fund TA priorities for the region. TA in areas not covered by the resident advisors can be provided through CAPTAC–DR, with appropriate backstopping from the IMF headquarters, as was the case for monetary-financial statistics and public finance statistics.

B. Objectives of the Mid-Term Evaluation 5. The Terms and Conditions governing the establishment and operation of the multi- donor trust fund for CAPTAC-DR provides for an evaluation of the Center’s activities. According to those Terms and Conditions, an independent evaluation of the activities financed under the Subaccount “will be initiated no later than 40 months after the activities financed under the Subaccount with respect to each funding cycle have begun”. The CAPTAC–DR 2009 Program Document4 therefore foresees that an independent evaluation will be initiated “…after no fewer than three years of operation…[to] assess the effectiveness and sustainability of the Center and the TA provided by it, bearing in mind the long-term nature of capacity building, and…[to] formulate recommendations for improvement. The findings of the evaluation will inform discussions on the Center’s operations for the remainder of the initial five-year period and beyond.”5 The period covered by this evaluation is from the start of the Center’s operation in June 2009. The overall objective of the evaluation is therefore to assess the extent to which CAPTAC–DR is achieving its objectives in an efficient and effective manner, and the extent to which the TA and training delivered is relevant and sustainable. 6. An important task of the evaluation will be to assess the extent to which CAPTAC– DR is achieving the advantages typically associated with the Regional Technical Assistance Centers (RTACs) delivery modality: better identification of countries’ TA needs, rapid and flexible TA delivery, closer interaction with beneficiary country authorities, strengthened country ownership, and greater exchange of information with other TA providers and donors in the region. The evaluation will also look at the challenges and risks that the Center has faced in implementing TA and training, and whether additional steps from those indicated in the Center’s strategic logframe can be taken to address these challenges and/or mitigate risks, including the authorities’ response in member countries. In view of the possible extension of CAPTAC–DR activities beyond the first five–year duration, it is important for the evaluation to look at CAPTAC–DR's achievements and lessons learned and provide recommendations for improvement. 4 Page 20, paragraph 36 of the CAPTAC-DR Program Document. 5 The Program Document for CAPTAC–DR can be found at: http://www.imf.org/external/np/exr/key/RTACs.htm

http://www.imf.org/external/np/exr/key/RTACs.htm


104 | P a g e

DAC Criteria Key Evaluation Question(s) Relevance • How CAPTAC–DR determines the need for technical

assistance and training, including the processes used by members to determine their needs;

• The extent to which CAPTAC–DR activities are linked to national/regional strategies;

• Whether CAPTAC–DR is meeting the priority needs of member countries;

• Whether the national authorities are effective in ensuring strong country ownership of CAPTAC–DR activities;

• Whether CAPTAC–DR activities are appropriately focused in terms of subject areas, taking into account the IMF’s expertise and the priority needs of the beneficiary countries, integration with HQ and other Fund activities (e.g., other training and TA delivered including through the topical trust funds), and the work of other development partners.

Efficiency • Whether CAPTAC–DR is delivering activities in an efficient manner including: i) process and implementation efficiency (e.g., timeliness in executing the work plan); ii) efficient use of resources (financial and human), including attention to ensuring cost-effectiveness and, iii) monitoring and reporting.

Effectiveness • Whether CAPTAC–DR is appropriately focused on delivering outputs and outcomes that contribute to the achievement of priority reforms, including assessing, to the extent possible, outcomes and impacts at topic, country, and/or regional level;

• The quality and timeliness of activities undertaken and outputs produced, and the reporting and monitoring of these activities and outputs;

7. To address these objectives, the evaluator(s) will consider a set of interlinked evaluation questions (outlined in more detail below) focusing on: a) relevance of the TA and training activities provided by CAPTAC–DR; b) efficiency (i.e., the processes by which resources, e.g. human resources/expertise, financial resources and time, were converted to achieve outcomes and results); c) effectiveness of CAPTAC–DR TA and training – i.e., the extent to which the outcomes identified in the 2009 Program Document at country and regional levels are being achieved and, d) the extent to which these are likely to be sustained. While CAPTAC-DR is still at an early stage in its program cycle, some impacts may be discernible at individual, organizational or regulatory level. The evaluators should report on any discernible impacts. In addition, the evaluators should use the above information to form a view on the extent to which CAPTAC–DR and the RTAC model remains relevant in the regional context, including whether it is operating at an appropriate scale.

8. At the time of launching this evaluation, CAPTAC–DR is in the process of introducing new tools to support Results Based Management (RBM), in line with IMF strategy and the commitment set in the 2009 Program Document, including a strategic logical framework (log frame) for the Center overall, a series of topical log frames to guide the work of the resident advisors and a new format work plan to track inputs.6 While the primary focus of the evaluation will be assessment against the objectives set out in the 2009 Program Document, the evaluator may refer to these new tools as appropriate. The evaluator should also comment on the suitability of these instruments, and the extent to which an appropriate process is in place to support implementation of the RBM framework and, if relevant, propose refinements to the tools and/or suggestions to enhance RBM implementation.

C. Steering Arrangements for the Mid-Term Evaluation

9. While the Global Partnerships Division (GPD) of the IMF’s Institute for Capacity Development (ICD) will manage the procurement process and support information gathering by the evaluator, an Evaluation Sub-Committee (ESC) has been established to guide the evaluation. The creation of ESCs is in line with IMF evaluation practice and allows active participation of RTAC SCs in the evaluation process.

10. The role of the ESC is to provide strategic guidance for the evaluation and to ensure that key issues relevant to stakeholders are appropriately integrated into the evaluation process. The ESC will: 1) review, comment and agree on draft ToR; 2) review and comment on the Inception Note (see para. 18) prepared by the evaluator(s), and 3) review and comment on the Draft Evaluation Report. The ESC comprises 11 members, distributed as follows:

6 Ref. to IMF RBM Report.

• Member countries (3) • Donors (3) • IMF Area Department (1) • IMF TA Departments (3) • IMF Institute for Capacity Development (1)

II. KEY EVALUATION QUESTIONS 11. As noted above, the evaluation will address a set of interlinked evaluation questions that are in line with best international practice and reflect the Development Assistance Commitment (DAC) criteria of relevance, efficiency, effectiveness and sustainability. 12. The key evaluation questions can therefore be summarized as shown in Table 1:


105 | P a g e

• The extent to which beneficiaries have followed up advice/training provided.

Sustainability • The extent to which CAPTAC–DR TA and training is leading towards tangible and lasting results, and any factors affecting sustainability (e.g., country ownership, political support, an enabling environment for implementation of recommendations, etc.);

• The contribution of CAPTAC–DR to building regional capacity, as well as identifying and using, whenever possible, regional and local expertise;

• The extent to which the funding model is sustainable.

B. Information Sources

16. The evaluation will draw on information from a range of sources including documents and data available from the IMF (see Appendix B for a list of documents to be provided to the evaluators), interviews with a selected group of country authorities, the SC (including technical staff of the beneficiary authorities), donors (at both HQ and country level as appropriate) and a set of case studies. Importantly, each evaluation criteria should be assessed using at least two different information sources.

• Document and data analyses: The evaluators will review and analyze the full set of

available materials, including work plans, SC minutes, and internal transaction documents produced by the Coordinator, Resident Advisors and Short-Term Experts. Financial information on implementation will also be provided.

13. Building on the answers to the evaluation questions above, the evaluator(s) will consider: (i) the extent to which CAPTAC–DR has responded adequately to challenges, (ii) whether it continues to deliver additionality, i.e. whether the underlying intervention logic is still valid and, hence whether there is still a clear role for CAPTAC–DR; and (iii) whether the Center is operating at an appropriate scale and with an appropriate skill mix given its mandate and in an efficient and cost-effective manner.

14. The evaluation should also record and report on any significant lessons that can be drawn from the CAPTAC–DR experience, highlighting its strengths and weaknesses, and provide recommendations for improvement. The evaluation team will take note of the suggestions received from all interested parties during the course of the review on the direction of CAPTAC–DR operational modalities and areas of work.

III. METHODOLOGY

A. Evaluation Criteria and Ratings

15. A quantitative rating scheme will be used to a) ensure transparency in the judgments made by the evaluator(s) and b) to allow for aggregation across RTACs or functional areas. Appendix A therefore proposes a rating scheme consistent with that used in the 2009 External Evaluation of the African Regional Technical Assistance Centers. The Evaluator(s) will assess the adequacy of this scheme and propose amendments or refinements as necessary in the Inception Note, while ensuring consistency with the methodology used in the 2009 Evaluation.

• Interviews: The evaluator will conduct a series of semi-structured interviews with

country authorities and SC members. Interviews with country authorities are expected to cover the appropriateness and responsiveness of the TA and training provided (by both Resident Advisors and Short-Term Experts), and how well it addresses their needs, as well as explore and document any specific results that have emerged. Discussions with CAPTAC–DR staff should also be planned. The evaluators will also meet with staff in Washington, DC from the IMF’s TA departments, area departments and Institute for Capacity Development, as well as interviews with donors (at HQ and country level as appropriate).

• Survey tools: The evaluators may seek to consult a wider range of relevant

individuals from beneficiary authorities or the SC by means of a survey. • Case Studies (sample of countries/projects): The evaluators will visit at least 3–4

countries in the region to develop a series of case studies that can be used for dissemination purposes. Selection criteria will be discussed and agreed during the Inception Phase and outlined in the Inception Note.

IV. TIMING AND KEY DELIVERABLES

A. Timing

17. The evaluation is expected to commence in mid-October 2012, and end by Spring 2013. The evaluators will be contracted for a maximum of 145 person-working days (including travel) during that period. The evaluation process will be carried out through three phases: a desk phase, a field phase, and a synthesis phase.

• Desk Phase: At the latest 5 weeks after contract signing and before the commencement of the field phase, the evaluators will: (i) conduct a desk review of the relevant documents; (ii) visit IMF Headquarters to interview relevant staff in


106 | P a g e

Global Partnerships Division (GPD), TA departments and the Western Hemisphere Department (WHD), including the Center Coordinator and, (iii) prepare an Inception Note (see below on the contents), which will then be finalized in consultation with GPD and the ESC. During this phase, the evaluators will make contact with any important stakeholders (key donors, Chair of SC, etc.). The evaluator will hold a briefing with the IMF representatives before proceeding to the field phase. Total work time for this phase is estimated to be about 50 person days.

• Field Phase: The evaluators will visit CAPTAC–DR and beneficiaries in at least four

member countries. The evaluators will ensure adequate contact and consultation with, and involvement of the different stakeholders, working closely with the relevant government authorities and agencies, civil society and other relevant stakeholders in relevant academia/think tanks and, where relevant, donor offices. The field phase is estimated to take up to 40 person days, including travel time.

• Synthesis Phase: This phase is mainly devoted to the preparation of the draft report, as

well as any necessary follow-up interviews with IMF staff. The evaluator(s) will make sure that the assessments are objective and balanced and recommendations realistic. The evaluation team will prepare a draft evaluation report presenting the main findings, lessons learned, and recommendations, accompanied by a summary of the information gathered during key meetings. The draft report will be prepared in English and Spanish and submitted electronically and in hard copy format to the ESC in early 2013. The evaluation team will consider the comments provided on the draft report at their discretion and will draft a final report. The report should be finalized (with comments incorporated) by March 15, 2013. The evaluators will present their findings (final report) to the CAPTAC–DR Steering Committee, due to be held in April 2013. Subject to the approval of the Steering Committee, the final report will be posted on the Center's website. The total work time for this phase of the project will be up to person 55 days (including the briefing of the CAPTAC–DR SC and associated travel time).

B. Key Deliverables

18. The evaluation team will provide 3 key deliverables, to be produced in both English and Spanish, as set out below. Deliverables should be succinct, quality controlled by the evaluators and, in the case of the Final Evaluation Report and case studies, of a standard suitable for publication.

• An Inception Note: The Inception Note will set out the methodology for data

collection and analysis (including the criteria for selection of samples or case studies), draft interview guidelines and draft survey instruments, together with a detailed work plan for data collection, a list of potential interviewees and an outline of the Draft Mid-Term Evaluation Report table of contents.

• A Draft Evaluation Report, including a set of recommendations clearly identifying the main target and a timeline for implementation. Recommendations can address all aspects of CAPTAC-DR governance (including the roles of various actors), organizational issues and processes.

• A Final Evaluation Report.

V. EVALUATOR QUALIFICATIONS 19. The evaluation team should demonstrate the following qualifications:

• Extensive knowledge of the issues covered by IMF TA and training; experience and expertise in the delivery and review of TA in the areas covered by CAPTAC-DR; and some capacity and background in macroeconomic policy making would be desirable;

• Experience in the region and countries covered by CAPTAC–DR;

• Extensive experience in evaluation, including evaluation of TA and training;

• Substantive input from local experts insofar as possible;

• Ability to work effectively in English and Spanish.

20. In addition, the evaluator is also expected to outline and implement a quality control mechanism to ensure that drafts of key deliverables are of an appropriate quality.


107 | P a g e

DAC Criteria and Weighting

Key Evaluation Question(s) Sub-Criteria and Weights

surveillance, and lending activities of IMF HQ; evidence of consistency could be examined by drawing on the results obtained from a review of documents and interviews with staff of the WHD Department, IMF Resident Representatives and the TA Departments.

Efficiency (22%) • Whether CAPTAC–DR is delivering activities in an efficient manner while ensuring the timeliness of expert input.

The mid-term evaluation will also consider issues of efficiency, including the management and use of resources and the extent to which locational efficiencies have been achieved, .e.g. the cost benefits of being based in the region, in assessing the operations and administration of CAPTAC–DR.

Ratings and weights will be distributed as follows:


(i) Process and Implementation Efficiency (40%): covered such factors as IMF’s internal management of CAPTAC–DR activities, appropriate counterpart/workshop participant selection and the quality and timeliness of management and backstopping of CAPTAC–DR TA and training by HQ staff; the timely recruitment of qualified Resident Advisors, the efficiency of planning and executing the TA and training delivery.

(ii) Efficient Use of Resources (40%): Whether expenditures have been in line with annual work plans, whether CAPTAC–DR TA and training has proven to be cost-effective compared to other TA delivery modes; whether efficiency gains can be made; respondents to the CAPTAC–DR TA and training evaluation survey may be

APPENDICES

A. Evaluation Sub-Criteria and Weights



Relevance (32%) • Whether CAPTAC–DR is meeting the priority needs of member countries and whether there has been strong country ownership of CAPTAC–DR activities;

• Whether CAPTAC–DR activities are appropriately focused in terms of subject areas, taking into account the IMF’s expertise and the priority needs of the beneficiary countries, HQ activities and the work of other development partners.

A starting point for the evaluation will be to compile an overview of quantitative and qualitative data on activities (TA and training) delivered by CAPTAC–DR since its inception. This will include an assessment of whether the TA and training was relevant in terms of: a) priorities identified in the Program Document, b) the needs of the member countries and region, and c) whether it was appropriately coordinated with other stakeholders, in terms of timing and/or substance.


(i) Consistency with Program Document and Government Priorities (60%): particular attention to the linkage between CAPTAC–DR technical assistance and training and macro economic reform programs and capacity development programs formulated by ministries of finance, central banks and statistical agencies, regional organizations and other recipients of CAPTAC–DR TA and training.

(ii) Coordination with Development Partners (20%): Whether there has been sufficient coordination with development partners (the importance of donor coordination was highlighted in the 2009 AFRITACs evaluation).

(iii) Consistency with IMF Headquarters Activities (20%): The extent to which CAPTAC–DR TA and training is integrated with TA,


108 | P a g e



asked to give their perceptions on the relative cost efficiency of CAPTAC– DR TA and training compared to other TA providers and IMF HQ.

(iii) Monitoring and Reporting (20%): degree to which CAPTAC–DR uses self evaluation (i.e. monitoring) and better reporting to improve the efficiency and effectiveness of their activities. That involved examining the use CAPTAC-DR is making of TAIMS and efforts to put RBM in place.

Effectiveness (28%)

• Whether CAPTAC–DR is appropriately focused on delivering outputs that contribute to the achievement of priority reforms, including assessing, to the extent possible, outcomes and impacts at topic, country and/or regional level;

• The quality and timeliness of activities undertaken and outputs produced and the reporting and monitoring of these activities and outputs.

(i) Use of CAPTAC–DR Outputs (40%): including an assessment of the use of the outputs of each topic area and whether the outputs are leading, or are likely to lead, to the planned outcomes (identified in the Program Document); whether the TA and training has delivered outputs that contributed to the achievement of beneficiary country reforms capacity building.

(ii) Planned vs. Actual Achievements (30%): The actual outputs were compared to the planned outcomes stated in the CAPTAC–DR Program Document, work programs and other documentation. In many cases, there will be an unfinished agenda. In such cases, the Evaluator(s) may form a judgment as to whether the expected outcomes are being achieved (increased knowledge, skills, etc.) and where possible, a change in the level of beneficiary performance).

(iii) Significance of Contribution to Developing Core Economic Functions and Institution Building (30%): CAPTAC–DR activities and outputs assessed to estimate their contribution to the development of core economic functions and



institution building. Evaluators need to be careful about distinguishing between attribution and contribution. While it may be true that progress was made, this progress may have reflected the joint efforts of the CAPTAC–DR, HQ assistance, support provided by other development partners and government initiatives. The Evaluator(s) may estimate the relative importance of CAPTAC–DR contributions by considering whether the results could have been achieved without CAPTAC–DR involvement.

Sustainability (18%)

• The extent to which CAPTAC– DR TA and training is leading or has led to tangible and lasting results;

• Whether constraints faced by CAPTAC–DR member countries prevent them from taking full advantage of CAPTAC–DR TA and training, and how such constraints can be addressed.

(i) Executing Agency Ownership and Use of the Outputs (75%): An important factor to be considered is whether the workshop participants/trainees use the knowledge gained on the job. Participant selection may be examined in the context of this sub-criteria. Whether the TA outputs are embedded in the routine businesses practices of the executing agencies.

(ii) Promoting the Use of regional Expertise (25%): Has CAPTAC–DR promoted the use of regional expertise, something that should contribute to sustainability in the longer term by building local capacity. The Evaluation Team assessed how effective the Centers were in identifying regional expertise.

B. List of Documents to be Provided to the Evaluator(s) • Program Document,


109 | P a g e

• Work Plans, • Annual Reports, • Minutes of Steering Committee meetings, • Activity reports, • Budget, • Technical Assistance reports, • Previous evaluations of RTACs, • General information on the Fund TA, RTACs, etc.


110 | P a g e

II. RESPONSES TO TORS QUESTIONS IN THE REPORT This Section provides an index of the specific questions referred to in the Evaluation’s TORs/SOW and the relevant sections in the Report where we propose to provide responses to them. Where responses are covered in more than one section, we will at a minimum sought to indicate the section where the principal component of the response is dealt with. Whilst reference is made to the KEQs in the TORs in this section, in the final report we will provide a similar report relating the sections with the revised KEQs. Question Response in Section Overall Objective (Section 5 of SOW) Assess the extent to which CAPTAC-DR is achieving its objectives, in an efficient and

effective manner, and the extent to which the training and TA delivered is sustainable 3, 4, 5, 6, 8

Extent to which CAPTAC-DR has built on the previous multi-year cycles and whether results of previous phases have been sustained in the current phase

2, 4

Achievement of Objectives of CAPTAC-DR Model (Section 6) Better identification of TA needs; rapid and flexible TA delivery; closer interaction with

beneficiary country authorities; strengthened country ownership; and greater exchange of information with other TA providers and donors in the region.

6

Challenges, known risks faced by Centre in implementing TA and delivering training, and additional steps from those indicated in Centre’s strategic logframe to address risks/take mitigating action, including authorities’ response in member countries.

4,5, 9

Achievements, lessons learned and recommendations 3, 11 Other Issues (Sections 7 and 8) Extent to which CAPTAC-DR and its model remains relevant in the regional context, including

whether it is operating at an appropriate scale and in an efficient manner. 6, 7, 8

Refer to relevance and expected effectiveness of new tools to support RBM (strategic logframe, topical logframes per country, new format work plan to track inputs, separate reporting database, suitability of tools, and extent to which appropriate process is in place to support implementation of RBM, and if relevant propose refinements to the tools and/or suggestions to enhance RBM implementation.

9

Key Evaluation Questions (Section 12) Relevance How CAPTAC-DR determines the need for TA and training, including the processes used by

members to determine their needs. 8.2, 6.7

The extent to which CAPTAC-DR activities are linked to national/regional strategies. 3.3,3.6 Vol. II Whether CAPTAC-DR is meeting the priority needs of member countries. 3.5, 4.3, 6.2, Vol. II Whether the national authorities are effective in ensuring strong country ownership of

CAPTAC-DR activities. 3.6,6.8

Whether CAPTAC-DR activities and staffing are appropriately focused in terms of subject areas, taking into account the IMF’s expertise and the priority needs of the beneficiary countries, integration with HQ and other Fund activities (e.g. other training and TA delivered including through the topical trust funds) and the work of other development partners.

8

Efficiency Whether CAPTAC-DR is delivering activities in an efficient manner including: i) process and

implementation efficiency (e.g. timeliness in executing the work plan); ii) efficient use of resources (human and financial) and attention to cost effectiveness and, iii) monitoring and reporting.

3.5,4.3

Effectiveness Whether CAPTAC-DR is appropriately focused on delivering outputs and outcomes that

contribute to the achievement of priority reforms, including assessing, to the extent possible, outcomes and impacts at topic, country and/or regional level.

3.5, 4, 5, Vol. II

Whether interventions reflect learning from experience both in CAPTAC-DR and beyond to ensure the most effective practices are used to secure maximum output and outcome effectiveness.

The quality of TA delivered, including the added value of backstopping. 3.3, 8.8 The extent to which beneficiaries have followed up advice/training provided. 4, 5, Vol. II

Independent Interim Evaluation of CAPTAC-DR. Volume I: Main Report

111 | P a g e

Question Response in Section Sustainability The extent to which CAPTAC-DR TA and training has led to tangible and lasting results and

factors affecting sustainability. 6.2, 4, 5

The contribution of CAPTAC-DR to building regional capacity, as well as identifying and using regional expertise, including local and regional expertise.

6.6, 6.9, 10.3

Whether staff turnover is well managed to ensure adequate institutional learning in CAPTAC-DR and with beneficiary country relationships.

8.5, 8.7

Sustainability of the funding model. 10.6, 6.5 Other issues (Sections 13, 14 and 15) Extent to which CAPTAC-DR interventions have directly or indirectly contributed to

impacts especially on enhanced organizational performance, improved regulatory frameworks, etc. Consider the extent to which interventions are designed and implemented to ensure impact. Not to be rated.

3, 4, 5, Vol. II

Drawing on questions above, consider (i) the extent to which CAPTAC-DR has responded adequately to challenges, (ii) whether it continues to deliver additionality, i.e. whether the underlying intervention logic is still valid and, hence whether there still a clear role for CAPTAC-DR; and (iii) whether the Center is operating at an appropriate scale and with an appropriate skill mix given its mandate and in an efficient and cost-effective manner.

6

Record and report on any significant lessons that can be drawn from the CAPTAC-DR experience, highlighting its strengths and weaknesses, and will provide recommendations for improvement, including implications for the RBM framework.

11

Take note of the suggestions received during the course of the review on the direction of CAPTAC-DR’s operational modalities and areas of work.

6.11, 6.8, 8


112 | P a g e

III. EVALUATION TOOLS AND MAJOR TASK GROUPS

DAC Criteria

Questions Desk Top Reviews

Surveys TA & Work-shops

IMF HQ TA Depts, WHD & ICDGP

SC & Other Donors (Incl.

Survey)

CAPTAC-DR & RAs

Field Trips (3 countries)

Incl. Stakeholder Interviews

Relevance 1. How CAPTAC-DR determines the need for TA & training, including members’ processes to determine needs.

Project files- include aide memoirs, diagnostics, and other sources

Yes Yes Yes High-level officials, country plans (as available), studies.

2. The extent to which CAPTAC-DR activities are linked to national/regional strategies.

Sectorial strategies where available, action plans

Yes Yes Yes Yes Country plans, sector strategies.

3. Whether CAPTAC-DR is meeting the priority needs of member countries.

Yes Yes Yes Nat’l plans; high level officials; PRSPs FSAP,

4. Whether national authorities ensuring strong ownership of CAPTAC-DR activities.

Yes Yes Yes Yes High-level officials.

5. Whether activities are well focused: subject areas, regard to IMF’s expertise & country priority needs, integration with HQ & other Fund activities (e.g. training), & other development partners.

IMF surveillance reports. Overall review of IMF’s and other development partners’ technical working areas

Yes Yes Yes Yes

Effective-ness

1. Whether CAPTAC-DR is appropriately focused on delivering outputs and outcomes that contribute to the achievement of priority reforms, including assessing, to the extent possible, outcomes and impacts at topic, country and/or regional level.

Project monitoring reports versus Strategic& topical logframes Other IMF surveillance reports

Yes Other donor delivered TA and coordination with CAPTAC-DR

Yes Case studies with emphasis on Results planned and measured

2. Whether interventions reflect learning from experience both in CAPTAC-DR and beyond to ensure the most effective practices are used to secure maximum output & outcome effectiveness.

Yes Yes Yes Use of logframes

Yes

3. Outputs (including quality and timeliness). Project files Yes Yes Yes Yes 4. Outcomes achieved (including extent to which

beneficiaries have followed up and implemented TA/training lessons).

Project files Yes Yes Yes Yes

5. Outcomes in development. Project files Yes Yes Yes Yes 6. Project monitoring reports, IMF rating. Project files Yes Yes Yes


113 | P a g e

DAC Criteria





Survey)

CAPTAC-DR & RAs



7. Project monitoring- recipient rating. Yes Yes Yes 8. Recipient overall satisfaction rating of

workshops. Yes Yes

Efficiency/ Opera-tions

1. Whether CAPTAC-DR delivered efficiently: process & implementation (e.g. work plan timeliness); use of human % financial resources & attention to costs; monitoring & reporting.

Monitoring reports, backstopping, budget vs actual reports, CARTAC cost effective-ness study, etc.)

Yes Yes Yes Yes Yes

2. Support of HQ MIS. MIS system Yes Yes 3. Project design & budgeting process. Project files Yes Yes Yes 4. TA risk assessment. Project files Yes Yes Yes Yes 5. Work plan/RAP process. SC reports Yes 6. Procurement process. Yes Yes 7. Speed of project cycle- application to approval. Yes Yes Yes 8. Approval to commencement. Yes Yes Yes 9. Effectiveness of monitoring, post completion &

incorporation of lessons learnt. Monitoring reports, TA reports, Briefing Papers

Yes Yes

10. Effectiveness/added value of backstopping. Comments received Yes Yes 11. RBM implementation Status. Files, RBM Working Group

Report Yes Yes

Sustainability

1. The extent to which CAPTAC-DR TA & training led to tangible & lasting results & factors affecting sustainability.

TA reports Yes Yes Yes Yes

2. The contribution of CAPTAC-DR to building regional capacity, + identifying & using local and regional expertise.

Yes Yes Yes Yes Yes

3. Whether staff turnover ensures inst. learning in CAPTAC-DR & with beneficiary country relationships.

Staff turnover information Yes Yes Yes Yes mainly beneficiaries

4. Sustainability of funding model. Yes Yes Strategy Regional versus country approach. SC reports Yes Yes Yes Yes Yes

Coordination with SC donors. SC reports Yes Yes Yes Coordination with other donors. SC reports Yes Yes Yes How CAPTAC-DR adjusts/cooperates with other IMF topical trust funds e.g. Tax policy and Administration

Yes Yes Yes


114 | P a g e

DAC Criteria





Survey)

CAPTAC-DR & RAs



TTF. Adequacy of funding vis a vis demand. Commitment statements, work

plan, financial reports Yes Yes

Extent to which CAPTAC-DR strategy is geared to achieving advantages re., to RTAC Model (actual achievements covered under other headings): Better identification of TA needs; Rapid/Flexible TA delivery; Closer interaction with beneficiaries Strengthened country ownership Greater exchange of info with stakeholders /

donors.

Yes Yes Yes Yes Yes Yes

Review existing strategic logframe & suggest improve-ments, esp. in respect to risk assessment & mitigation.

Yes Program Doc. Yes Yes Yes Yes

Gover-nance

Adequacy of reports to SC. SC reports Yes SC influence on CAPTAC-DR policy/strategy. SC minutes Yes Yes Recipient view on multi-donor benefit. Yes Yes IMF view on multi-donor benefit. Yes Yes Clear charter for SC. LoUs, RTAC Guidance Note.


115 | P a g e

IV. KEY EVALUATION QUESTIONS

DAC Criteria KEQs at CAPTAC-DR Program level KEQs at individual case study/TA/training intervention level

Relevance Consistency with Program Document and Government Priorities Consistency of CAPTAC activities with Program

Document and CPTAC strategy. Consistency of CAPTAC activities with

national/regional priorities: o Extent to which activities are linked to

and support national strategies; o Extent to which the Program supports

regional initiatives; o Extent to which activities are linked to

and support sector/topical reform strategy;

o Effectiveness of TA selection process;23 o Extent to which strong country ownership

of CAPTAC activities has been demonstrated.

Whether CAPTAC’s work plans are appropriately sequenced, given the completed/outstanding reform needs for that sector.

Consistency with IMF Headquarter/other activities Whether CAPTAC’s activities are appropriately

focused in terms of subject areas, taking into account the IMF’s expertise, and integration with HQ and other Fund activities (e.g. other training and TA delivered including through the topical trust funds).

Coordination with Development Partners Whether CAPTAC’s activities generally are

effectively coordinated with and complement the work of other development partners.

Context of donor/TA landscape of IMF TA How relevant is the CAPTAC program when

compared to other programs on offer? Why do countries choose the CAPTAC as the

TA provider?

Consistency with Program Document and Government Priorities Consistency of activity with Program

Document, CAPTAC strategy, and approved work plan.

Consistency of intervention(s) with national/regional priorities: o Extent to which activity is linked to and

supports national strategy ; o Extent to which activity is linked to and

supports regional strategy (if one exists) o Extent to which activity is linked to and

supports sector/topical reform strategy; o Extent to which strong

country/institutional ownership of activity has been demonstrated.

Whether the activity is appropriately sequenced, given the completed/outstanding reform needs for that sector/institution: o E.g., appropriateness of project

expected outcomes given country/institutional absorptive and implementation capacity.

Consistency with IMF Headquarter/other activities Whether activity is appropriately focused in

terms of subject area, taking into account the IMF’s expertise, and integration with HQ and other Fund activities (e.g. other training and TA delivered including through the topical trust funds).

Coordination with Development Partners Whether the intervention has been

effectively coordinated with and complements the work of other development partners.

Context of donor/TA landscape of CAPTAC TA Why did the TA recipient choose the CAPTAC

as the TA provider as opposed to other donors?

How do workshop participants compare those provided by CAPTAC with workshops from other donors?

23 Including what is done with projects that are not taken forward


116 | P a g e


Effectiveness

Impact CAPTAC and aggregated project-level Impact24

as defined in the Program Document, achieved or likely to be achieved.

Outcomes RTAC and aggregated project-level Outcomes

(short, medium and long term, is/as defined), as defined in the Program Document, achieved or likely to be achieved, including e.g., implementation of advice.

Significance of CAPTAC’s contribution/likely contribution to developing core economic functions and institution building in the country/region.

Outputs Outputs achieved or likely to be achieved at

aggregated project level, including: o Timeliness; o Quality of reports/workshop

presentations; o Appropriateness/applicability of advice,

given existing capacity/constraints; Whether Outputs are likely to produce intended Outcomes.

Impact Intervention’s expected Impact achieved or

likely to be achieved (to the extent defined in the activity’s logframe).

Outcomes Intervention’s Outcomes achieved or likely to

be achieved (as defined in its logframe25): Short, medium and long term Outcomes

Significance of CAPTAC’s contribution/likely contribution to developing core economic functions and institution building in the country/region, through the activity.

Outputs Intervention’s Outputs achieved or likely to

be achieved (as defined in its logframe), including: o Timeliness of reports/workshop

presentations; o Quality of reports/workshop

presentations; o Appropriateness/applicability of advice,

given existing capacity/constraints; o Whether Outputs are likely to produce

intended Outcomes. Efficiency

Process and implementation efficiency Management/Steering mechanism for

developing CAPTAC’s strategy and work plans is efficient and optimizes results delivery;

Timeliness in executing work plans; Appropriateness/effectiveness of IMF’s

internal management of CAPTAC activities; Appropriateness of selection of

counterpart/workshop participants; Quality and timeliness of management and

backstopping; Efficient use of resources (human and financial) and attention to cost effectiveness Appropriateness of staffing

composition/effectiveness of HR function, including RA turnover.

Appropriateness of work allocation between HQ and CAPTAC;

Appropriateness of degree and nature of use

Process and implementation efficiency TA/workshop design has been efficiently

carried out; Timeliness in executing the activity; Appropriateness/effectiveness of IMF’s

internal management of the activity; Appropriateness of selection of

counterpart/workshop participants (where relevant);

Quality and timeliness of management and backstopping in relation to activity;

The efficiency of planning and executing the TA/training delivery.

Efficient use of resources (human and financial) and attention to cost effectiveness Appropriateness of staffing composition for

the intervention; Appropriateness of work allocation between

HQ and CAPTAC in relation to activity; Appropriateness of degree and nature of use

of regional/ local expertise;

24 Impact assessed qualitatively only, and only where practical. 25 We reviewed the logframes for project proposals where available and/or where needed, suggest refinements and indicators to measure performance against.


117 | P a g e


of regional/ local expertise; Whether expenditures have been in line with

medium term/multi-year and annual work plans and evidence of analysis of variances;

Whether CAPTAC TA and training, given its results, has proven to be cost-effective compared to other TA delivery modes, given results;

Whether opportunities for efficiency gains are considered and explored;

Allocation of time between TA delivery and project management;

Perceptions on the relative cost efficiency of CAPTAC TA and training compared to other TA providers and IMF HQ.

Monitoring and reporting Appropriateness of overall M&E framework for

CAPTAC; Degree to which CAPTAC uses self-evaluation

(i.e. monitoring) and better reporting to improve the efficiency and effectiveness of their activities;

Effectiveness in making use of TAIMS to monitor and manage projects;

Effectiveness/progress in efforts to put RBM in place and its usage in managing CAPTAC.

Whether systems exist and they are effectively used for interventions to reflect learning from experience both in CAPTAC and beyond to ensure the most effective practices are used to secure maximum output and outcome effectiveness.

Whether expenditures have been in line with activity budget and evidence of analysis of variances;

Whether activity, given its results, has been cost-effective compared to other TA delivery modes, given results;

Whether opportunities for efficiency gains during the intervention have been considered and explored.

Monitoring and reporting Evidence of effective use of self-evaluation

(i.e. monitoring) and reporting to improve the efficiency and effectiveness of activity;

Effectiveness in making use of TAIMS to monitor and manage project/workshop;

Effectiveness/progress in use of RBM and its usage in managing the intervention.

Use of RBM to manage activity; Incorporation of lessons learnt in project

design and implementation.

Sustainability

Sustainability of CAPTAC activity The extent to which CAPTAC TA and training

has led to tangible and lasting results. Whether CAPTAC’s Outcomes will last beyond

the completion of the current phase of CAPTAC activity.

Degree to which CAPTAC’s systems allow for retention of organizational memory (e.g. to avoid duplication of effort, improved hand-overs, etc.)

Sustainability of the CAPTAC funding model How sustainable is the CAPTAC funding model? Is there a clear strategy/progress toward funding sustainability being achieved?

Sustainability of TA activity Outcomes from intervention will last

beyond/continue after completion of TA/training.

For TA/training designed to deliver Sustainability, local institutions/ capacities have been strengthened to sustain results beyond the life of the intervention:

E.g., absorptive capacity improved and developed to sustainable level; in-house training capacity built, where relevant.

For interventions designed to deliver sustainability, financial sustainability has been achieved (i.e., recipients will be implement the advice from own/other financial resources in a sustainable manner).

Effectiveness in maintenance and use of institutional memory relating to the activity;


118 | P a g e


The contribution of CAPTAC to building regional human capacity CAPTAC’s contribution in identifying and using

local and regional expertise to build local advisory capacity.

Contribution to building sustainable regional TA, implementation capability.


119 | P a g e

V. LIST OF MEETINGS AND INTERVIEWS IMF HEADQUARTERS Department Position Name Global Partnerships Division, ICD Division Chief L. Effie Psalida

Deputy Division Chief Ulrich Jacoby Deputy Division Chief Xiangming Li TA Officer (AFE) Bineta Ba TA Officer (AFW) Jonathan A. Wolsey TA Officer (CAPTAC-DR) Katarzyna Kardas Senior TA Officer Liz Cunningham Budget Officer Barrie Williams Senior TA Officer (re RBM IT systems) Harish Mendis Senior Budget Assistant Harriet Adria Senior Budget Assistant Maria McClain

Strategy & Evaluation Division, ICD Assistant Director John Green Western Hemisphere Department Deputy Director Miguel Savastano Regional Representative Fernando Delgado Deputy Director Adrienne Cheasty Division Chief Lorenzo Figliuoli Fiscal Affairs Department Division Chief, Resource and Information

Management Brian L Christensen

Deputy Director Gilbert Terrier Division Chief, Fiscal Policy and Surveillance Division

Julio Escolano

Economist Eliko Pedastaar Budget Assistant Walid Saifullah Division Chief, Revenue Administration II Katherine O. Baer Senior Economist Patricio Castro TA Advisor Enrique Rojas Deputy Division Chief, Revenue Administration Division

David Kloeden

Deputy Division Chief, Public Financial Management Division II

Mario Falcao Pessoa

Technical Assistance Advisor, Public Financial Management Division II

Carlos Tamarit

Statistics Department Technical Assistance Officer, Resource Management Division

Claudia Mariel

Senior Economist, Resource Management Division

Subramanian S. Sriram

Deputy Division Chief, Balance of Payments Division

Eduardo Valdiva-Velarde

Deputy Division Chief Robert Dippelsman Deputy Division Chief Maria Mantcheva Economist Michael Stanger Senior Economist Marcelo Dinenzon

Monetary and Capital Markets Division Chief Mark O’Brien Technical Assistance Officer Ava Ayrton Technical Assistance Advisor Dilek Goncalves Deputy Division Chief, TA Division Karl Driessen

Office of Budget and Planning Assistant to the Director Heikki Hatanpää

Budget Officer

Nina. Kappor


120 | P a g e

Finance Accountant Gaielle Latortue Accounts Officer Felincia Lofton IMF Institute Director Jorge Roldos Marco Espinoza CAPTAC-DR Institution Position Name Center Coordinator Ugo Fasano External Statistics Advisor Consuelo Soto National Accounts Statistics Advisor Lisbeth Rivas Public Finance Management Advisor Jean Baptiste Gros Revenue Policy and Administration Advisor-

Customs Asael Perez

Revenue Policy and Administration Advisor-Tax Oscar Vazquezi Monetary Policy and Operations Advisor Alfredo Blanco Bank Supervision Advisor Oscar Basso Office Manager Astrid Perdomo

GUATEMALA Institution Position Name Banco de Guatemala Governor Edgar Barquin General Manager Sergio Recinos Economic Manager Oscar Monterroso Financial Manager Romulo Divas Director Supervision and Member of CCSBSO

Liaison Committee Carlos Perez

Supervisor of Department of Supervision of Integral Risk (DISR)

Rodrigo Caal

Supervisor of DISR Sergio Picen Supervisor of DISR Luis Mazariegos Coordinator of TA projects Guadalupe Barral Caballero

Steering Committee Members CIDA Tensae Bernhanu EU- Regional Head Debora Marignani GiZ- Guatemala Office Hans Fuchs IADB Alberto Barreix

CASE STUDY: Cross Border Consolidated Supervision & Operational Risk STX employed for all countries in the region on

Cross Border Socorro Heysen

COSTA RICA Institution Position Name Banco Central President and President of CAMC Rodrigo Bolanos General Manager of Supervision of Financial

Entities & CCSBSO President Javier Cascante

Director of Division of Supervision of Private Banks (SUGEF)

Mrs. Cecilia Sancho

Supervisor SUGEF Genara Segura Supervisor SUGEF Guillermo Zeledo Acting Secretary/Deputy Director CAMC Angel Arita CAMC member Jorge Barboza CAMC member Jose Iraheta


121 | P a g e

CASE STUDY: Customs Administration Head of the Area

Relations and External Affairs Maria Iris Céspedes Núñez

Relations and External Affairs Vanessa Vargas Sancho Advisor of the Minister of Finance Ana Y. Rodríguez Director of Auditing Maribel Abarca Sandoval Head of Department of Origen Yamilet Miranda Carbajal Head of Department of Internal Audit Lilian Gondez Morales Head of Dept. of External Audit Floribel Chaves Arce Head of National Body Customs Valuation and

Verification Arturo Roldán

Head of Department of Control Posterior María Cecilia Sánchez Garita Directorate of Risk Management Kathia Varela Directorate Technical Management Heberto Noguera Suarez Directorate Technical Management Liliana Ureña Solís Area of Relations and External Affairs Vanessa Vargas Directorate of Risk Management Kathia Varela Analyst in Intelligence Department José Luis Vargas Murillo Head of the Department of Risk Management Kirma Chaves Hernández Head of Department of Planning Auditing Hannia Campos Katia Varela González Risk Department Analyst José Luis Vargas Murillo Risk Department Analyst Kirma Chaves Hernández Head of the Department of Risk Management Tania Solares Campos Customs Procedures Yesenia Morales Martínez Customs Procedures Mauricio Morales Barrasal Customs Procedures Roberto Oviedo Umaña Customs Procedures Liliana Ureña Solís Directorate of Regulations Karina Villalobos Muñoz Directorate of Regulations Narzic Mendoza Solís

NICARAGUA AND EL SALVADOR CASE STUDY: External Statistics NICARAGUA Executive Director, FUNIDES, Carlos Muniz Senior Economist, FUNIDES, Luis Alberto Alaniz Area Executive, Central Bank of Nicaragua Francisco Morales Chief National Accounts, Central Bank of

Nicaragua Hiparco Loaiziga

Chief Economic Studies Division, Central Bank of Nicaragua

Nina Conrado

Consultant- CAPTAC-DR Alfonso Sales Economist, FIDEG (think tank) Alejandro Guido Economist. FIDEG Enrique Alaniz EL SALVADOR Chief, Economic Studies Division, Central Bank

of El Salvador Luis Alberto Aquino

Analyst, Central Bank of El Salvador Ileana Bolanos Analyst, Central Bank of El Salvador Cesar Mateo Villamariona Chief Researcher, FUSADES (think tank) Pedro Argumedo Researcher, FUSADES Carlos Orellano Researcher, FUSADES Luz Maria de Portillo Chief Financial Officer, Caterpillar Walter Barrios Chief, Accounting Department, Caterpillar Oscar Bonilla