imf central america‐ panama‐dominican republic regional ... · costa rica’s relative trade...

IMFCentralAmerica‐Panama‐DominicanRepublicRegionalTechnicalAssistanceCenter

(CAPTAC‐DR)

Independent Mid-Term Evaluation

Phase I: July 2009 to date

Volume II: Case Studies and Desktop ReviewsNovember 2013

Consultants

Robert Woodbridge

Subhrendu Chatterji

Pablo Lo Moro

James Hanson

Pedro Belli

Alejandro Garcia Caballos

Independent Interim Evaluation of CAPTAC-DR. Volume II: Case Studies and Desktop Reviews

1 | P a g e

CONTENTS 1. EVALUATION RATING METHODOLOGY ..................................................................... 4

1.1 Evaluation Criteria and Rating System .......................................................................................... 4

1.2 Desktop Reviews ........................................................................................................................... 5

1.3 Case Studies .................................................................................................................................. 5

CASE STUDIES ............................................................................................................ 6

2. COSTA RICA: CUSTOMS ADMINISTRATION ............................................................... 7

2.1 Introduction and Summary of Results .......................................................................................... 7

2.2 Information Reviewed .................................................................................................................. 7

2.3 Overview ....................................................................................................................................... 8

2.4 The Results Chain for Customs Administration in Costa Rica ....................................................... 9

2.5 CAPTAC-DR Technical Assistance ................................................................................................ 11

2.6 Principal Conclusions .................................................................................................................. 16

2.7 Key Findings and Recommendations .......................................................................................... 16

2.8 Annex .......................................................................................................................................... 18

2.8.1 Suggested Logical Framework ................................................................................................. 18

2.8.2 Evaluation KEQs and Ratings ................................................................................................... 19

3. REGIONAL: CROSS-BORDER CONSOLIDATED SUPERVISION ........................................ 22

3.1 Introduction and Summary of Results ........................................................................................ 22

3.2 Information Reviewed ................................................................................................................ 22

3.3 Overview ..................................................................................................................................... 23

3.4 The Results Chain for Cross-Border Consolidated Supervision .................................................. 26




3.8 Annex .......................................................................................................................................... 36




2 | P a g e

4. REGIONAL: OPERATIONAL RISK SUPERVISION ......................................................... 43



4.3 Overview ..................................................................................................................................... 44

4.4 The Results Chain for Operational Risk Supervision ................................................................... 45




4.8 Annex .......................................................................................................................................... 53



5. EL SALVADOR: EXTERNAL SECTOR STATISTICS ......................................................... 58



5.3 Overview ..................................................................................................................................... 58

5.4 The Results Chain for external STATISTICS IN el salvador ........................................................... 59




5.8 Annex .......................................................................................................................................... 66

5.8.1 suggested Logical framework ................................................................................................. 66


DESKTOP REVIEWS.................................................................................................... 70

6. EL SALVADOR AND HONDURAS: LARGE TAXPAYERS UNITS ........................................ 71


6.2 Background ................................................................................................................................. 71


6.4 Suggested Logical Framework..................................................................................................... 74

6.5 Evaluation KEQs and Ratings....................................................................................................... 75



3 | P a g e

7. NICARAGUA: TAX-CUSTOMS INFORMATION SHARING .............................................. 79


7.2 Background ................................................................................................................................. 79



7.5 Evaluation Ratings ....................................................................................................................... 82


8. REGIONAL: SINGLE TREASURY ACCOUNT ................................................................ 85


8.2 Background ................................................................................................................................. 85


8.4 Suggested logical Framework ..................................................................................................... 89

8.5 Evaluation Ratings ....................................................................................................................... 90


9. MULTI-COUNTRY: SUPPORT TO MONETARY OPERATIONS ......................................... 94


9.2 Background ................................................................................................................................. 94



9.5 Evaluation Ratings ..................................................................................................................... 100

9.6 Key Findings and Recommendations ........................................................................................ 102

10. REGIONAL: NATIONAL ACCOUNTS STATISTICS ...................................................... 104

10.1 Introduction and Summary of Results ...................................................................................... 104

10.2 Background ............................................................................................................................... 104

10.3 CAPTAC-DR Technical Assistance .............................................................................................. 105

10.4 Suggested Logical Framework................................................................................................... 107

10.5 Evaluation Ratings ..................................................................................................................... 110

10.6 Key Findings and Recommendations ........................................................................................ 112


4 | P a g e

1. EVALUATION RATING METHODOLOGY

1.1 EVALUATION CRITERIA AND RATING SYSTEM The Terms of Reference for the Evaluation (‘ToRs’) set out a series of Key Evaluation Questions (‘KEQs’) for each of the OECD DAC criteria that the Evaluation is expected to address. It proposed a quantitative rating methodology, along with a framework for assigning weights to the scores for the different DAC criteria in order to arrive at a unitary score for each evaluated project, workshop or the CAPTAC-DR program as a whole. We adopted the rating system for individual DAC criteria of: Relevance (‘R’) Effectiveness

o Impact (‘E/I’)1 o Outcomes (‘E/OC’) o Outputs (‘E/OP’)

Efficiency (‘E’) Sustainability (‘S’) We deployed a rating system for each of the above evaluation criteria that is set out in the table opposite to the extent feasible based on the project reviews, field trip case studies and an overall CAPTAC-DR level assessment. Achievements of targets within these criteria have been rated as Excellent (‘E’); Good (‘G’); Modest (‘M’); Poor (‘P’); or Not Demonstrated (‘ND’). In addition, we have given numeric scores to allow for gradation within each rating criterion. We had some reservations about assigning weightings to the scores for individual DAC criteria and arriving at a consolidated score. We have outlined these in our main report. ICDGP and Evaluation Sub-Committee feedback was that a weighted scoring approach is needed to maintain comparability of results with previous evaluations. We have therefore applied the weights recommended in the ToRs (see Appendix I, Annex B in Volume I), in separate scoring tables for comparability purposes. In addition to the scoring methodology, we suggested some minor changes to the KEQs to improve their consistency and technical robustness. We also suggested slightly modified KEQs for individual TA interventions and that for the CAPTAC-DR program. These suggestions were accepted by ICDGP and the Evaluation Sub-Committee, and have been adopted for the Evaluation.

1 Impact was assessed, but not scored, as provided for in the Evaluation ToRs.

CAPTAC-DR Evaluation Rating Criteria

Rating Achievement Base Score

Range

Excellent (‘E’) All or substantially all objectives met

4 3.6-4.0

Good (‘G’) Majority of objectives met

3 2.5-3.5

Modest (‘M’) Few/minority of objectives met

2 1.5-2.4

Poor (‘P’) Very few objectives met

1 1.0-1.4

Not Demonstrated (‘ND’)

Criteria cannot be assessed

ND


5 | P a g e

1.2 DESKTOP REVIEWS We reviewed all available documentation at CAPTAC-DR for TA projects selected for desktop reviews, including Briefing Papers (mission ToRs); TA Reports; and Back-to-Office Reports (‘BTOs’) for all missions undertaken. We also discussed the project with the relevant RAs, where they were still in place, or received feedback from previous RAs who had undertaken the project. In some cases, feedback was also obtained from STXs who had been involved. In a number of cases we obtained feedback from the TA beneficiary, especially relating to the implementation status of TA recommendations. Finally, we referred to relevant survey feedback. Information reviewed was then used to respond to the KEQs to rate the projects in accordance with the DAC criteria. 1.3 CASE STUDIES We worked with a results chain approach to provide the logical underpinning for the logframes used in the case studies. For the three case studies selected, we constructed ‘projects’ out of a series of missions, some extending for as long as seven years. We then developed the sector results chain to place the project into context, and assess the degree to which the project was appropriately sequenced, and the wider reform needs to achieve higher level outcomes were being addressed, and the associated risks mitigated. Ex post logframes were designed from the available data from CAPTAC-DR and wider research, and output and outcome indicators developed. Based on review of documents and interviews with the responsible RA, as well as the in-country meetings with TA beneficiary stakeholders and other institutions indirectly affected by the TA, we completed the KEQs in respect to each of the case studies and arrived at our evaluation ratings and scores for each of them.


6 | P a g e

CASE STUDIES


7 | P a g e

2. COSTA RICA: CUSTOMS ADMINISTRATION 2.1 INTRODUCTION AND SUMMARY OF RESULTS

The above project was selected as a Case Study, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s Inception Report. Since its inception, the Center has supported customs administrations throughout the region. This work has been designed within the framework of a regional plan for tax and customs administrations adopted in FY2010. This technical area had already been singled out as a key topic across member countries. The objectives for TA in customs administration relate to the need to (i) eliminate illegal activity; (ii) increase government revenue; (iii) improve monitoring capabilities; and (iv) facilitate regional and international trade. Although CAPTAC-DR’s technical assistance program for customs administration

is not yet complete, the project under review already shows remarkable results. An impressive range of outputs has been produced, and there is strong evidence of progress towards achieving the project’s short and medium term outcomes. The Costa Rican Government, in turn, has assumed ownership of the technical assistance and has pushed forward the customs reform agenda. These characteristics are reflected in the high score assigned to this initiative. This study takes stock of the work already carried out. In addition, our evaluation presents a number of recommendations that seek to consolidate the gains already achieved, and to make the most of the remaining years of CAPTAC-DR technical support to Costa Rica. 2.2 INFORMATION REVIEWED The evaluation was carried out primarily based on: Document and data analysis: We reviewed and analyzed the materials at the disposal of the Costa Rican Customs

Service (‘CCS’) along with a series of CAPTAC-DR reports (briefing papers, aide memoires, etc.) Questionnaires and surveys: We produced a structured questionnaire, which was sent to the CCS and the CAPTAC-

DR’s RA. In addition, this review benefits from a survey done within the context of the Evaluation. Interviews: We conducted a number of semi-structured interviews with CCS officials; as well as a series of

discussions with Costa Rican authorities focused on the TA project’s appropriateness and responsiveness. Meetings: We held a series of meetings with the CCS staff. Minutes of these meetings are available.


8 | P a g e

2.3 OVERVIEW By definition, small economies suffer from a limited domestic market size. In order to prosper, they must therefore sell their products and services abroad. In fact, most successful small countries exhibit a strong external trade sector. Belgium, Hong Kong, Mauritius, Singapore, and Switzerland inter alia provide examples of how openness to trade can help create prosperity for small, relatively resource-poor, national economies (see chart below). Costa Rica, with a population of less than five million, is also considered a successful small economy. Its indicators such as per capita GDP and life expectancy, human development, and others consistently exceed those of its Central American neighbors (i.e. El Salvador, Guatemala, Nicaragua, & Honduras). Costa Rica’s relative trade openness and stability partially explain these achievements. In fact, its merchandise exports, when measured as a percentage of GDP, show levels similar to those of successful small countries such as Mauritius and Switzerland. On another measure, Costa Rica’s ports handle over a million containers annually2, roughly of the same order of magnitude as much larger countries such as Morocco, Peru, Poland or the Ukraine.

Despite these achievements, there remains room for improvement. As the chart opposite shows, Costa Rica trails all the other countries selected with respect to per capita GDP (even at purchasing power parity). Its life expectancy at birth is 79 years, but still lags behind all other countries in the list, except for the case of Mauritius.3 One factor negatively affecting economic development

in the country is its relatively business unfriendly environment. In fact, the country ranks 110 out of 185 in the World Bank/IFC “Ease of doing business” tables, though it has been improving in recent years. The relative ease of trading across borders constitutes one of the key criteria in the above-mentioned ranking. Having an efficient customs administration influences this criterion. The customs service is also a key contributor to government revenues because of its role in collecting resources, controlling illegal exchanges, and facilitating trading activities. Tariff revenues provide vital income to the treasury, particularly in developing countries. An IMF report points out that customs revenue averages approximately five percentage points of GDP across upper middle-income countries.4 In the case of Costa Rica, excise and customs revenue reached approximately US$1.6 billion in 2011.5 Countries with weak customs services may become susceptible to narcotic cartels; counterfeit traders; poachers of protected species; and smugglers of archeological treasures. 2 20-foot equivalent units. 3 Source for all data: World Bank and IMF. 4 IMF/FAD. Revenue Mobilization in Developing Countries. Approved by Carlo Cottarelli on March 8, 2011. 5 IMF. Costa Rica: Third and Final Review Under the Stand-By Arrangement—Staff Report. IMF Country Report No. 10/162. June 2010. Own

exchange rate calculations.


9 | P a g e

Since its inception, CAPTAC-DR has supported the strengthening of customs administration across the region through the provision of TA. 2.4 THE RESULTS CHAIN FOR CUSTOMS ADMINISTRATION IN COSTA RICA A sector level Results Chain for customs in Costa Rica would have been ideally developed ex ante, as a project planning tool. An illustrative one is provided below. It is intended to highlight the key technical assistance/reform needs beyond the project if the high level outcomes are to be achieved i.e. what other steps need to be taken supported by other TA providers or by the customs administration itself.

Analysis of the Results Chain suggests that work was undertaken in a broadly logical sequence (diagnostic, followed by strategy/action plan, necessary legislation/regulation and then capacity building). The section below provides an overview of the key components of this Results Chain.


10 | P a g e

Overall Objective and Impact The objective or high level outcomes to which the project contributes are: (i) increasing customs revenues; (ii) improving monitoring & law enforcement; and (iii) facilitating trade). CAPTAC-DR’s project on its own may not be able to deliver this objective, the achievement of which may be dependent on support and action from other sources. The majority of the short and medium-outcomes have been achieved. A more detailed presentation of the project’s outcomes is provided later in the report. Diagnosis Previous FAD missions provided the underlying diagnostics for the project. Follow-up CAPTAC-DR needs assessment missions completed this initial diagnostic. Finally, CCS undertook a self-evaluation suggested by CAPTAC-DR, which identified all critical gaps and the major areas requiring improvement. Strategy On the basis of the initial assessments above, action plans were developed at two levels, regional and national. The regional tax and customs administration work plan provided the overall strategic framework. This plan was developed in support of regional customs integration efforts. Concurrently, TA from the Center led CCS to adopt a national action plan that lists fifty-three areas for improvement. Milestones have been set for the short, medium and the long-term. This plan has been fitted into a time-bound matrix, which also serves as the main monitoring mechanism for its implementation. This matrix complements a range of indicators specifically developed for CCS, which allows close monitoring of the organization’s performance. Legislation and Regulation As part of this work, the applicable legal framework was reviewed. In particular, this exercise covered (i) the customs procedures manual; (ii) the Costa Rican General Customs Law; (iii) the various regulations appended to the latter; (iii) the Unified Central American Customs Code III (‘CAUCA III’), the third version of which has been ratified by the Costa Rican Parliament; and (iv) CAUCA IV and its accompanying RECAUCA IV rules which are still pending congressional approval. Activities, Outputs and Outcomes The majority of the short and medium-outcomes have been achieved. A more detailed presentation of the project’s outcomes is provided later in the report.


11 | P a g e

Donor Coordination The project was relatively self-contained with CAPTAC-DR being the principal provider of TA to CCS. Nonetheless, a number of other agencies are also active in this area in CAPTAC-DR member countries. The Center has made substantial efforts to coordinate a common development agenda, on the basis of regional and national priorities. In particular, IMF/FAD convened two coordination meetings, attended by the IADB, World Customs Organization (‘WCO’), World Bank, USAID, as well as the Central American Economic Cooperation Secretariat (‘SIECA’). Furthermore, a number of joint training events have taken place, particularly together with IADB and WCO. The Center has worked with SIECA in setting regional agendas. Pre-conditions / Risks The Results Chain assumes certain preconditions need to be met and also attempts to cover the risks that could be faced in achieving the high level outcomes or objectives. We have listed by way of illustration some of the types of risk and pre-conditions: political, economic management, external shocks, sound infrastructure, good governance, adequate legal framework and sufficient resources for the customs authorities. Finally, the achievement of higher level outcomes is also dependent on the outcome of other national initiatives, such as (i) the regional integration negotiations; (ii) opening national IT systems to individual agencies; (iv) the overall policy frameworks dealing with both trade and fiscal issues; and (v) national policies dealing with illegal trade. 2.5 CAPTAC-DR TECHNICAL ASSISTANCE Background Since its inception in 2009, CAPTAC-DR has sought to strengthen capacity of the member states’ customs administrations. Technical assistance in this area has been coordinated by a Resident Advisor (‘RA’) working full-time and managing four STXs. The objective is to support the adoption of comprehensive customs control systems (expanding the use of risk management techniques, post-clearance audits, and review of procedures) to facilitate trade. Supporting the regional integration process was also a key objective. CAPTAC-DR’s programme for customs administration strengthening grew out of strategic and operational planning exercises across the region. This work included both enhanced diagnoses and the identification of performance indicators. The Center’s work also involved formulating pilot plans for implementing new processes seeking to raise risk perceptions, boost revenues and improve service levels. Finally, CAPTAC-DR fostered the creation of a Regional Customs and Tax Administration Working Group (GAT). This group develops standards for both processes and services in seeking to fill gaps in operational capacities throughout the region.


12 | P a g e

The region is working towards establishing a Central American Customs Union. Within this framework, regional customs administrations are supporting integration efforts, while keeping appropriate levels of control and ensuring sustainability. CAPTAC-DR’s TA initiatives at the regional level have focused on supporting regional or bi-national technical working groups on trade facilitation and on the free movement of merchandise. Technical assistance has been provided to GAT on the formulation of a work plan, through is Council of Treasury / Finance Ministers (COSEFIN). CAPTAC-DR submitted a draft medium-term strategic plan for customs (2011-2015) for COSEFIN’s approval. Furthermore, CAPTAC-DR mobilized consensus in order to find a mechanism to eliminate double duties merchandise imported from the EU. Also at the regional level, CAPTAC-DR has developed a number of methodologies and technical papers. For example, the Center has identified a set of indicators to allow customs administrations across the region to measure their performance, against international best practice and in the context of their own reform process. Finally, CAPTAC-DR has prepared and circulated a set of recommendations to establish a common customs transit system for the region. Key strategic focus areas for reform and TA have been identified, notably: (i) customs management and planning, (ii) rules and regulations; (iii) customs control; (iv) trade facilitation; and (v) IT & communication technologies. These areas have also been addressed through TA at the national level. The Center has supported its member countries in implementing enhanced planning and management techniques in areas where existing regulations and practices needed adjustment. Gaps and opportunities have been identified. The Center’s TA has been aimed at helping member countries to find the right balance between trade facilitation and customs control, within the context of resource mobilization efforts and compliance with various regulations (e.g. non-tariff or safety rules). At the national level, the CAPTAC-DR project has sought to improve risk management techniques at the various stages of the import/export cycle: namely (i) on transit, (ii) while at the customs point, as well as (iii) post clearance inspections. Overall surveillance has also been stepped up for cargo flows before they reach the customs point, in order to minimize smuggling. These efforts are also aimed at achieving more transparency while simplifying formalities. Other support has included (i) revisiting measures to foster collaboration among government bodies (mainly customs and internal revenue services); (ii) reviewing overall customs procedures; and (iii) implementing audit indicators. Improved IT systems constitute a critical resource to implement this initiative. Support to the Customs Administration of Costa Rica started in earnest in FY2011. Here too, the focus was on applying international best practices. The overall goal has been to strengthen customs control using risk-based methods. Ex ante, immediate, and ex post controls have been particular areas of focus. CAPTAC-DR TA has also sought to increase revenues without negatively affecting legal trade. It also strived to improve surveillance of priority risks, such as smuggling and under-invoicing of sensitive goods. CCS has made important efforts to implement the numerous recommendations received from CAPTAC-DR. Even though TA is still under way, preliminary analysis reveals a number of significant improvements and impressive results. First, CCS has developed a risk-management audit improvement plan, with objectives identified for the short, medium and long-term. Secondly, the pilot methodology mentioned above has been implemented at the Santa Maria customs point, involving fewer, but more thorough inspections. Another example has been a comprehensive proposal to overhaul CCS’ IT system, in order to align it more closely with the new customs procedures manual.


13 | P a g e

Activities CAPTAC-DR’s customs administration TA has focused on the operational aspects/activities listed below: Adapting, formulating and initiating implementation of performance measurement indicators; Setting up protocols for risk assessment and management; Improving customs value measurements of imported merchandise and dealing with under-declaration; Designing and executing a pilot integrated initiative to improve joint controls between central customs office and

border units seeking to improve average receipts per customs declaration; Carrying out an importer segmentation exercise seeking to improve risk profiling; Improving operational procedures, particularly during inspections carried out at Costa Rica’s ports and airports; Identifying specific areas for improvement, seeking to leverage customs capacity through IT systems in order to

improve both control and trade facilitation in the country’s ports, airports, and land borders; and Standardizing procedures and actions applicable at the beginning and the conclusion of the customs process,

improving information flows between both types of action. Outputs The following list illustrates relevant project outputs: Informed ex-post audit strategies based on risk management; Strengthened control of the flow of air and ocean freight in the phase prior to the destination customs; Performance indicators to monitor the performance of primary and ex-post clearance controls; Opportunities for improvement identified (based on conclusion of the self-assessment process); Action plan for short, medium, and long-term improvements in risk management, control & audit systems at all the

stages of the customs process; Mechanisms to eliminate double payments of customs duties; Improvement plan for the short, medium, & long-term for risk management and audits; Methodology for the overall management of the control process in a pilot at Santa Maria customs point (quality

parameters of selectivity, fewer inspections, and improving the technical quality of inspections); Preliminary proposals to adapt norms and functional specifications for the customs IT system in line with the Customs

Procedures Manual (preliminary version); and Customs Procedures Manual (revised).


14 | P a g e

Outcomes We have constructed the project’s principal expected outcomes as summarized in the table below:6

Outcomes Short-term

(up to 1 year) Medium-term

(2-3 years) Long-term

(4-10 years) Approval and adoption of customs indicators by CCS.

Improved CCS performance as measured by the selectivity performance index).

Customs and excise receipts grow faster than imports.

Improvement action plan for risk mgm’t, control & audits adopted.

Increased voluntary compliance (higher declared values).

Customs revenue exceeds 7% of GDP.

Improved performance at Santa Maria pilot project.

Increased anomaly detection rates (by number and by volume).

Trade transactions facilitated.

Adoption of Revised Customs Procedures Manual approved by management.

Manual fully implemented. Service deemed efficient & user friendly.

Illegal activity (e.g. under-reporting & smuggling) decreases.

Acceptance of Preliminary plan to adapt specs of IT system to customs procedure manual.

TICA system becomes more useful adapted.

More generally some of the relevant indicators to monitor might include: Established key baseline statistics (including key performance index); Percentage increase of customs revenue collection (vs percentage growth of import volume and value); Customs revenues as percentage of GDP; Opinions of informed experts about the prevalence of smuggling and other illegal practices. Analysis of mirror (export) statistics; Percentage increase on customs-related cost (budget); Percentage increase on findings during examinations of goods and in Post Clearance Audits; Percentage increase on improving the performance of the selectivity; Percentage increase on penalties collection; Percentage increase on voluntary compliance; Submission error rates from Agents/Brokers; Detection Rates (e.g. scanners); Average values of customs declarations; Number of anomalies found in customs declarations; Level of satisfaction of customs staff and of customs users with the new manual; Expert opinion on the compliance/adaptation of the TICA system with customs rules, regulations & manuals; Evolution of the selectivity performance index; Indicators adopted by CCS management on the basis of CAPTAC-DR advice not covered in the above list; as well as Indicators from the list above, applied to the specific case of the Santa María customs post. The IMF’s new RA-FIT model would additionally support monitoring of customs performance.

6 These were not produced at the outset of the project. They have been prepared ex post by the Evaluators to provide benchmarks for assessing outcomes.


15 | P a g e

Achievements Most of the project’s short-term outcomes have been attained and good progress has been made on many of the medium term planned outcomes. CCS has already adopted a list of 11 customs monitoring indicators, which are linked to CCS’s reform action plan. Use of these indicators is already under way. Furthermore, the key elements of the revised operational manual have been produced, along with the specifications to adapt the organization’s TICA information system to this new manual. Revised customs procedures are already being implemented, particularly for the pilot initiative being executed at the Santa Maria customs post. As a result, Costa Rica already boasts: A 5.3 percent increase of its audit selectivity performance index between 2010 and 2011; During the same period, at the Santa Maria site, this index rose by 13 percent; Over 130 percent improvement in the 2011 results of ex-post audits, particularly in customs value audits; and A sharp increase in customs revenues of 23.5 per cent in 2011 (imports grew by 10.9 percent during the same

period). Remaining Challenges Despite the impressive progress already made by CCS, CAPTAC-DR’s TA still has some ways to go. Firstly, CCS needs to finalize the new operational manual. A number of problems or issues can be expected to emerge once the process of implementing the new manual starts. These will require immediate adjustment and will constitute a source of feedback in order to fine tune the manual. A similar process is expected to take place with the TICA system, once its revised version is available on-line, to support the new manual. These iterative processes are expected to last several months and are essential in order to complete the work already started. Furthermore, adjustments are also required at the Santa Maria customs post. Once enough experience has been gained, the new customs processes should also be replicated in additional border posts. This extension of coverage will in turn require the adaptation of the procedures to fit the specific characteristics of each site (e.g. land border posts, airports, maritime ports, etc.)

Additional work is also necessary to continue improving control over air and ocean freight prior to arrival in Costa Rica, as well as risk-based audits in general. The action plan has a number of outstanding medium and long-term actions that need to be executed in order to attain optimal results. Finally, the issue of double customs levies on EU source merchandise is yet to be resolved.


16 | P a g e

2.6 PRINCIPAL CONCLUSIONS The Annex presents our detailed responses to the KEQs and the basis of arriving at our ratings for the various DAC criteria. In summary: We rate Relevance of the project as Excellent (score 3.8), demonstrating its close alignment with CAPTAC-DR, IMF HQ, regional, and Costa Rican strategies and plans. There has been strong ownership demonstrated by CCS and broad coordination with development partners. With respect to Effectiveness, our rating at the Outcomes level is also Excellent (3.6) reflecting the substantial achievements of outcomes at a very early stage. The high quality of advice delivered is reflected in a rating of Good (3.5) for Outputs. Efficiency is again rated as Good (3.5), reflecting a high ratio of outputs to inputs. Outputs and a number of outcomes have been attained quickly. Sustainability is rated on the upper range of Good (3.3), as the majority of the advice delivered so far is likely to be retained within CCS. As project implementation continues, prospects for sustainability are likely to improve even further. 2.7 KEY FINDINGS AND RECOMMENDATIONS Findings

1. There is a risk that the CAPTAC-DR recommendations may not be fully implemented because, at the moment, the

current government’s priority appears to focus on fiscal reform. 2. The centralization of the IT in the Ministry of Finance hinders the implementation of improvements to the customs IT

system, notably the use of selectivity modules to implement probabilistic risk assessments, deterministic filters, and neutral networks.

3. No mechanism has yet been set up to eliminate double tariff collection despite CAPTAC-DR’s best efforts. Costa Rica and Guatemala have different points of view regarding the distribution modalities of ensuring customs duties returns.

4. 80 percent of all consignments have been introduced through the customs warehousing and temporary deposit regime, as it is often perceived as more convenient. This is not an ideal situation.

5. The assessments made in the Customs Procedures Manual are valuable. It identifies, for example, weak points in the customs clearance procedure in order to make corrections and improve adaptation to the TICA customs system.

6. Ideally, goods for consumption should be cleared at the point of arrival, without interim transfer to warehouses. This goal requires enhanced logistics, appropriate customs facilities and sufficient budgetary resources at CCS.

7. Two important proposals are included in the Master Plan notably (i) for re-engineering the customs clearance procedures for goods at point of arrival and (ii) for making land border points more fluid.

8. The use of non-intrusive technologies (X-ray or gamma) constitutes an effective tool for examining both large volumes and a wide variety of goods.

9. Training will still be required for customs officers to interpret the images derived from 8 above effectively.


17 | P a g e

Recommendations

1. Recommendations for operational improvements would benefit from more detail, ideally containing more diagrams. 2. CAPATC-DR should insist on implementing the mechanism for eliminating double customs duty payment, under the

Auspices of COESFIN. 3. Training is needed on (i) origin verification; (ii) review of cases regarding under-invoicing; (iii) PCA audits; and (iv)

valuation based on product codes. 4. TA should include hands-on learning based on real cases. Mentoring and coaching would be particularly useful in the

areas of Post Clearance Audits and handling fraudulent activity. 5. Development of in-house training should be a priority. 6. Even though the Customs Procedures Manual was developed with professionalism, we suggest drafting new

modalities for “Joint procedures” (with the various intervening ministers, e.g. education, agriculture & livestock, health, tax police, etc.). Implementation of common modalities would speed up clearance process.

7. In the long-term, radio frequency (‘RF’) technology at seaports should also be promoted. This goal requires building broad partnerships while also interfacing IT systems.

8. While impressive results have been achieved, CCS should not relent in improving risk management processes. 9. Software tools are needed, as well as technical assistance in the following areas: (i) cargo manifest analysis; (ii)

selection of companies for audit; (iii) handling of statistical packages and modeling; and (iv) risk profiling. 10. There is scope for further efficiency gains and regional capacity building by more actively pursuing synergies with

other projects at the regional level.


18 | P a g e

2.8 ANNEX 2.8.1 SUGGESTED LOGICAL FR AMEW ORK A logical framework analysis / Results-Based Management (‘RBM’) has not been applied to this particular initiative, though many of its elements have been implicit. In line with other sections of this review, the table in the following page proposes a sketch of the basic suggested contents:

Inputs / Activities (CAPTAC-DR Only)

Outputs (*) (Overall project)

Outcomes (Overall project)

Short-term (up to 1 year)

Medium-term (2-3 years)

Long-term (4-10 years)

Support to Customs Administra-tion in Costa Rica

I/P: LTX / STX Backstopping & consulting time; travel costs; overheads.

Activities: Missions; backstopping; workshop; mentoring; training.

a) Diagnostic & recommendations.

b) Improvement plan for risk mgm’t and audits.

c) Staff trained.

d) Methodology for control process mg’t in Santa Maria pilot.

e) Revised Customs Procedures Manual.

f) Proposal to adapt TICA system to Manual.

g) Ex-post audit strategies based on risk mgm’t.

h) Strengthened control air and ocean freight prior to arrival at customs.

i) Performance indicators clearance controls.

j) Mechanisms to elimina-te double payments.

11 indicators adopted.

Performance at pilot improves.

Plan for risk mgm’t, control & audits adopted.

Procedure Man-ual approved.

Plan to adapt TICA to manual approved.

Improved CCS performance.

Increased voluntary compliance.

Increased anomaly detection rates.

Service deemed efficient & user friendly.

TICA becomes more useful.

Customs and excise receipts grow faster than imports.

Customs revenue exceeds 7% of GDP.


Illegal activity (e.g. under-reporting & smuggling) decreases.

Indicators: Percentage increase on: customs-related

cost; findings at examinations; penalties collection; and voluntary compliance.

Established key baseline statistics (incl. key performance index).

Average value of declaration; number of anomalies found.

Satisfaction of customs staff and users. Expert opinion on the adaptation of the

TICA with customs rules & manuals. Indicators above, applied to Santa Maria.

Indicators: Customs revenue

(versus imports). Customs income

as % of GDP. % businessmen

satisfied. Expert opinion

on prevalence of smuggling & illegal practices.

Sources of verifi-cation

Reports (aide memories, BTOs).

Workshop reports.

CAPTAC-DR reports. Physical inspection

(e.g. at ports, or at Santa Maria point).

Customs websites.

Physical inspection. Monitoring and other CAPTAC-DR reports. IT & work systems assessment. Surveys of users, of businessmen, and of CCS staff. National accounts, trade statistics, mirror statistics. Interview with experts (& staff) and CCS records.

Major risks & assump-tions

Local conditions allow for the activities to take place as planned.

CCS is willing to implement project recommendations.

Sufficient resources available for CCS.

Political stability & support. Sensible macroeconomic policy. Staff with the appropriate qualifications trained and retained. IT systems in place and functioning correctly. Effective legal systems and overall institutional infrastructure. Sufficient resources available. Absence of substantial external shocks. Coherence with national initiatives and policies.


19 | P a g e

2.8.2 EVAL UATION K EQS AND R ATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities

Consistency of activity with Program Document, CAPTAC-DR strategy, and approved work plan.

Consistent.

Consistency of intervention(s) with national/regional priorities:

o Extent to which activity is linked to and supports national/regional strategy (if one exists – e.g., PRSP);

Consistent (e.g. the regional Tax and Customs Administration Work plan, adopted in FY2010, strategic medium-term plan for customs (2011-2015); and National customs progress monitoring matrix with 53 recommendations.)

o Extent to which activity is linked to and supports sector/topical reform strategy (if one exists);

o Extent to which strong country/institutional ownership of activity has been demonstrated.

Recommendations have been taken seriously and implemented rather swiftly

Whether the activity is appropriately sequenced, given the completed/outstanding reform needs for that sector/institution:

An analysis of the project’s results chain (see above) suggests that project sequencing was carried out correctly. The impressive results in achieving desired outputs and outcomes at relatively early stage indicate high absorptive capacity among CCS staff. In fact, according to the responses to the questionnaire made by CAPTAC-DR, the capacity of the customs administration in Costa Rica may be among the highest in the region.

o E.g., appropriateness of project expected outcomes given country/institutional absorptive and implementation capacity.

Consistency with IMF Headquarter/other activities Whether activity is appropriately focused in terms of subject area, taking into account the IMF’s expertise, integration with HQ and other Fund activity (e.g. other training and TA delivered including through the topical trust funds).

The activity is in line with IMF’s core competences and is appropriately focused. FAD missions have also contributed to the project.

Coordination with Development Partners Whether the intervention has been effectively coordinated with and complements the work of other development partners.

The implementation of the regional customs plan is also coordinated with IADB and the WCO, particularly on single windows and border management. The Center annually organizes two seminars in coordination with the IDB and WCO.

Context of donor/TA landscape of CAPTAC-DR TA Why did the TA recipient choose the CAPTAC-DR as the TA provider as opposed to other donors?

The Center enjoys a solid reputation and buy-in from its counterparts.

How do workshop participants compare those provided by CAPTAC-DR with workshops from other donors?

In cooperation with IADB & WCO, the seminars delivered in April and June 2012 focused on risk management and its organizational impact: human resources, ethics & transparency, and risk management applied to Post Clearance Audit. Participants rated these seminars respectively at 4.85 and 4.84 out of 5.0.

Score for Relevance Excellent 3.8 EFFECTIVENESS Impact Intervention’s expected Impact achieved or likely to be achieved (to the extent defined in the activity’s logframe).

Highly likely.

Outcomes Intervention’s Outcomes achieved or likely to be achieved (as defined in its logframe): Short, medium and long term Outcomes.

Most recommendations on the short and medium-term have been implemented on a timely basis resulting in (i) an increase of the revenue collection, (ii) overall improvements in the performance index of selectivity and (iii) increasing the results in Post clearance Audit especially in customs.

Significance of CAPTAC-DR’s contribution/likely contribution to developing core economic functions and institution building in the country/region, through the activity.

This project is part of a series of customs-related initiatives being carried out throughout the region.

Score for Outcomes Excellent 3.6


20 | P a g e

Key Evaluation Questions Comments Outputs Intervention’s Outputs achieved or likely to be achieved (as defined in its logframe), including: o Timeliness of reports/workshop presentations; o Quality of reports/workshop presentations; o Appropriateness/applicability of advice, given

existing capacity/constraints; o Whether Outputs are likely to produce intended

Outcomes.

The TA reports and recommendations are comprehensive enough in scope to produce the desired outcomes due to the fact that most of them were accepted and implemented by the customs administration. A large number of outputs have already been produced.

Score for Outputs Excellent 3.6 EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; Yes. Timeliness in executing the activity; CAPTAC-DR has evaluated the request that gives more priority to the country

with a strategic plan that contributes to strengthen the customs administration and available budget. If the requesting country meets with the above requirements, the TA is carried out immediately.

Appropriateness/effectiveness of IMF’s internal management of the activity;

Appears good based on preliminary results. Delivery has been efficient but risks being undermined if the MF and CCS structure does not change in order to eliminate the centralization of IT systems and dysfunction of CCS’ directorates.

Appropriateness of selection of counterpart/workshop participants (as relevant);

Appropriate.

Quality and timeliness of management and backstopping in relation to activity;

Communication between the field and CAPTAC-DR seems good and the RA makes regular visits at high levels of the CCS to keep project on track.

The efficiency of planning and executing the TA/training delivery.

Briefing papers were generally focused and relevant.

Efficient use of resources (human and financial) and attention to cost effectiveness

As evident from backstopping correspondences/comments and their influence on quality of final TA report.

Appropriateness of staffing composition for the intervention;

Staffing is considered to be of high technical caliber.

Appropriateness of work allocation between HQ and CAPTAC-DR in relation to activity;

Difficult to assess based on available information.

Whether expenditures have been in line with activity budget and evidence of analysis of variances;

No evidence of recording project-specific costs or variance analysis.

Whether activity, given its results, has been cost-effective compared to other TA delivery modes

Given the extent of the outputs and outcomes produced, the inputs have been deployed in an efficient manner. However, no detailed cost information relating to the project has been available.

Whether opportunities for efficiency gains during the intervention have been considered and explored.

No evidence of this having taken place.

Monitoring and reporting Evidence of effective use of self-evaluation (i.e. monitoring) and reporting to improve the efficiency and effectiveness of activity;

A comprehensive monitoring matrix listing all key recommendations was developed and used appropriately.

Effectiveness in making use of TAIMS to monitor and manage project/workshop;

No information available.

Effectiveness/progress in use of RBM and its usage in managing the intervention.

RBM has recently been implemented. No evidence was reviewed of its intensive use as a project management tool.

Incorporation of lessons learnt in project design and implementation.

No evidence was reviewed on formal extraction/usage of lessons learnt.

Score for Efficiency Good 3.5 SUSTAINABILITY Sustainability of TA activity Outcomes from intervention will last beyond/continue after completion of TA/training.

At the end of the project the CCS will be able to continue strengthening and modernizing the customs administration. First, because of its high absorptive capacity as shown during the course of the project; secondly, in light of the


21 | P a g e

Key Evaluation Questions Comments appropriateness of the project.

For TA/training designed to deliver Sustainability, local institutions/ capacities have been strengthened to sustain results beyond the life of the intervention: o E.g., absorptive capacity improved and developed to

sustainable level; o E.g., in-house training capacity built, where relevant.

Absorptive capacity at CCS has been high. Strengthening in-house training should be a priority for the next stages of the project.

For interventions designed to deliver sustainability, financial sustainability has been achieved (i.e., recipients will be implement the advice from own/other financial resources in a sustainable manner).

The outlook is positive.

Effectiveness in maintenance and use of institutional memory relating to the activity

No pertinent information was evident.

Contribution to building sustainable regional TA, implementation capability.

The activity has built good capacity in Costa Rica. Ideally local expertise could also be deployed regionally. The regional strategy should also be leveraged to allow for more regional synergies

Score for Sustainability Good 3.3 Rating Based on Previous Evaluation Methodology

DAC

criteria Sub-criteria Score Weight

W’d score

Weight Total score

Rating

Relevance Consistency with Program Document and Government Priorities

3.80 60% 2.28

Consistency with IMF Headquarter/other activities

3.80 20% 0.76

Co-ordination with Development Partners

3.60 20% 0.72

3.76 32% 1.2 Excellent Effectiveness Impact - Outcomes: TA 3.6 43% 1.55 Outcomes: Regional capacity building NA NA NA Outputs 3.6 57% 2.05 3.60 28% 1.00 Excellent Efficiency Process and implementation efficiency 3.6 40% 1.44 Efficient use of resources (human & fin.)

& attention to cost effectiveness 3.6 40% 1.44

Monitoring and reporting 3.3 20% 0.66 3.54 22% 0.78 Excellent Sustaina-bility

Sustainability of RTAC/TA activity 3.3 75% 2.48

Contribution to building sustainable regional TA, implementation capability

3.4 25% 0.85

3.33 18% 0.60 Good

TOTAL 3.58 Excellent


22 | P a g e

3. REGIONAL: CROSS-BORDER CONSOLIDATED SUPERVISION 3.1 INTRODUCTION AND SUMMARY OF RESULTS

The above project was selected as a Case Study, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s Inception Report. Consolidated Supervision (‘CS’) has gained particular relevance given the relative importance of international financial conglomerates in CAPTAC-DR member countries. Perhaps for this reason, all banking supervisors involved show strong ownership and active contribution to the TA project. The TA has been technically solid, helped by excellent external STXs. The project so far has helped produce an impressive range of outputs within a short period of

time. The key challenges ahead center on the broad application of the methodologies developed so far as well as with the enactment of accompanying legislation. Implementation of the cross border CS project, which is ongoing, requires additional time in order to consolidate its achievements. Progress so far has been strong. 3.2 INFORMATION REVIEWED The evaluation was carried out based on: Review of project-related documentation available at CAPTAC-DR (Briefing Papers, Mission Reports, and BTOs); Interviews of CAPTAC-DR’s Center Coordinator and the RA for the project. Review of outputs and presentations related to the project; Interviews with heads and staff at the Costa Rican and Guatemalan bank Superintendencies; Interviews with the President of the Central Bank of Costa Rica and Governor of the Central Bank of Guatemala; Review of background information on CS in the region, including Association of Supervisors of Banks of the Americas,

the website of the Commission of Central American Supervisors of Banks, Insurance and Other Financial Institutions and an IMF working paper on “Development of Financial Supervision and the use of Macro-prudential Measures in Central America, Panama and Dominican Republic” December 2011;

Interviews with the STX who was primarily responsible for project implementation; Meetings with IMF Staff; and Survey feedback from the region relating to the project.


23 | P a g e

3.3 OVERVIEW The CAPTAC-DR countries, particularly the Central American ones, have a long history of interest in regional integration. However, this goal of integration is complicated by a number of factors, including the countries’ economic dissimilarities in a number of areas. In particular, Panama stands out as a regional financial centre that uses the US Dollar as currency, with rapid growth recently fuelled by an expansion of its canal and with a per capita GDI of almost US$8,000 measured at purchase power parity. But, even excluding Panama, per capita incomes differ substantially as do urbanization, literacy and financial systems. Public sector capacity levels also differ across the countries, reflecting not only differences in education but variations in the tenures of public sector management and ministers. Reforms in the countries in the areas of inflation, fiscal-consolidation and exchange rate policy have improved the standard measures of their macroeconomic stability. This economic heterogeneity is illustrated in the table below.

CAPTAC-DR Member Country Indicators

Country CRI DOM SLV GTM HND NIC PAN

GDI per capita PPP (US$) 7,760 5,240 3,480 2,870 1,970 1,170 7,910 GDP Growth 2011 (% est.) 4.2 4.5 1.5 3.9 3.4 4.7 10.6 Inflation CPI 2011 (%) 4.9 8.5 5.1 6.2 6.8 8.1 5.9 Dom. Credit Banks/GDP (%) 53.1 38.9 44.6 39.2 53.3 58.8 91.6 Fiscal Def % GDP ‘10. 2.8 3.5 (09) 2.7 3.3 3.1 1.0 3.4 Gov. Debt %GDP ‘10 42 42 49.5 24.4 n.a. n.a. n.a. Dollarization (%) 50 30 100 32 32 90 100 Inter. Res. US$bn 2011 4.8 4.1 2.5 6.2 2.8 1.9 2.3 Urban population (%) 65 70 65 50 52 58 75 Literacy (2005-2009, %) 96 88 84 74 84 78 94 Source: World Bank; “Dollarization Declines in Latin America” Finance & Development, Mar. 2010

Central America, Panama and the Dominican Republic suffered sharp GDP declines from the 2008 global financial crisis but are recovering reasonably well, especially given the slow recovery of the United States of America (‘USA’). The countries’ GDP declines reflected their links to OECD countries, especially the USA, through declines in their exports, tourism, remittances, and foreign investment. These factors also had a negative impact on government revenues. However, they avoided major banking crises, reflecting their banking sectors’ lack of toxic assets, strong capital, adjustments toward liquid assets, and, in the case of Guatemala, quick responses to problems in a major bank. All CAPTAC-DR members, except Panama, received IMF stand-by arrangements of various types as support after the crisis; Costa Rica’s and Guatemala’s programs were precautionary and were never drawn upon. Post-crisis, Panama has grown very rapidly, reflecting its investments in widening its canal, as well as its growth as a regional banking center. Growth in the other countries has recovered slightly but has not returned to pre-crisis levels. Variations in the countries’ recoveries reflect not only differences in their macroeconomic performance, but also fundamental issues such as infrastructure, the climate for doing business, security, etc.


24 | P a g e

Cross-border investments in the financial sector were a notable feature of the region even prior to the 2008 global crisis and these investments resumed after the crisis eased. The Table below shows the substantial presence of cross-border financial conglomerates in the region in 2010, including Colombian banks. Costa Rica has the largest presence of foreign financial conglomerates, which operate alongside its state owned banks.

Principal Financial Conglomerates Operating in the CAPTAC-DR Region

Name Promerica LAFISE Ficohsa Citi BI HSBC /

Davivienda Scotia G&T Banco Popular BHD Ban-

Colombia BAC

Supervisor at origin PAN PAN PAN USA GTM COL CANA

DA GTM DOM DOM COL COL

Regional supervisor PAN PAN PAN PAN n.a. GTM DOM DOM PAN PAN

Main supervisor NIC NIC HND GTM PAN GTM PAN

Presence in (1)

NIC, PAN, SLV, CRI GTM, HND, DOM, ECU, USA, MSR

HON, NIC, PAN, MSR CRI (2)

HND, PAN GTM

CRI,NIC SLV, HND GTM , PAN, , DOM

GTMSLV, HND

CRI, SLV HND, PAN

CRI, SLV, GTM, NIC, PAN, DOM

GTM, PAN, CRI, SLV

PAN, DOM

PAN, DOM

PAN, SLV GTM, CRI, HND, PAN, SAL, NIC

(1) Bold results indicate a significant presence of the financial conglomerate. (2) The CF also has a presence through non-bank entities in Guatemala, El Salvador, the Dominican Republic, Mexico, USA, Venezuela, Belize, Colombia, Ecuador, Bahamas and the Cayman Islands. Source: CAPTAC-DR. The Central American Council of Superintendents of Banks, Insurance and Other Financial Institutions, (Consejo Centroamericano de Bancos, de Seguros, y de Otras Instituciones Financieros – ‘CCSBSO’) has taken a substantial interest in improving cross-border financial supervision for some time, in order to reduce financial sector risks relating to regional financial conglomerates. CCSBSO was originally formed as a regional body in 1976 with Honduras, Guatemala, El Salvador, Nicaragua, and Costa Rica as founding members. Panama and the Dominican Republic joined in 1999. Significantly for the scope of effective, cross-border CS, Colombia also became a member in August 2012. The global crisis temporarily focused the Council on immediate concerns of liquidity, stress tests and supervision of the region’s two largest financial conglomerates. However, as the global crisis eased in the region, substantial regional cross-border financial investments resumed, including investments by Colombian banks. At the same time, there have been instances of withdrawal of banks from OECD countries who have been rebuilding their capital in their home country markets. For example, Davivienda (Colombia) took over HSBC’s investments in Costa Rica, Honduras and El Salvador in 2012. As a result of these developments, the CCSBSO’s concerns regarding reduction of potential financial sector risks shifted back to the issue of cross-border financial supervision. The IMF agreed with this diagnosis. CAPTAC-DR became the provider of technical assistance to the CCSBSO in cross-border, consolidated financial supervision, through projects that were endorsed by its Steering Committee. CS is a comprehensive approach that evaluates the strengths of an entire group. Financial conglomerates (which could be part of a wider non-financial group) are defined as operating at least two different entities in the financial sector such as, banking and insurance. Conglomerates may include non-financial activities, as well as those typical of a financial conglomerate as illustrated in the following page.


25 | P a g e

Panama has become a significant regional financial centre and many regional banking groups have established holding companies there. It hosts a large number of financial institutions. The website of the Panamanian Bank Superintendence shows that the country currently has in operation (i) two state owned banks; (ii) 48 general license banks; (iii) 28 international license banks; (iv) and 14 representative offices.7 In addition, the Superintendency lists 70 Trust Companies. By way of comparison, Guatemala bank superintendence supervises 18 banks, 14 financial companies, seven offshore

financial companies, as well as a number of insurance providers, credit card firms, etc.8 The percentages of total financial assets held by foreign banks reaches 51 in Panama, 13 in Guatemala, 22 in Colombia, and 15 in the USA.9 An IMF working paper10 dated December 2011 noted that financial regulation and supervision in the region lagged behind that of the larger countries in Latin America and international best practice. The paper applied a modified Basel Core Principles (‘BCP”) assessment (drawn from FSAPs except for Panama) of various sub-topics under the risk heading, where 100 represented full compliance. Using this measure the region as a whole (excluding Nicaragua for which no prior BCP assessments were available) scored 48 for risk-based supervision (‘RBS’) and 56 for consolidated cross-border supervision. They noted that the scores varied from country to country with Panama recording the highest score on both counts. The same paper also noted that cross-border supervision was often hampered by inappropriate legislation. Legal and regulatory shortcomings were mentioned for Costa Rica, El Salvador and Dominican Republic. We have already noted some of the legal issues confronting Panama. However, in other ways, Panama is advanced, with its banking laws and its close domestic coordination among all financial sector supervisors.

7 Information available at http://www.superbancos.gob.pa/en/igee-general-information. Accessed on 20 February 2012. 8 Information available at http://www.sib.gob.gt. 9 Source: ASBA working group 3 report 2010. 10 Developments in Financial Supervision and the Use of Macro-prudential measures in Central America, Fernando Delgado and Mynor Merza, IMF Working Paper WP 11/299, 2011.


26 | P a g e

Two particular issues represent a potential challenge for fully effective information-sharing within CCSBSO, namely: (i) Colombia’s full participation in information sharing, now that it has become a member, and (ii) Panama’s legal framework, which currently does not include a legal basis for supervising holding companies. Panama appears to be fully committed to the aims of the CCSBO and plans, with CAPTAC-DR/IMF assistance, to amend its legislation. The CAPTAC-DR RA estimates that the time frame for the necessary amendments is mid-2013. 3.4 THE RESULTS CHAIN FOR CROSS-BORDER CONSOLIDATED SUPERVISION A sector level Results Chain for CS would have been ideally developed ex ante, as a project planning tool. An illustrative one is provided below. It is intended to highlight the key technical assistance/reform needs beyond the scope of the project, and the requirements for appropriate sequencing, that need to be met in order for the achievement of the high level outcomes and impacts, and to identify related risks and assumptions. It is provided on an illustrative basis only as part of the Case Study.


27 | P a g e

An analysis of the Results Chain highlights the following: Overall Objective and Impact The project seeks to contribute decisively to the adaptation and implementation across CAPTAC-DR members of best international practices in supervision and regulation. Its major thrust has centered on risk-based supervision, cross-border consolidated supervision, and the adoption of prudential criteria based on international accounting standards. Given the relatively important presence of regional and global financial conglomerates, compliance with best practices in this area is expected to bring about significant gains in terms of increased financial stability across the region. Diagnosis The sixth annual regional conference for the CAPTAC-DR member countries, organised by the IMF and held in Cost Rica in June 2007, focused on CS of regional financial conglomerates as one of two priorities, the other being a customs union. The CCSBSO had taken a substantial interest in improving cross-border financial supervision, in order to reduce financial sector risks related to regional financial conglomerates. The global crisis temporarily focused the Council on the immediate concerns of liquidity, stress tests and supervision of the two largest financial conglomerates. With the easing of the global crisis and the resumption of substantial regional cross-border financial investments, the CCSBSO’s concerns have shifted back to cross-border supervision. In 2011 CAPTAC-DR conducted its own assessments of all seven of its member countries on their level of compliance and readiness to achieve effective CS domestically and regionally. Finally, in December 2011 an IMF working paper on RBS in Central America identified compliance shortcomings with BCP principles on cross-border CS and analyzed the main shortcomings’ possible origins. Strategy After conducting its own assessment of the priorities for effective cross-border consolidated supervision in 2008, CCSBSO completed a strategic plan for the period 2010-2014, with advisory input from CAPTAC-DR. The sixth IMF sponsored regional conference in mid-2007 and the further reports from the Association of Supervisors of Banks of the Americas (’ASBA’) working group (sponsored by both the IMF and World Bank) informed this process. The CCSBSO Strategic Plan11 focused on four main areas of development: Enhanced stability; Risk-RBS; International Financial Reporting Standards (‘IFRS’); and Cross-Border Consolidated Financial Supervision.

11 CCSBSO web site.


28 | P a g e

Legislation Overall, achieving regional harmonization and compliance with BCP standards requires an important effort in terms of amending both legislation and regulations. The IMF working paper of December 2011 referred generally to the need for legal reform and referred to legislative and regulatory shortcomings in several countries, including Panama, Costa Rica, El Salvador and Dominican Republic. The ASBA working group also referred to a need to address legal and regulatory frameworks, including incorporation of the analysis of offshore banks. General legal and regulatory issues to be covered include powers to supervise non-financial activities of a conglomerate, non-bank financial activities information sharing, and capital requirements of financial conglomerates12. CAPTAC-DR is addressing some of these issues in Panama. Nevertheless there seems to be no evidence of any TA in this area to other member countries in the 2013 work plans. Activities So far CAPTAC-DR has carried out a number of initiatives, namely: assessments/diagnoses, strategic advice to CCSBSO, and completion of a methodology for risk analysis and consolidated cross-border supervision of financial conglomerates. The development of the methodology (which includes procedures and report formats) was a sequencing priority. There was no evidence from the CCSBSO web site that strengthening legislation/regulation was part of the strategic plan. CAPTAC-DR also held a regional workshop on the issues of cross-border CS in April 2012. In April 2013 CAPTAC-DR plans to test the application of the methodology with respect to at least two of the region’s conglomerates. The Center will also assist the supervisors once the (multilateral) Colleges of Supervision have been established by CCSBSO and its Liaison Committee. This work will seek to facilitate cross-border supervisor coordination in analyzing financial conglomerates or holding groups that include financial subsidiaries. Outputs Project outputs follow a logical sequence. The first output is a report proposing suitable supervision methodologies on the basis of the needs assessed. The second is staff trained in these enhanced technical approaches for cross-border consolidated supervision. Third, information is to be produced in a way that can be easily shared between countries. Finally, supervisors are supported in implementing the new methodologies and systems on a day-to-day basis. Outcomes In the short-term, efforts to implement the methodology are already underway in Costa Rica and Guatemala. CAPTAC-DR has provided TA to Panama for its modification of laws to allow supervision of financial holding companies. In December 2012, the final version of the methodology, including comments from the IMF headquarters, was presented to the Superintendencies of the seven countries. Also in December, CAPTAC-DR set up TA to support Panama in drafting changes in legislation that would allow supervision of holding companies that included financial intermediaries.

12 Guatemala, Honduras and Nicaragua are working on NBFI reforms.


29 | P a g e

In the medium-term, substantially improved compliance with the BCPs on cross-border supervision is expected. The commitment to substantial improvement in information exchange over the medium-term seems realistic, given that (i) CCSBSO considers this a priority issue; (ii) all member countries’ signed a multilateral information sharing agreement in 2007; while (iii) in November 2009, the supervisors of El Salvador, Panama and Costa Rica also signed more detailed agreements to exchange information on entities operating in their jurisdictions. Full compliance or even substantial compliance will of course depend on the quality of supervision in each country and will also require strengthening the respective legal and regulatory frameworks. Donor Coordination There have been several complementary TA activities in the region over the past seven years, namely: The FIRST Initiative funded a $175,000 project in 2005 to strengthen the supervision of financial conglomerates in El

Salvador. The IADB has provided TA on RBS and IFRS in Panama and Nicaragua. Starting January 2013 a regional project focusing on AML/CFT is also to be launched (this initiative should also cover

holding companies that include financial subsidiaries as well as financial conglomerates). A $350 million IADB project for Panama incorporates a systemic risk component. The World Bank has been advising on supervisory structure in El Salvador. The FIRST Initiative has also funded, since 2009, crisis simulation and preparedness projects in Costa Rica,

Guatemala and El Salvador, including a regional workshop on these topics. It seemed from our interviews that CAPTAC-DR was unaware of some of these specific TA projects, although there had been coordination with both the IADB and the World Bank on their specific projects. Other TA from CAPTAC-DR TA in the region has varied from country to country. At a regional level for year ended June 2012 there has been TA on operational risk through development of or improvement in the prudential norms, related training of supervisors and banks, and the development of IFRS. During the year ending 2013 there was also TA on liquidity risk for Costa Rica and Guatemala and on RBS of the insurance sector in Honduras. Regional workshops included: credit risk (September 2011), basics of RBS (December 2011), stress testing (run by the IMF in 2011) and CS (April 2012); there was also a regional seminar on “Success Stories” in CS. Pre-conditions/Risks Fully effective compliance with cross-border CS requires all CAPTAC-DR members to achieve similar standards in RBS and strengthening of their respective legal and regulatory frameworks to enable comprehensive supervision and regional information sharing. Legislation may not proceed expeditiously or at an even pace in each country and, especially implementation of the legislation, may require a higher-level commitment from governments, over and above that of the superintendent members of the CCSBSO.


30 | P a g e

3.5 CAPTAC-DR TECHNICAL ASSISTANCE Background Since early 2008, CCSBSO endorsed the current project phase, which focuses on enhancing inter-institutional coordination and strengthening supervisory capacities. Another key component was the harmonization of prudential regulations as per the recommendations of earlier TA from the IMF. With the advent of the global financial crisis late on that year, this regional counterpart requested to modify the work program seeking to establish ad hoc mechanisms to monitor and supervise liquidity-related risks of regional financial conglomerates. The other key area was the collaboration with the regional liaison committee in order to improve the supervision of the most important financial conglomerates in the region. As the crisis was surmounted priorities slowly re-centered themselves around risk monitoring, supervisory capacity building, and harmonization of regulations at the regional level. Within the latter, the most important aspects identified were capital adequacy, classification of financial losses, reserve requirements, credit risk concentration, as well as management of supervisory issues emerging from real estate credit activities. Key TA Activities, Outputs & Outcomes TA by CAPTAC-DR on a regional basis started, in FY2010, with advice to the CCSBSO on its strategic plan. Regional TA on methodology, guidelines and procedures began in FY2011 with a diagnostic process to assess existing procedures as a basis on which to develop an improved methodology. In FY2012 CAPTAC-DR TA aimed at improving the methodology further and developing risk-based profiles. In addition, the Center held a regional seminar to review some success stories in CS (which took place in April 2012 with 25 participants). The time allocated for the 2011/12 plans above was for 2 weeks of RA time and 7 weeks of STX time. For FY2013 the plan is to continue monitoring the implementation of the methodology. As part of this initiative, a specific information exchange of analysis and data is planned for April 2013. TA inputs are planned to comprise 1.5 weeks from the RA and 4 weeks from the STXs. During FY2012 the activity was aimed at partial implementation of the new methodology as a test, through effective coordination with the Liaison Committee of the CCSBSO. Some TA was also directed at strengthening Panama’s supervision of banking groups in order to provide constructive feed-in to cross-border cooperation. The FY2012 TA project on CS was sequenced appropriately. It began with scoping activities in each country; followed by development of a simple, common reporting methodology that was consistent with best international practices; then followed by workshops in each of the countries to train the supervisors in implementing the methodology and finally a presentation of a summary of the report. The final report, incorporating IMF HQ comments, was sent to the CCSBSO and the supervisory agency in each country in December 2012. The TA was generally appropriate to the level of supervisory capacity in each country. Specifically, during the period 2011 to end of 2012 the project assessed, country by country, the level of development reached for contribution to effective cross-border CS and what would be desirable for a general information framework exchange. As a follow up, a methodology was developed for consistent data collection and analysis in each country, with a view to organized exchange of information on a regular basis (monthly and quarterly).


31 | P a g e

The diagnostic phase was completed by May-2011. During mid-2011 CAPTAC-DR produced guidelines on governance, solvency, contagion, credit, liquidity, market and operational risks. Workshops of about two days each were then carried out in each country and following these between two and four days in each country on assessing the supervisors’ implementation of the methodology. In order to coordinate and maintain consistency the CCSBSO established a Liaison Committee which is also a counterpart body for CAPTAC-DR. Nevertheless, without an effective enabling legislative and regulatory framework, it will not be possible to implement CS fully. Planned future activities include working with the College of Supervisors as formed by CCSBSO and its Liaison Committee. CAPTAC-DR will jointly test the implementation of the methodology working with at least two regional financial conglomerates and continuing to support the Liaison Committee. CAPTAC-DR TA also will provide support to Panama on strengthening its legal framework, to enable it to demand information and inspection of all parts of any group from holding company downwards that includes financial activities and or financial conglomerates. The outputs so far are a methodology based on a) scoping missions to each country, b) presentations of the methodology at workshops and c) practice in using the methodology alongside supervisors in each country, leading to a further output of “trained supervisors”. The methodology includes reporting forms to cover corporate governance, solvency, contagion risks, credit risk, liquidity and market risk, and operational risk — an appropriate set of issues. The outputs of the TA were clearly-defined, well-sequenced (other than addressing legal frameworks), and produced in timely fashion over the relatively short period of 21 months. The methodology was well received, as were the workshops. A summary of the results was presented to the CCSBSO and the final report, including IMF HQ comments, was presented to CCSBSO and each supervisory agency in December 2012. These outputs were not formally identified in a log-frame at the time the TA was initiated because CAPTAC-DR had not yet begun to use log-frames formally. The TA provided by CAPTAC-DR represents a base for starting the collection of the necessary data, sharing it between national supervisors, and making future improvements in supervision based on international best practices. However, financial supervision is not simply collecting data; reducing financial risk — the longer run goal — means keeping pace with market developments and that supervisors must have the willingness and the legal power to engage in prompt corrective action to strengthen weak banks and intervene promptly if necessary, which may involve legal changes, including changes to manage the assets across countries if a cross-border financial conglomerate fails, as is now being discussed in the US and UK. Thus, like all financial supervision, cross-border, consolidated supervision involves continual changes in regulations to keep pace with market developments, updating the types of information that are collected, and ensuring that the legal framework supports strong supervision. The CCSBSO’s own implementation targets for cross-border CS are as follows:13

Task / Year (in percentages) 2012 2013 2014

Methodology/procedures 30 60 100 Standard classification criteria - 50 100 Pilot - - 50 Final implementation - - - Automated exchange of information- monthly/quarterly 50 100 - College of Supervisors-pilot 30 100 -

13 CCSBSO website.


32 | P a g e

The principal expected outcomes are summarized below:

Key Outcomes Short-term


(from 1-3 years) Long-term

(from 3-10 years) Strategic plan accepted. Methodology accepted and implemented- initially on test basis. Legal and regulatory frameworks

harmonized and structured to permit full BCP compliance.

College of Supervisors using methodology on all relevant groups.

Consistent and agreed classification criteria.

Improved methodology accepted and deployed.

Automated information exchange system working.

Pilot implementation achieves 80% to full compliance.

80% to full compliance with Basel II Core Principals 24 and 25 (presently less than 50 except for Panama which is higher than 50).

The short-term outcomes are largely complete, although the methodology remains to be tested and further improved in the light of those tests. The medium-term outcomes appear to be on track given the strength of CCSBSO’s commitment. The main challenges lie with assuring the quality of supervision in each country and improving and harmonizing the legal and regulatory frameworks sufficiently to ensure good coverage in supervision of all activities of holding groups that include financial entities. The revision of Panama’s legal and regulatory framework is needed urgently because Panama also hosts the largest number of foreign financial groups. The long-term outcome (more than three years from the beginning of the project) is for each country in the region to comply effectively with at least 80 per cent of the principles on cross-border CS of the Basel Core Principles (mainly numbers 24 and 25 in Basel II).; CAPTAC-DR’s TA should be a major contributor to this long-term outcome alongside TA from some other donors and the unaided efforts of each of the countries in the region through the work of the Liaison Committee of the CCSBSO, and the soon to be established College of Supervisors. The medium-term TA regional outcome of a common and harmonized approach to information gathering, analysis and sharing is a good basis for cross-border CS, but it is only a first step in a long process. Achieving the medium and long-term goals of reduced financial risk will take time and involve a number of changes, including legal changes in some cases. For example, Panama currently does not supervise its bank holding companies, which means it cannot provide information on them to other countries. To support legal improvements in this case, CAPTAC-DR is providing TA to Panama to support necessary changes to the relevant laws. Another example is the growing importance of Colombian financial conglomerates in the region, which will need to be incorporated into the cross-border framework (although a promising start has been made with Colombia’s supervisors joining the CCSBSO in August 2012).


33 | P a g e

At least two of the financial groups operating in the region are unlikely to fully cooperate: the Citi Group from the USA and Bank of Scotia (Canada). Until the financial crisis in 2008, full compliance in information sharing by strong economies such as the USA and Canada would not have been of great concern. Since 2008 we have seen a potential contagion risk by, for example, a troubled financial group from an OECD country pulling assets back to home base. Achievements This work constitutes a first and significant step in improving cross-border consolidated supervision and exchange of quality information by supervisors across the region. CAPTAC-DR’s main achievement has been to help set this regional process in motion while the CCSBSO provides a sound forum to achieve the necessary harmonization. The Center was able to support its member countries with sound needs assessment, sensitization, methodology development, training, as well as with TA seeking to accompany the first steps in implementing these new approaches. As methodologies have become pilot tested in Guatemala and Costa Rica, their validity and usefulness have become apparent to supervisors across the region. In its short implementation time, the project has attained its key short-term expected outcomes. The methodology was produced, accepted, and pilot-tested. Supervisors from the region now have a more solid base to step up efforts to exchange information. Remaining Challenges The key challenges ahead are four-fold. The implementation needs to become deeper and more sophisticated in Guatemala and Costa Rica. These methodologies need to be generalized to other countries in the region. For cross border supervision to become a reality, each supervisor needs access to appropriate information from its peers in adequate time and form. Achieving this goal will require determination on their part, as well as patient and systematic on-the-job training from CAPTAC-DR. At the same time, the general quality of supervision will need to improve and keep pace with changes in the market. The second challenge relates to the enactment of accompanying legislation. Now that the reform process has gathered enough momentum, it becomes critical to adapt national legislation to the reality of cross-border conglomerates. Panama, for example, needs to find a modality to extend supervision to holding companies. These types of companies are not only outside of the regulatory framework but also they are at the core of regional financial conglomerates. Regulatory changes will also be necessary in order to grant the necessary legal power and political willingness to the supervisors for engaging in prompt corrective action to strengthen weak banks if necessary. Third, since Colombian banks have substantially stepped up their presence in CAPTAC-DR member countries, the participation of this country at CCSBSO becomes an extremely welcome development. Transforming this new membership status into active collaboration and information sharing is one of the remaining challenges in this area. Finally, sustainability will depend on the supervisory agencies’ capacity to retain a high quality staff through reasonable salaries and through legal protection for their activities as indicated in the BCPs, which is part of supervisory agencies self-assessment programs and the World Bank-IMF Financial Sector Assessment Program (FSAP). The RA for CAPTAC-DR has expressed concern that some of the country supervisory agencies face high staff turnover, which frustrates training initiatives.


34 | P a g e

3.6 PRINCIPAL CONCLUSIONS Annex 3.8.2 presents our detailed responses to the Key Evaluation Questions and the basis of arriving at our ratings for the various DAC criteria. In summary: We rate Relevance of the project as Excellent with a score of 3.6. The TA project is fully consistent with the priorities of the CCSBSO and with the overall strategic objective of CAPTAC-DR in the area of financial supervision. The outcome of the TA complements the work of other development partners in the region in the financial sector and more generally. The score also reflects the risk that none of the CAPTAC-DR governments have a clear national strategy for overall development that includes financial sector development and that, despite CAPTAC-DR’s communication of its work plans with the World Bank and the Inter-American Development Bank, the limited awareness of some of the complementary TA provided in the region suggests that donor consultation could be improved. In addition, updating legal and regulatory frameworks need prioritization for the TA to deliver outcomes, and therefore require appropriate sequencing. With respect to Effectiveness, our rating at the Outcomes level is Good with a score of 3.2. The high quality of advice delivered is reflected in in a rating of Excellent at 3.6 for Outputs. We could not rate outcomes as Excellent because it remains to be seen how comprehensive and quickly the legal and regulatory frameworks can be upgraded, how quickly each country can implement the methodology and even then some gaps will remain in obtaining the cooperation of some jurisdictions such as the USA and Canada. Financial supervision standards and capacity will need to be continually updated to deal with financial innovations. The short-term outcome of applying the methodology in gathering analyzing and reporting on financial groups on a consolidated basis has already started in Costa Rica and Guatemala. Efficiency is rated at the high-end of Good at 3.3. TA input information is available in person-weeks (in the field only), and not in monetary terms. We therefore have data of time planned in person-weeks (work plans in the CAPTAC-DR Annual Reports) versus the actual number of weeks spent on the overall topic of supervision (Annual Reports). The TA appears to be delivered close to plan. Our principal concern is whether the TA, which was sufficient for this initial stage, may need to be increased substantially and how much more may be needed to ensure high quality supervision. Sustainability is rated as Good at 3.0. The project represents the first effort to share good quality, consistent information on consolidated supervision of cross-border financial conglomerates regionally, based on international best practices of supervision. Sustainability of the TA project’s results will depend not only on adoption of the methodology and improvement and maintenance of the capacity of the supervisory institutions but on upgrading the methodology and the capacity of the supervisors to deal with new developments in financial markets. The capacity of the supervisors in the region is reasonable but would benefit from improvement. Finally, sustainability will depend on the supervisory agencies’ capacity to retain a high quality staff. The RA for CAPTAC-DR has expressed concern that some of the country supervisory agencies face high staff turnover, which frustrates training initiatives. We were unable to provide a higher score because it is still too early to judge how much traction there will be from the recently completed TA on methodology and what specific obstacles, if any, might be posed by the need to upgrade the legal and regulatory framework.


35 | P a g e

3.7 KEY FINDINGS AND RECOMMENDATIONS Findings 1. The project design has gaps in so far as the ultimate objective of a fully compliant system of cross-border

consolidated supervision requires further interventions than those planned. The Results Chain approach facilitates this analysis.

2. Interventions focusing only on the methodology alone cannot deliver the final objective not only because of the capacity of the supervisors but because of gaps in the legal and regulatory framework, as well as likely shortcomings in the implementation of a full range of risk supervision tools (such as liquidity).

3. The TA on the methodology has already lasted about 21 months. Further TA is planned and needed to test the implementation, assist with legislation in Panama and advise the yet to be established College of Supervisors. Adding all the TA components may require around three or more years. The planning process is however organised annually; this process obscures an overall programmatic approach and can lead to fragmented TA efforts that miss the big picture.

4. The pace of development is likely to vary from country to country yet, as a regional project, the TA delivered appears to be divided equally among the seven members.

5. The time dedicated to training via workshops and mentoring seemed very low (about two days and three days respectively per country). It is likely that more training and hands-on support would be desirable.

6. The TA has been well received and two of the more advanced countries in supervision, Costa Rica and Guatemala, are starting implementation already.

7. It seems unlikely that other donors would have been willing to fund this work; the TA required is too small for IFIs in the region and too specialised for most bilateral agencies. The CAPTAC-DR intervention was therefore essential to stimulate progress in this area.

8. Using only one or two STXs supported by an experienced RA seems a good formula for ensuring a consistency of TA quality and good communication at country as well as regional level.

9. The process was also facilitated by working through the CCSBSO’s Liaison Committee. Recommendations 1. Attention should be paid to sequencing and addressing further issues and gaps, which often may be highlighted by

initial reforms. CAPTAC-DR is addressing some of the other risk areas on a country-by-country basis and regionally with support for improved supervision of operational risk. We believe country-level interventions should also more explicitly integrated with the regional CS project.

2. CAPTAC-DR could help address the gaps and perhaps either plan to cover them itself or with IMF HQ support or approach other donors to contribute (e.g. on the legal and regulatory front). At the moment there is a risk that the good results achieved on developing the methodology could be prejudiced by the lack of strengthening the legal and regulatory frameworks, especially as this process will require TA and also the commitment of higher levels of government in each country.

3. We recommend that an absorptive capacity analysis be carried out for each member country and the TA delivery effort adjusted accordingly.


36 | P a g e

3.8 ANNEX

3.8.1 SUGGESTED LOGICAL FR AMEW ORK The table below lists activities and outputs from project documentation (annual reports, work plans, etc.): Mission Date Inputs Activities Principal Outputs Notes/Comments

2010 1. 6 weeks

RA The activities in FY2010 focused on identifying member countries’ TA needs. The STX worked with the CCSBSO on its Strategic Plan and assessed the developments in regulation and supervision, with emphasis on standards relating to consolidated cross-border risk management.

1. Identified the relevant development in the areas of regulation and supervision in the member countries, with specific emphasis on the standards relating to cross-border CS risk mgm’t.

2. Strategic Plan of CCSBSO.

The plan was largely the work of the CCSBSO with improvements proposed by the CAPTAC-DR RA.

2011 2. 1.5

weeks RA 4.5 weeks STX

Diagnostic assessment of pre-existing procedures and reports and recommendations for improving the methodology for assessing the risks associated with the financial conglomerates in the region.

The CCSBSO Liaison Committee was provided with a more complete tool to be used as a transition mechanism until a new methodology is formally adopted in mid-2012. Mission Reports.

CCSBSO established a Liaison Committee to act as a buffer with the supervisors country by country.

2012 3. 5.5

weeks RA 17 weeks STX

Development of a new methodology for all countries for effective CS (conducted in coordination with the Liaison Committee). This methodology will apply to all financial groups present in each country during the next FY.

1. Trained staff. 2. Methodology for cross-

border CS: report formats and guidelines for the following risks: Governance, Contagion, Solvency, Credit liquidity, Operations.

3. Information systems.

A comprehensive set of reporting formats and accompanying guidelines was delivered regionally to the supervisory agencies in each country through short workshops.

1. Banking supervision IMF Institute: capital adequacy, supervision of large & complex financial institutions and risk evidence analysis.

2. Seminar on RBS. 3. Seminar on regional experience

with supervision of financial conglomerates.

The courses were well received by participants according to feedback reports.

2013 4. 1.5

weeks RA 5.5 weeks STX

Monitoring of implementation of the Consolidated Trans-border Supervision (CTS) methodology and guided support.

Information exchange on selected regional financial conglomerates is set for April 2013. The goal is to apply the methodology to all regional conglomerates by 2015. This will entail legal changes in Panama


37 | P a g e

Mission Date Inputs Activities Principal Outputs Notes/Comments

and Colombia’s exchange of information with the other CCSBSO members.

Note: Until now work plans did not include TA on the legal and regulatory frameworks. Nevertheless, TA in this area is currently being delivered to Panama. 3.8.2 EVAL UATION K EQS AND RATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities


The Consolidated Cross-border Supervision TA is fully consistent with the Program Document, CAPTAC-DR’s overall strategy and strategy in the area of financial supervision, and the work plan endorsed by the Steering Committee.


Fully consistent with priorities expressed by CCSBSO, in which all supervisory authorities in CAPTAC-DR are represented but no country has a national strategy.


This project is compatible with regional plans and priorities.

o Extent to which activity is linked to and supports sector/topical reform strategy;

The Activity is the basis of strengthened consolidated cross-border supervision through provision of a common methodology for supervising cross-border financial conglomerates, based on international best practices, including Basel Core Principles 24 and 25.


The CCBSBSO and its Liaison Committee requested this assistance. The activity was endorsed by the Steering Committee of CAPTAC-DR.


The activity is the first step, after development of the strategic plan, in improving cross-border CS and exchange of good quality information by supervisors. The specific activity was carried out though a sequence of diagnostic missions to each country, design of the methodology to be used by the supervisors in all countries and for the exchange of information, workshops in each country to train the supervisors in the use of this methodology, specific assistance in each country to accompany the supervisors in implementation, and delivery of a final report, including IMF HQ comments, in December 2012 to supervisory agencies in the region and CCSBSO. Additional activity will be needed including legal changes. To build momentum and kick-start the process, including exchange of information across borders, it was appropriate to implement a common methodology before addressing various legal and regulatory issues that vary from country to country, although it is important that the latter issues are now addressed.


Capacity in the recipient countries is appropriate for implementation of the project’s outcome. The pace of development will vary from country to country with the more advanced leading the process; the pace will also depend on progress in any legal issues.

Consistency with IMF Headquarter/other activities Whether activity is appropriately focused in terms of

subject area, taking into account the IMF’s expertise and integration with HQ and other Fund activities (e.g. other training and TA delivered including through the Topical Trust Funds).

The consolidated cross-border supervision task is appropriately focused taking into account IMF knowhow and HQ activities in the CAPTAC-DR region and CAPTAC-DR’s expertise.

Coordination with Development Partners


38 | P a g e

Key Evaluation Questions Comments Whether the intervention has been effectively

coordinated with and complements the work of other development partners.

The activity falls within the core areas of IMF expertise and specialization. There was dialogue with IDB and World Bank but some of the other TA initiatives in the area did not appear to have been discussed intensively.

Context of donor/TA landscape of IMF TA Why did the TA recipient choose the CAPTAC-DR as the

TA provider as opposed to other donors? The counterparts had worked with the IMF in this area beginning in FY2008. Once CAPTAC-DR was up and running in 2010 it was the natural partner to work with the CCSBSO in the TA. The IMF is well- qualified in this area; other donors have not been involved extensively over time.

How do workshop participants compare those provided by the CAPTAC-DR with workshops from other donors?

Workshop participants rated the CAPTAC-DR workshops on cross-border CS highly; there is no data on workshops from other donors in this specialized area of TA.

Score for Relevance 3.6 Excellent EFFECTIVENESS Impact Intervention’s expected Impact achieved or likely to be

achieved (to the extent defined in the activity’s log frame).

No logframes had been developed in CAPTAC-DR at the time of initiation of the activity (March 2011). Regional financial stability would likely improve as a result of well-coordinated cross-border, consolidated supervision of regional financial conglomerates. The CAPTAC-DR TA is expected to contribute to this goal.

Outcomes Intervention’s Outcomes achieved or likely to be

achieved (as defined in its log frame): Short, medium and long term Outcomes.

The short-term outcome was fully achieved in terms of the providing the methodology for and training in the use of a reporting system of cross-border activities by international financial conglomerates that can be used to exchange information among CAPTAC-DR members. The method is now beginning to be used in Costa Rica and Guatemala. Achieving the medium- and long-term objectives will take time and involve a number of complicated changes, including legal reform. First, the system of exchanging information will need to be adopted. Second, some cross-border firms are holding companies in Panama, which are not yet supervised there. Third, Colombian financial conglomerates are important in the region and should be incorporated in the cross-border supervision framework. Fourth, the general quality of supervision will need to improve and keep pace with changes in the market. Fifth, reducing risk entails the legal power and willingness of the supervisors to engage in prompt corrective action to strengthen weak banks if necessary. In the case of subsidiaries of cross-border conglomerates, this may require some legal changes, as well dealing with a failure of an international conglomerate located in more than one of the countries.


The CAPTAC-DR activity has made a significant contribution to the supervisory agencies capacity to carry out cross-border supervision, through development of a methodology that is beginning to be used in the region and training in that methodology.

Score for Outcomes 3.2 Good Outputs Intervention’s Outputs achieved or likely to be achieved

(as defined in its log frame), including: o Timeliness of reports/workshop presentations; o Quality of reports/workshop presentations; o Appropriateness/applicability of advice, given


Outcomes.

Logframes were not used by CAPTAC-DR at the time of initiation of the project. Outputs were clearly defined and well-sequenced in the Project Documents expressed as: (i) a methodology for supervising cross-border financial conglomerates and exchanging information in the key areas of corporate governance, solvency, contagion risks, credit risks, liquidity and market risks, and operational risks; (ii) workshops that explained use of the methodology ; and (iii) activities accompanying supervisors in implanting the methodology. The reports and workshops were of high quality, appropriate to the problem and the capacity of the supervisory agencies, and well-received by the participants. However,


39 | P a g e

this is very much a work in progress and capacity building will require substantial further training and TA. Outputs represent an important first step toward the long-run, complicated goal of reducing financial risk due to cross-border financial conglomerates, by improving cross-border, consolidated supervision. Some countries have already adopted key parts of the methodology and CAPTAC-DR is already providing follow up assistance. Some of the countries’ supervisors are likely to require more extensive TA, including “hands-on” capacity building.

Score for Outputs 3.6 Excellent EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; The decentralization of the activity in CAPTAC-DR and the substantial

involvement of the RA, backstopped by IMF HQ, was a major factor in the efficiency of the activity and its management, in particular the multiple visits to each country in a short period to deliver an appropriate product and begin its absorption by the supervisors. The geographic closeness and involvement of the CAPTAC-DR RA facilitated close monitoring of the project and contributed to the rapid transfer of the new methodology to the supervisory agencies in the seven countries. Our concern is whether the TA, which was sufficient for this initial stage, may need to be increased substantially and how much more may be needed to ensure high quality supervision. Considering that TA data is provided in person/weeks as opposed to monetary terms, the time necessary for specific activities helps assess efficiency. Thus training workshops in country were about two days per country and actual accompaniment of supervisors in their tasks was about three days, even though the level of supervisory development in the countries varied. The entire project (to June 2012) appears to have been about 130 days of TA from the STXs. The questions are whether sufficient “hand-holding” has been provided and will more TA be made available if necessary for this and for contingencies. CAPTAC-DR already has responded appropriately to provide support for legislative changes in Panama. Another good test will be the implementation starting in April 2013, which may demonstrate whether further capacity-building TA is needed.

Timeliness in executing the activity; The principal part of the activity took 14 months, from the beginning of scoping activities to the workshops and activities to support the implementation of the methodology. Incorporation of comments and delivery of the final product to the supervisory authorities, in December 2012 took an additional 6 months. The outputs appear to have been delivered within budget in terms of person-weeks, based on a comparison of work plans and actual data in Annual Reports.


The management of the activity was largely decentralized into CAPTAC-DR, as was appropriate.

Appropriateness of selection of counterpart/workshop participants (where relevant);

The counterparts were the supervisory agencies in the CAPTAC-DR countries, the workshop participants were the supervisors in these agencies.


The location of CAPTAC-DR in the region and the substantial involvement of the RA in this activity provided for good, timely management, with backstopping from HQ. Policy direction from IMF HQ seemed appropriate but the reporting demands between CAPTAC-DR and HQ may have been unnecessarily onerous.


Efficiency of planning and execution of the TA was highly satisfactory; it involved development of a new methodology and its transfer through training in a short period.

Efficient use of resources (human and financial) and


40 | P a g e

attention to cost effectiveness Appropriateness of staffing composition for the

intervention; The RA and the two STXs carrying out the activity were highly experienced supervisors. The small team was able to provide consistency and continuity in approach.


The work allocation between HQ and CAPTAC-DR was appropriate. CAPTAC-DR made it possible to use regional STXs and permitted three visits to each country (scoping mission, training workshops and accompaniment of supervisors to help them implement the methodology), as well as the close involvement of RA.

Appropriateness of degree and nature of use of regional/ local expertise;

Regional expertise in financial supervision in Latin America of the IMF. CAPTAC-DR and the STXs were appropriately used.

Whether expenditures have been in line with activity budget;

The expenditures appear to be in line with the activity budget in terms of person-weeks. Dollar expenditure budgets by activity are not easily available in the IMF system.

Whether activity, given its results, has been cost-effective compared to other TA delivery modes, given results;

The activity appears cost effective given the outputs and compared to other TA delivery modes. It benefitted from the location of CAPTAC-DR with a consequent low logistical cost of multiple missions. However, given the lack of cost information, we cannot be definitive in this area.


The TA used the standard CAPTAC-DR model of substantial involvement of regional STXs, managed by the CAPTAC-DR RA and back--stopped by IMF HQ. The opportunity for efficiency gains during the intervention was limited given the short-time frame of this TA. Arguably more TA resources may be needed rather than less, spread out over a longer-term and allocated where additional training in the new methodology was needed.

Monitoring and reporting Evidence of effective use of self-evaluation (i.e.

monitoring) and reporting to improve the efficiency and effectiveness of activity;

The need for close monitoring was facilitated by the closeness and involvement of the CAPTAC-DR RA in financial supervision.


None evidenced.

Use of RBM to manage activity; RBM was, in effect, used informally through close involvement of the RA and management of STX output deliveries in line with the Project Document. Formally, the use of tools such as Results Chain & logframes for the project would help identify other areas of TA needed to achieve high level outcomes. Armed with this approach CAPTAC-DR could either decide to fill the gaps itself or could bring them to the attention of other donors in the financial sector- such as IADB, World Bank, IMF HQ, FIRST, etc


No evidence of this yet, as the final project output was only recently been completed.

Score for Efficiency 3.3 Good SUSTAINABILITY Sustainability of TA activity Outcomes from intervention will last beyond/continue

after completion of TA/training. Sustainability of the TA project’s results will depend not only on adoption of the methodology and improvement and maintenance of the capacity of the supervisory institutions but on upgrading the methodology and the capacity of the supervisors to deal with new developments in financial markets. The capacity of the supervisors in the region is reasonable but would benefit from improvement. The CCSBSO is well aware of the need to continue upgrading the staff of its member agencies and their supervision in line with improvements in best international practices, to deal with new developments in financial markets. Finally, sustainability will depend on the supervisory agencies’ capacity to retain a high quality staff through reasonable salaries and through legal protection for their activities as indicated in the BCPs, which is part of supervisory agencies self-assessment programs and the World Bank-IMF Financial Sector Assessment Program. The RA for CAPTAC-DR has expressed concern


41 | P a g e

that some of the country supervisory agencies face high staff turnover, which frustrates training initiatives. Maintenance of the full potential outcomes will depend on specific activities, including probably some additional, hands-on-training linked to cross-country Colleges of Supervisors that are being set up legal and regulatory upgrading is also needed in member countries to take full advantage of the TA. A particular example is Panama, where supervisors currently do not cover the activities of holding companies that may include banks, other financial institutions, and even some non-financial activities. Until there is evidence of the needed legal reforms it is hard to assess the prospects for sustainability. Another issue is the integration of Colombian supervisors in CCSBSO after the recent entry. Colombian banks are increasingly important in the CAPTAC-DR financial system. Finally, the outcomes will last but require updating in response to market changes, in order to maintain their quality. Moreover, high salaries in private sector financial institutions represent a continual threat to retaining high quality supervisors in most developing countries.

For TA/training designed to deliver Sustainability, local institutions/ capacities have been strengthened to sustain results beyond the life of the intervention:

The TA strengthened local capacity and institutions to serve as the basis for the information needed for cross-border consolidated financial supervision. The TA introduced the supervisors to these issues and the potential for changes in the markets, but sustainability of financial supervision is a continuous process. The CCSBSO is aware of the need for continual upgrading and will be able to provide upgrading with the support of the IMF and CAPTAC-DR. Additional TA may be required.

o E.g., absorptive capacity improved and developed to sustainable level;

The TA and the associated training improved capacity for the supervision of cross-border consolidated financial institutions, but this capacity will need to be continually upgraded to respond to changes in financial markets. It is not yet clear that all of the CCSBSO supervisors will eventually develop the capacity for upgrading without external TA. A train-the-trainers program has not been launched yet.

o E.g., in-house training capacity built, where relevant.

No in-house training capacity was built; the intention of the TA was solely to develop a methodology that could be used promptly for a risk that had been clearly diagnosed by the CCSBSO.


Financial sustainability is not currently an issue for the supervisory agencies; if it becomes one, they can simply charge the financial institutions for the supervision, as is done in many countries.

Effectiveness in maintenance and use of institutional memory in the intervention.

The six studies and the associated forms for information exchange on corporate governance, solvency, contagion risks, credit risk, liquidity & market risk, and operational risk form the basis for institutional memory. However these assessments need to be continually upgraded over time in response to changes in financial markets.


The TA provides the necessary information base for sustainable, cross-border consolidated supervision of regional conglomerates.

Score for Sustainability 3.0 Good


42 | P a g e

Rating Based on Previous Evaluation Methodology

DAC criteria Sub-criteria Score Weight Weighted

score Weight

Total score

Rating


3.7 60% 2.22


3.9 20% 0.78

Co-ordination with Development Partners 3.2 20% 0.64 3.64 32% 1.16 Excellent Effectiveness Impact - Outcomes: TA 3.2 30% 0.96 Outcomes: Regional capacity building 3.2 30% 0.96 Outputs 3.6 40% 1.44 3.36 28% 0.94 Good Efficiency Process and implementation efficiency 3.4 40% 1.36 Efficient use of resources (human and

financial) and attention to cost effectiveness 3.2 40% 1.28

Monitoring and reporting 2.7 20% 0.54 3.18 22% 0.70 Good Sustainability Sustainability of RTAC/TA activity 3.0 75% 2.25 Contribution to building sustainable regional

TA, implementation capability 3.0 25% 0.75

3.00 18% 0.54 Good TOTAL 3.34 Good


43 | P a g e

4. REGIONAL: OPERATIONAL RISK SUPERVISION

4.1 INTRODUCTION AND SUMMARY OF RESULTS

The above project was selected as a Case Study, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s Inception Report. Operational Risk Supervision has formed an increasingly important element of risk-based supervision (‘RBS’), the evolution from Basel I to Basel III and its adoption by OECD member countries. CAPTAC-DR countries are moving toward international best practices of regulation and supervision, with the overall objective of managing financial sector risks and improving regional financial stability. Supporting that effort, the Center has supported an upgrade of the regulatory norms for operational risk in five of the CAPTAC-DR countries, with Guatemala

and Costa Rica expected to complete revisions of their norms by the end of 2013. The actual supervision of operational risk will probably need further TA for capacity improvement in priority technical areas, which will be based on the completion of the risk events’ survey in the countries. CAPTAC-DR and the IMF-HQ support other areas of RBS through other TA projects and workshops. Overall we consider the project has achieved good results as reflected in our ratings. We have not rated sustainability of the project because it is too early to assess the trend towards the final goal of effective supervision and management of operational risk without other important interventions having already taken place, such as notably long-term capacity building. 4.2 INFORMATION REVIEWED The evaluation of the Operational Risk Supervision project was carried out based on: Review of project-related documentation available at CAPTAC-DR (Briefing Papers, Mission Reports, and BTOs); Interviews of CAPTAC-DR’s Center Coordinator, and RA for the project; Review of outputs and presentations related to the project; Interviews with the Costa Rican and Guatemalan Superintendents of banks and their staff; Interviews with the President of the Central Bank of Costa Rica and Governor of the Central Bank of Guatemala; Review of background information related to consolidated supervision and the website of the CCSBSO; Relevant IMF technical papers; and Meetings with IMF staff, mostly from the Western Hemisphere Department.


44 | P a g e

4.3 OVERVIEW

Operational risk is defined as the “risk of direct or indirect loss resulting from inadequate or failed internal processes, people and systems or from external events” by the Basel Committee on Bank Supervision. Examples to avoid include internal fraud; violation of internal trading rules; hiding trading losses by violating internal rules; illegal market manipulation; fiduciary breaches; failures in carrying out financial processes; failures and frauds from external counterparties; natural disasters that affect the bank directly; and computer hardware and software problems. In OECD country banks, well-known, specific examples of poor management of operational risk include “rogue” traders avoiding internal rules on risk, traders’ hiding their losses, illegal insider trading, and failure to enforce internal lending rules even before the 2008 crisis. The result has often entailed major losses and even failures of banks. In the CAPTAC-DR region there have been some losses from poor management of operational risk but most of the large losses and bank failures involved the owners of the bank and their associated violations of limits on lending exposures and related party lending, areas that are already covered under standard supervision of credit risk and liquidity. Trading losses are less of an issue in CAPTAC-DR countries than in OECD countries, since banks in the CAPTAC-DR region do not usually engage in substantial market trading. The supervision of operational risk management was included in Basel II and Basel III Recommendations for Banking Regulation and the related Core Principles for Bank Supervision used in World Bank-IMF Financial Sector Assessments. The potential effects of operational risks on profits, capital, and in some cases even financial stability are fully recognized. In 2009-2010, three countries in the CAPTAC-DR region adopted regulatory norms on operational risk. However, supervision under these regulations was deemed to need improvement. In FY2011-FY2013, CAPTAC-DR undertook a regional TA project to improve Supervision of Operational Risk Management. The expected results would enable bank supervision in the CAPTAC-DR countries to achieve fuller compliance with Basel II/III, reduce operational risks that could lead to bank losses, and, ultimately, improve financial stability in the region. Moreover, carrying out the improvement in all of the seven countries has the advantage of avoiding regulatory arbitrage that would allow some banks to avoid the regulations. Operational risk management and its supervision are still in their infancy worldwide, despite their inclusion in Basel II and Basel III. Supervision of operational risk also represents a major addition to supervision tasks (so far focused on exposure to and management of credit and liquidity risks). As such it requires assessment of bank systems in many new areas to prevent fraud, IT problems, fiduciary breaches, and to support management’s ability to enforce internal rules among trading staff.


45 | P a g e

4.4 THE RESULTS CHAIN FOR OPERATIONAL RISK SUPERVISION

A sector level Results Chain for addressing operational risk in CAPTAC-DR members would have been ideally developed ex ante, as a project planning tool. An illustrative one is provided above. It is intended to highlight the key technical assistance/reform needs beyond the scope of the project, and the requirements for appropriate sequencing (e.g., diagnostic, followed by an action plan to address issues identified, followed by legislation/regulation updating where necessary, followed by capacity building for implementation). These requirements need to be met in order for the project to achieve its higher level outcomes and impacts, and to identify related risks and assumptions. It is provided on an illustrative basis only as part of the Case Study. An analysis of the Results Chain highlights the following: Overall Objective and Impact This TA initiative seeks to increase the stability and robustness of the financial sector in CAPTAC-DR countries by enabling country regulators to supervise banks’ operational risks in a more effective manner.


46 | P a g e

Diagnosis Operational risk has become an increasingly important issue in the movement from Basel I to Basel III of the best practices of banking regulation and supervision in the OECD countries. Regulation and supervision among the CAPTAC-DR countries could be substantially improved, based on a survey of FSAPs done in the countries and a recent survey of their regulatory and supervisory practices14. These studies did not identify operational risk as a priority though. Only three of the member countries had adopted norms on operational risk in 2009-10 and their implementation could have been improved. Strategy The CCSBSO completed a strategic plan for 2010-2014 with the overall goal of promoting, developing and adapting best international practices of regulation and supervision, which would improve regional financial stability. The CCSBSO Strategic Plan15 focuses on four main areas of development: Enhanced stability; Risk Based Supervision; International Financial Reporting Standards; and Cross-Border Consolidated Financial Supervision.

Operational risk appears within the general objective of moving forward on best international practices of regulation and supervision and, under Initiative 1, standardizing the norms of operational risk across the members. The CAPTAC-DR TA program in financial supervision supports the CCSBSO plan and is endorsed by the CAPTAC-DR Steering Committee. Legislation TA to address regulatory upgrading has been provided by CAPTAC-DR. In principle, no new legislation is required. Activities The work was appropriately sequenced with respect to country coverage, beginning in countries without a regulation on operational risk management, then moving to countries where existing regulation needs improvement. We believe that an operational risk survey/analysis should have been carried out before the drafting of the regulations, rather than the other way around. Nonetheless, CAPTAC-DR has provided high quality TA on new supervisory regulations on operational risk and modification of existing regulations in countries where the regulations already existed. CAPTAC-DR also contributed to the development of training plans for supervisors and staff of banks in operational risk and advised on dissemination. As a next step a CCSBSO analysis should provide data to help supervisors in fine-tuning their work by providing a focus on the more likely risk areas within the region (which will likely differ from other areas of the world). Perhaps this activity should have been undertaken ahead of adopting the regulatory amendments.

14 Delgado and Merza, Development of Financial Supervision and the use of Macro-prudential Measures in Central America, Panama and

Dominican Republic, IMF WP/11/299. 15 CCSBSO website.


47 | P a g e

Outputs Project outputs stem from the activities mentioned above. Key expected outputs related to assessment of needs, survey of probably risk events, sensitization and training both to private and public sector officials, as well as improved operational-risk related regulations. In that regard, operational risk management regulations have been enacted in El Salvador, Honduras, Panama, Nicaragua and Dominican Republic. Superintendents have been trained on this issue. In addition, similar outputs are likely in 2013 in Costa Rica and Guatemala. Outcomes In the short term the project has sought to support publication of new/revised regulations on operational risk in El Salvador, Honduras, Panama, Nicaragua, and the Dominican Republic. The new regulations aim at helping banks recognize the main areas of operational risks. In the medium-term, more effective operational risk supervision should result from the interventions of various providers of TA in the region as well as from the continued efforts of both the CCSBSO and the country-level supervisors. The dissemination of practices and the incidence of actual risk events will also form part of the results expected from the Center’s Consolidated Cross-border Supervision TA. Progress in strengthening operational-risk supervision among other risk areas can be measured periodically by various means including CCSBSO data, IMF surveillance missions, and reports from other donors such as IADB. Supervision of operational risk management is ultimately aimed at reducing banks’ risks in non-lending and funding risk areas. It is still in its infancy worldwide. In many ways It differs from supervision of banks’ liquidity and credit risks, since supervision of operational risk management requires assessment of banks’ systems in many areas to prevent fraud, IT problems, fiduciary breaches and failure to handle client’s assets correctly. Donor Coordination Donor coordination has been limited in this area, given the specific nature of operational risk regulation and supervision and IMF/CAPTAC-DR’s expertise in this area. We were not made aware that other donors or IFIs have addressed operational risk. Other TA from CAPTAC-DR TA in the region has varied from country to country. At a regional level there has been TA on consolidated cross-border supervision that is beginning to be adopted, along with additional support on legislative changes that would be desirable. There has also been training of supervisors and banks staff and development of IFRS. During FY13, there was also TA on liquidity risk for Costa Rica and Guatemala and on RBS of the insurance sector in Honduras. Related regional workshops have included: credit risk (September 2011), basics of Risk Based Supervision (December 2011), stress testing (run by IMF in 2011) and consolidated supervision (April 2012); there was also a regional seminar on “Success Stories” in consolidated supervision. So far there have been no workshops specifically on the topic of operational risk.


48 | P a g e

Pre-Conditions and Risks Operational risks that have not been managed well have led to major losses in OECD countries. For example, there have been major, well publicized losses in large, investment banks from derivatives where the traders were not well managed, for example, recently, in Société Générale, UBS and JP Morgan. There also have been large losses through mismanagement of cash and transactions (MF Global). The earlier costs of similar problems in developed countries have led to changes in Basel Core Principles to reduce operational risk and improve its supervision, including the allocation of capital as provisions against such risks. 4.5 CAPTAC-DR TECHNICAL ASSISTANCE

Background The CAPTAC-DR project in operational risk supervision prioritized supporting the development of a regulatory norm for operational risk. The TA began with El Salvador, Honduras, and Panama in FY 2011, followed by Guatemala, Nicaragua, the Dominican Republic, and Costa Rica. The strategy was that once a regulation had been put in place, the banks and supervisors would be able to compile a list of problems (main event risks) that had occurred in bank operations in the past in order to focus training of supervisors, as well as to develop (i) a training program for the supervised financial institutions, including practical management methodology and examples of implementation in banks and (ii) a dissemination plan for senior bank management in order to convey the importance of operational risk management In the case of the CAPTAC-DR region, improving operational risk regulation and supervision hinges on two key aspects. Firstly, this type of work is substantially different from traditional activities in supervision of credit quality, lending practices, and liquidity. Supervisors therefore require targeted training in order to discharge these new obligations. The number of supervisors may need to be increased as well. Secondly, the assessment of operational risks across the region would most probably show that operational risks are outweighed by traditional risks such as over-exposures in lending, rollovers and forbearance of debt service, and liquidity issues. A large proportion of the well-known instances of operational risk in OECD countries relate to market trading activities and creation of new assets such as Collateralized Debt Obligations (‘CDOs’) and Credit Default Swaps (‘CDSs’). Banks from CAPTAC-DR countries are not substantially exposed to these types of risks. A description of the “success story” in the CAPTAC-DR FY2012 annual report states that:

“The recent international financial crisis highlighted the need for countries to have schemes of financial supervision and regulation that are consistent with international best practices. The Centre’s technical assistance project in the region seeks to improve operational risk management by financial institutions and to establish a monitoring framework of this risk in line with the best international practices. The project started in FY2011 in El Salvador, Honduras and Panama, and in FY2012 was extended to Guatemala, Nicaragua and the Dominican Republic; in Costa Rica, the project will begin in FY2013. The project covers four stages: 1. Development of a draft norm on operational risk in line with international best practices in countries that did not

have one (El Salvador, Guatemala, Honduras and Panama), and recommendations to strengthen it in those countries that already had one (Nicaragua and Dominican Republic).

2. Adoption of a training plan for supervisors in all countries, which seeks to emphasize the medium-term nature of operational risk and the need to "accompany" the supervised entity for long periods of time.


49 | P a g e

3. Adoption of a training plan for risk areas of supervised financial institutions, in order to provide them with a practical management methodology and examples of implementation in banks in the region.

4. Adoption of a dissemination scheme aimed at the senior management of financial institutions in order to convey the importance of operations risk management, contribution to the financial performance of a company that can implement it correctly and the importance of their support.

One of the most important challenges in implementing the project was to achieve regional consensus on the importance of operations risk management. In some countries, this required modifying the supervisor’s strategic plans in order to prioritize implementation.”

Activities The TA has also helped in the development of training plans for supervisors by the Superintendence and staff at the banks. This work has also supported the dissemination of information on the importance of operational risk to senior management in the banks. The revision of the regulations on operational risk in Costa Rica and Guatemala are expected to take place in 2013. So far, the process has been delayed because of the preeminence of other activities in these countries’ Superintendencies; for example, a revised bank law in Guatemala. The TA training includes workshops and training plans for supervisors and banks' staff and dissemination to senior managers of banks on the importance of operational risk. However, it seems likely that substantial additional training will be needed for supervisors and banks given the substantial difference between supervision of operational risks and standard supervision. The project’s training plans for banks’ staff and the seminars for senior management will need to be carried out to support continuation of the outcomes of the project. The source of substantial longer-term capacity building has not been identified in the reports available to us. Outputs Operational risk management regulations have been enacted in El Salvador, Honduras, Panama, Nicaragua and Dominican Republic. Superintendents have been trained on this issue. In addition, similar outputs are likely in 2013 in Costa Rica and Guatemala. The TA on operational risk regulations remains to be completed in Costa Rica in 2013 and 4.5 weeks of RA time has been built into the work plan for that year. No other specific TA is planned going forward but it could be encompassed in some more general areas: “assessing the current status of RBS” in Honduras and “Road map for Basel III” in Nicaragua. Outcomes The Project’s short-run outcomes were achieved. They relate broadly to regulations and training plans. The longer-run outcome of reducing banks’ risks will depend on further training of supervisors, and may be limited as a some of the major objectives of operational risk supervision in Basel II/III — enforcement of banks’ internal trading rules to avoid rogue traders and illegal insider trading – is related to banks’ trading in financial markets, which is not a major activity of Central American banks.


50 | P a g e

Overall the project’s achievements to date are relatively limited. Whilst CCSBSO is an active partner for CAPTAC-DR it would be interesting to know how they see the task of building sufficient supervisory and sector capacity in the future, especially as this topic is highly complex.

Key Outcomes




Supervisors become aware of key operational risks in their/regional financial institutions, through surveys/analyses.

Compliance with Basel principles Decreased vulnerability of the financial system (operational risk events, magnitude of reported losses, banking supervision reports, etc.)

Strategic plan accepted Effective supervision of operational risk in each country as evidenced by CCSBSO reports and other independent surveillance (e.g. FSAP. BCP self-assessments)

Regulations/ amendments (Costa Rica and Guatemala) for operational risk adopted

Effective operational risk management in banks (bank supervision reports)

Achievements The project’s main achievement is to have solidly placed the issue of operational risk supervision in the regional and country agendas. In fact this was no easy task as urgent issues arising from the global financial crisis and its aftermath capture the attention of the regulators. Furthermore, CAPTAC-DR provided quality advice on the formulation of the operational risk regulations, as well as on the organization of specialized training workshops for the supervisors. Five countries have already completed regulations on operational risk and the other two are likely to complete them in 2103, thereby helping with compliance with Basel II/III principles. Furthermore, it seems very likely that Costa Rica and Guatemala will follow during the first half of 2013 calendar year Remaining Challenges The key challenge ahead revolves around attaining medium term outcomes, despite the good progress with short term objectives. Because trading-related risk appears less important in CAPTAC-DR countries than in their OECD counterparts, extrapolations do not seem appropriate. As the project has helped put in place a sound regulatory framework, substantial training will be required for supervisors to implement them appropriately. The project’s training plans for banks’ staff and the seminars for senior management will need to be carried out to support continuation of the outcomes of the project. Experience with problems in Central American and Mexican banks suggests that the major types of risk, are related to foreign developments or to owners’ behavior leading to lending to related-party and excessive exposures, and money laundering, all of which should be covered under standard supervision and AML/CTF activities by supervisors. Substantial capacity building will be needed to achieve effective supervision in this potentially complex area together with possible organization structure change in the supervisors’ office as well as controls in banks themselves.


51 | P a g e

4.6 PRINCIPAL CONCLUSIONS Relevance is rated Good (2.8) because it is important for CAPTAC-DR banking regulators to understand the implications of this new area of work which broadly falls within the objective of achieving ‘improved legal and regulatory framework for bank supervision.’ as stated in the Program Document. The Center, together with the IMF, constitutes the ideal partner for this work. Nevertheless, this particular technical area is less relevant to regional banks than other parts of the world where banks engage in active securities market trading activities (e.g. France, UK, etc.). Moreover, we believe, that regulations should have been sequenced after identifying specific operational risks in the financial institutions, not the other way around. A solid assessment is also made of the project’s effectiveness. Its deployment of a range of outputs of high technical quality warrants a rating as Good (3.5). In addition, short-term outcomes have been achieved though medium- to long-term outcomes are likely to be far more complex. A solid 3.0 points (Good) is assigned to outcomes. Efficiency is also considered Good (3.0), as activities have been carried out on a timely basis, deploying technically solid professionals. As the project is still in its early stages, its sustainability could not be properly assessed. The development of in-house, national, and regional capacities is considered critical aspects to watch, as project implementation continues. 4.7 KEY FINDINGS AND RECOMMENDATIONS Findings 1. The time dedicated to training via workshops and “hand-holding” seemed low given the complexity and depth of the

topics. 2. The TA has been well received and five of the seven countries have already adopted new regulations and it seems

very likely that Costa Rica and Guatemala will follow during the first half of 2013 calendar year. 3. It seems unlikely that other donors would have been willing to fund this work. The TA required is too small for

multilateral financial institutions in the region and too specialised for most bi-lateral donors. The CAPTAC-DR intervention was therefore essential to stimulate progress in this area.

4. There is room for improvement in results-based management although this particular project was focused. A results chain would helped highlight all the interventions needed (even if not in CAPTAC-DR’s TA plan) allowing perhaps a better understanding of all the implementation risks to achieve the final goal of effective operational risk management and supervision.

5. The logical frameworks are a promising development but need to better articulate outcomes both short and medium-term, together with measurable indicators (whether or not the measures are self-assessed or independent).

6. Doubts may linger on whether the choice of this particular project was fully demand driven. Operational risk is a complex area, which is relatively more relevant to OECD banks.

7. We appreciate that CAPTAC-DR has also been delivering TA projects in other areas of supervision such as consolidated cross-border, credit, and liquidity, and also for institutions other than banks, such as insurance and pensions. Most of these have been so far non-regional (except consolidated cross-border supervision) and we believe that these areas are far more relevant to the region at this stage of development. Operational risk would have been a somewhat longer- term priority given that RBS generally still needs considerable strengthening.


52 | P a g e

Recommendations 1. Project design would benefit by building in the entire context of interventions required to reach the goal of effective

operational risk management, thus shedding light on other activities needed to reach that goal. The Results Chain approach facilitates this analysis. Interventions focusing only on the regulations and limited supervisor training may not deliver the final objective, not only because of capacity constraints of the supervisors but also a similar limitation amongst the banks tasked with identifying their particular risks and managing them.

2. The danger of only delivering pieces of TA is that achievement of the high level outcomes fails because of lack of support in other key areas. As a consequence we believe the overall intervention needs should be assessed and if CAPTAC-DR is not set up for covering all of them at least the assessment might stimulate using CAPTAC-DR as a catalyst to bring other support in.

3. Attention should be paid to sequencing (e.g., conducting risk surveys before drafting regulations) and addressing further issues and gaps, which often may be highlighted by initial reforms. CAPTAC-DR could then better address the gaps and perhaps either plan to cover them itself, with IMF headquarters support and/or by including other donors. There is a risk that the good results achieved on developing the regulations could be prejudiced by the lack of long-term capacity building at both supervisor and sector level in some countries.


53 | P a g e

4.8 ANNEX 4.8.1 SUGGESTED LOGICAL FR AMEW ORK

Inputs / Activities

(CAPTAC-DR Only)






Operational Risk Supervision


Activities: Missions; backstopping; workshop; mentoring; training.


b) TA plans (including training).

c) Staff trained.

d) Senior staff trained.

e) Bank executives sensitized.

f) Risk events survey.

g) New/amended operational risk management regulations.

Supervisors aware of key operational risks.

Strategic plan accepted.

Regulations / amendments for operational risk adopted.

Compliance with Basel principles

Effective supervision of operational risk in each country

Effective operational risk management in banks

Decreased vulnerability of the financial system.

Indicators: Frequency of discussions operational

risk issues at CCSBSO or individual superintendence meetings

Opinion of CCSBSO, superintendence & bank staff & management

Work adheres to strategic plan Regulations published Expert opinion from IMF / CAPTAC-DR

Indicators: Operational risk

events Magnitude of

reported losses, banking

Expert opinion on operational risk management.

Sources of verification

CAPTAC-DR reports (Aide memories, BTOs). Review of risk event survey. Workshop reports. Counterpart websites

CCSBSO reports Independent surveillance (e.g. FSAP. BCP self-assessments) Bank supervision reports Monitoring and other CAPTAC-DR reports. Interviews with of superintendents, bank executives, etc. Interview with experts. Official bulleting

Major risks & assump-tions


Counterparts willing to implement recommendations.

Sufficient resources available at counterpart.

Political stability & support. Sensible macroeconomic policy. Staff with the appropriate qualifications trained and retained. IT systems in place and functioning correctly. Effective legal systems and overall institutional infrastructure. Sufficient resources available. Absence of substantial external shocks.


54 | P a g e

4.8.2 EVAL UATION K EQS AND R ATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities


Activity is fully consistent with the Program Document, CAPTAC-DR’s Overall and Area of Action Strategies, and the work plan endorsed by the CAPTAC-DR Steering Committee.



In line with overall country objectives.


The operational risk supervision activity is related to Basel II and III, which include operational risk as one of the Basel Core Principles that are evaluated.


National supervisors have demonstrated their interest by issuing regulations on operational risk in five of the CAPTAC-DR countries; the other two countries are expected to complete their regulation in 2013.


The activity was appropriately sequenced with respect to country coverage, beginning in countries without a regulation on operational risk management, then moving to countries where there is a regulation that needs improvement. We believe that an operational risk survey/analysis should have been carried out before the drafting of the regulations, rather than the other way around.


Supervision of operational risk management is ultimately aimed at reducing banks’ risks in non-lending and funding risk areas. It is still in its infancy worldwide. In many ways It differs from supervision of banks’ liquidity and credit risks, since supervision of operational risk management requires assessment of banks’ systems in many areas to prevent fraud, IT problems, fiduciary breaches and failure to handle client’s assets correctly. Such assessments are similar to auditors’ functions. Moreover, major components of operational risk in OECD banks — trading that violates bank norms or uses insider information — are not widely present in Central American banks, because they do not engage in substantial asset trading, which is typically done through brokerage houses and non-bank intermediaries.

Consistency with IMF Headquarter/other activities Whether activity is appropriately focused in terms of

subject area, taking into account the IMF’s expertise and integration with HQ and other Fund activities (e.g. other training and TA delivered including through the Topical Trust Funds).

The activity is consistent with IMF expertise and activities. Operational risk is part of Basel II and III and is evaluated as part of the Basel Core Principles in all Financial Sector Assessments conducted by the IMF-World Bank. When operational risk regulations are not in place, the FSAP recommends such regulations. For example, in the case of Costa Rica, the TA follows up on improving a regulation on operational risk that was put in place as recommended by the 2007 FSAP.

Coordination with Development Partners Whether the intervention has been effectively

coordinated with and complements the work of other development partners.

Given the link between the FSAPs and financial supervision, other development partners have generally delegated activities related to financial supervision, such as operational risk supervision, to the IMF.

Context of donor/TA landscape of IMF TA Why did the TA recipient choose the CAPTAC-DR as the

TA provider as opposed to other donors? CAPTAC-DR and the IMF were chosen as the TA provider given their experience with financial supervision, as exemplified in FSAPs and Article IVs.

How do workshop participants compare those provided by the CAPTAC-DR with workshops from other donors?

Ratings of the CAPTAC-DR workshops were good. No comparable specialized workshops have been given by other donors.

Score for Relevance 2.8 Good EFFECTIVENESS Impact The interventions expected Impact achieved or likely to

be achieved (to the extent defined in the activity’s logframe).

No logframes were used by CAPTAC-DR at the time when the activity began, FY2011. Impact is likely to be a deeper and more stable banking sector, supporting economic growth. It is too early to assess the project’s likely contribution to this goal.


55 | P a g e

Key Evaluation Questions Comments

Outcomes Intervention’s Outcomes achieved or likely to be

achieved (as defined in its log frame16): Short, medium and long term Outcomes.

Log frames had not been fully adopted by CAPTAC-DR at the time of project origination. The Project’s short-run outcomes were achieved. The longer-run outcome of reducing banks’ risks will be depend on further training of supervisors, and may be limited as a some of the major objectives of operational risk supervision in Basel II/III — enforcement of banks’ internal trading rules to avoid rogue traders and illegal insider trading – is related to banks’ trading in financial markets, which is not a major activity of Central American banks. Experience with problems in Central American and Mexican banks suggests that the major types of risk, are related to foreign developments or to owners’ behavior leading to lending to related-party and excessive exposures, and money laundering, all of which should be covered under standard supervision and AML/CTF activities by supervisors. Substantial capacity building will be needed to achieve effective supervision in this potentially complex area together with possible organization structure change in the supervisors’ office as well as controls in banks themselves.


CAPTAC-DR’s contribution to the basic development of supervision of operational risk management has been important, through its support for the new regulations in the region. CAPTAC-DR provided good advice on drafts of the operational risk regulations and workshops for the supervisors. As noted, five countries have already completed regulations on operational risk and the other two are likely to complete them in 2103, thereby helping with compliance with Basel II/III principles. The likely contribution to the core functions of the supervisors and institution-building for the future is difficult to predict, as trading risks are not a generally a major issue in Central American banks and supervision of operational risk differs significantly from standard supervision of liquidity and credit risks. For example, supervision of operational risk involves evaluation of bank systems such as systems for managing cash, counterparty risk, fiduciary risk, computer security and backup risks. These areas are typically functions of auditors and have not been major functions of supervisors in Central America. Supervisory agencies also have noted the difficulty in obtaining additional training in supervision of operational risk--such programs are supplied in the region privately by accounting firms but are expensive.

Score for Outcomes Good 3.0 Outputs Intervention’s Outputs achieved or likely to be achieved

(as defined in its log frame), including: o Timeliness of reports/workshop presentations; o Quality of reports/workshop presentations; o Appropriateness/applicability of advice, given


Outcomes.

Operational risk management regulations have been published/ updated in El Salvador, Honduras, Panama, Nicaragua and Dominican Republic, training workshops provided for superintendents, and similar outputs are likely in 2013 in Costa Rica and Guatemala. The quality of the CAPTAC-DR comments and the workshops has been good, and timeliness of the project was satisfactory. The advice and workshops were satisfactory given the capacity of the supervisory agencies in Central America. As noted above, supervisory agencies in Central America have not been involved in supervising operational risk and have commented on the difficulty of obtaining additional training in operational risk to maintain and improve capacity. Moreover, Central American banks do not yet have major risks related to market trading (although this could develop as government bond markets in the region become more complicated). Banks’ major risks in the past have often been related to the ownership of the banks, not the banks’ and systems.

Score for Outputs 3.5 Good EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; TA workshops have been efficiently carried out. Timeliness in executing the activity; CAPTAC-DR support has been timely; the new regulations have been put in 16 We will review the log frames for project proposals where available and/or where needed, suggest refinements and indicators to measure performance against.


56 | P a g e

Key Evaluation Questions Comments place in a relatively short time in five countries and are progressing toward likely announcement in the other two countries in 2013, closely following the schedule in the TA’s work program.


IMF management of the activity, which was carried out entirely by the RA was appropriate.

Appropriateness of selection of counterpart/workshop participants (where relevant);

Workshop participants were appropriately chosen— i.e. the supervisors who would implement, and supervise compliance with, the new regulations.


Quality and timeliness of management and backstopping was appropriate, the RA participated substantially in the implementation of the TA.


Planning and execution of the TA was efficient, as indicated by the rapid issuance of new/updated regulations on operational risk supervision in five countries.



The staffing of the TA, entirely delivered by the RA was appropriate, especially given the general lack of experienced supervisors in this area of risk.


CAPTAC-DR was responsible for all of the work on the TA, which was appropriate given its location and the need to carry out the TA in the seven countries in the region.

Appropriateness of degree and nature of use of regional/ local expertise;

Regional expertise was available and well qualified in the person of the RA.


Expenditures appear to be in line with the project’s budget although reports make it difficult to reconcile planned versus actual cost at project as opposed to topical level.


The activity appears cost effective relative to likely other TA delivery modes, reflecting CAPTAC-DR’s location close to the country it serves.


The project did not last long and was of limited scope. Efficiency gains unlikely to be feasible.

Monitoring and reporting Evidence of effective use of self-evaluation (i.e.

monitoring) and reporting to improve the efficiency and effectiveness of activity;

Given that the TA was delivered entirely by the RA, monitoring was probably effective since it involved no other inputs.


TAIMS not used.


Formal RBM framework existed in the form of a logframe for the last two fiscal years country by country, including the operational risk project. There was no evidence of its use as a monitoring tool in the form of reports. The structure of the logframe could be improved but it is certainly a good start towards application of RBM.

Use of RBM to manage activity; See above Incorporation of lessons learnt in project design and

implementation. NA, no projects were comparable.

Score for Efficiency 3.0 Good SUSTAINABILITY Sustainability of TA activity Outcomes from intervention will last beyond/continue

after completion of TA/training. The TA training includes workshops and training plans for supervisors and banks” staff and dissemination to senior managers of banks on the importance of operational risk. However, it seems likely that substantial additional training will be needed for supervisors and banks given the substantial difference between supervision of operational risks and standard supervision. The project’s training plans for banks’ staff and the seminars for senior management will need to be carried out to support continuation of the outcomes of the project. The source of substantial longer-term capacity building has not been identified in the reports available to us.


As noted, plans for staff and managers of banks will need to be carried out and additional training will probably be needed for supervisors. Accounting firms in the region provide courses in operational risk, but they are expensive.


Absorptive capacity has been improved.


57 | P a g e

Key Evaluation Questions Comments o E.g., in-house training capacity built, where

relevant. In-house training capacity was not an objective of the TA project and not consistent with the short length and specialized nature of the project.


There is no major issue of financial sustainability in bank supervision, if necessary; bank supervision can be paid for by charging fees to the banks, as is done in other countries.

Effectiveness in maintenance and use of institutional memory in the intervention.

Regulations on operational risk have been put in place in the region and workshops provided for their implementation. Institutional memory will depend on these outputs of the project and implementation of plans for bank staff and further training of supervisors.


Satisfactory, given the short length of the project and its focussed intention.

Score for Sustainability Not Demonstrated Rating Based on Previous Evaluation Methodology

DAC criteria Sub-criteria Score Weight Weighted

Score Weight

Total score

Rating


3.1 60% 1.86


3.3 20% 0.66

Co-ordination with Development Partners 3.0 20% 0.60 [39%] 3.12 32% [1.22] Good Effectiveness Impact - Outcomes: TA 3.0 30% 0.9 Outcomes: Regional capacity building 3.0 30% 0.9 Outputs 3.5 40% 1.4 [34%] 3.2 28% [1.09] Good Efficiency Process and implementation efficiency 3.1 40% 1.24 Efficient use of resources (human and


Monitoring and reporting 2.7 20% 0.54 [27%] 2.98 22% [0.80] Good Sustainability Sustainability of TA activity ND 75% Contribution to building sustainable regional

TA, implementation capability ND 25% [0%]

18%

TOTAL 3.11 Good Note: Figures in square brackets represent the revised weights, as sustainability could not be measured.


58 | P a g e

5. EL SALVADOR: EXTERNAL SECTOR STATISTICS 5.1 INTRODUCTION AND SUMMARY OF RESULTS The above project was selected as a Case Study, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s Inception Report. Since its inception, the CAPTAC-DR has supported external sector statistics through a dedicated regional harmonization program, which covered all beneficiary countries including El Salvador. The Center has provided solid technical advice to El Salvador in an area of particularly high relevance, in light of the relative importance of remittances, exports, and imports for the national economy. The Salvadorian Government, and more specifically the country’s central bank (Banco Central de Reserva de El Salvador - ‘BCRES’), has taken up ownership of the TA and has pushed forward the reform agenda. Improvement in the way statistics are compiled can already be noticed. In fact, the project is on its way to achieve most of its intended outcomes. Furthermore, prospects for sustainability of this intervention are high. In light of the above, this project has been assigned an unusually high score. This section takes stock of the work already carried out. In addition, this assessment presents a number of recommendations seeking both to consolidate the gains already achieved, and to make the most out of the remaining years of CAPTAC-DR technical support to El Salvador. 5.2 INFORMATION REVIEWED The evaluation was carried out primarily based on: Document and data analysis: The evaluators reviewed and analyzed the materials at the disposal of El Salvador’s

Central Bank, along with a series of CAPTAC-DR reports (briefing papers, aide memoires, etc.) Surveys: The evaluators produced a structured questionnaire, which was posted online. Recipients of CAPTAC-DR’s

TA were invited to provide their views and recommendations on the Center’s work. Interviews and meetings: The evaluators conducted a number of semi-structured interviews and meetings in El

Salvador.

5.3 OVERVIEW 17 El Salvador has a population exceeding 6 million and is classified as a lower middle-income country. In fact, per capita GDP at current prices reached US$6,640 in 2011. The global financial crisis of late 2008 hit El Salvador hard, as both remittances and exports fell. As a result, real GDP contracted by 3.1 percent. The economy began a tepid recovery in 2010 fuelled by increased export and remittances. Nevertheless, the IMF World Economic outlook database does not forecast GDP growth in El Salvador to rise above 3 per cent per year until after 2017.

17 The World Bank constitutes the source for all figures in this sub-section.


59 | P a g e

As the chart opposite depicts, the country is highly dependent on remittances, which accounted for 16 per cent of GDP in 2011. In contrast, for neighboring Costa Rica, this figure was about one percent. Exports of goods and services also constitute an important component of GPD in El Salvador, accounting for 28 percent. The USA constitutes by far the most important source of the remittances, and also the biggest destination of Salvadorian exports. For this reason, El Salvador is highly dependent on the US economy, as well as on its immigration policies. Finally, El Salvador runs a sizeable trade deficit, with imports reaching 47 per cent of GDP in 2011. In light of the importance of imports, exports and remittances for the overall economy of El Salvador, it is vital to produce timely and reliable external sectors statistics. For this reason, the BCRES has developed a relatively advanced set of statistics in the region and is keen to continue improving this work. Given the IMF’s technical knowhow in this area, CAPTAC-DR’s TA to El Salvador is both welcome and relevant. 5.4 THE RESULTS CHAIN FOR EXTERNAL STATISTICS IN EL SALVADOR The sector level Results Chain for external sector statistics in El Salvador would have been ideally developed ex ante, as a project planning tool. This approach seeks to highlight the key TA/reform needs beyond the project, and the requirements for appropriate sequencing, that need to be met in order for the project to achieve its higher level outcomes and impacts, and to identify related risks and assumptions. The indicative Results Chain in the next page is provided on an illustrative basis and only as part of the case study. Analysis of the Result Chains graph indicates that work was undertaken in a broadly logical sequence (typically, diagnostics, followed by strategy/action plans, before necessary legislation/regulation and then capacity building to implement changes). The section below provides an overview of the key components of this results chain. Overall Objective and Impact The strategic objective of this initiative is to support El Salvador in compiling both comprehensive and consistent statistics of the country’s external sector. A key concern is to ensure compatibility across the region and to apply international practices, standards, and basic principles of compilation and dissemination. This project ultimately seeks to provide quality information to Salvadorian decision-makers in order for them to steer economic policy, against the backdrop of high sensitivity to shocks affecting either exports or remittances.


60 | P a g e

Diagnosis The diagnostic efforts trace their roots to an original report on the observance of standards and codes. This report was produced by a technical mission that visited El Salvador in December 2004, several years before CAPTAC-DR’s inception. An additional mission from the IMF took place in 2007. The first diagnostic mission from the Center took place in May 2010. This diagnostic is updated with every follow up mission from CAPTAC-DR. Strategy On the basis of the initial assessment above, a concrete plan of action was set. Progress was monitored against this plan. The country’s central bank has high absorptive capacity. For this reason, the project strategy has focused on providing advice at regular intervals, while steering the timely implementation of the project’s work plan. When technical difficulties are encountered, CAPTAC-DR experts support their counterparts at BCRES in order to address them. This project is carried out within the context of a CAPTAC-DR implemented regional initiative, the Regional Program for the Harmonization of External Sector Statistics (‘PRAESE’ in Spanish). The first phase of this initiative began in late 2009. This project is currently on its second phase and seeks to improve capacity, quality and harmonization throughout the region.


61 | P a g e

Legislation In 2011, the Salvadorian parliament enacted a new organic law for BCRES. CAPTAC-DR has supported this process as well as the formulation of rules and regulations of article 64 of this law, which sets forth mechanisms for gathering the necessary information from the private sector. Activities, outputs and outcomes The project’s activities center on regular technical missions to support BCRES’s implementation of the project. Furthermore, this advisory work is complemented by arranging the participation of technical officers from El Salvador to a series of regional workshops covering key aspects relating to the compilation of external sector statistics. The project’s outputs mostly consist in trained technical staff and new compilation techniques. A third output is a series of recommendations to allow this work to be carried out in a more conducive environment. These outputs contribute towards project outcomes, which basically consist in improved capacity at BCRES to produce sound external sector statistics while also converging towards closer regional harmonization. Donor Coordination The project is mostly self-contained, CAPTAC-DR being the principal provider of TA to the BCRES. We have seen no other information indicating the involvement of other donors in this particular technical area, in which the IMF is the global standards setter. Pre-conditions / Risks The Results Chain typically assumes certain preconditions that may need to be met in order to attain higher-level

impact objectives. We are illustrating below the type of pre-conditions which could apply without knowing the entire context of the particular project in El Salvador. More concretely, achieving the project’s goals could require the following external preconditions at the political, policy, and macroeconomic levels: political stability and consensus on need for statistics, sound macroeconomic policies, and absence of external economic shocks.

In addition, preconditions relating to systems & infrastructure also affect the project’s chances of success, namely: Appropriate infrastructure to allow collection and analysis of reliable source data including selection and management

of relevant surveys. Effective legal system, particularly as regards the BCRES’ organic law and its related regulations. Strong central bank: with good staff technical capacity and qualifications.


62 | P a g e

5.5 CAPTAC-DR TECHNICAL ASSISTANCE Background The importance of external sector statistics for all member countries has been recognized since CAPTAC-DR’s inception. The Center therefore took up the implementation of Regional External Sector Statistics Harmonization Project (‘PRAESE’) in which El Salvador takes part. The first phase of this initiative finished in September 2001 and aimed at developing and disseminating external sector statistics, jointly by all members. The technical areas covered were (i) quarterly balance of payments (‘BoP’), the international investment position (‘IIP’) and external debt statistics; (ii) monthly statistics for the Template on International Reserves and Foreign Currency Liquidity (Reserves Template ‖); as well as (iii) metadata for the BoP and IIP. PRAESE’s second phase, currently under implementation, seeks to ensure that consistent and comprehensive external sector statistics are compiled by all member countries. During this phase, technical assistance focused on helping member countries to develop harmonized external sector products and to improve BoP statistics compilation and their IIP in line with the sixth edition of the Balance of Payments and International Investment Position Manual (‘BPM6’). CAPTAC-DR paid a particular emphasis in reviewing BoP service statistics compilation for the items highlighted by the BPM6. Other work included monitoring the implementation of TA recommendations in the above-mentioned areas. Activities The project’s activities center around a series of advisory missions. They have provided technical inputs seeking to support the implementation of the project’s work plan in areas such as BoP and IIP within the overall context of the BPM6. Key technical visits to El Salvador have included the following: April 26-May 7, 2010 – advisory mission on BoP and IIP; providing general advice and refining the project’s overall

work plan; May 17-21, 2010 – concentrating on IIP statistics, mostly by seeking to improve coverage, valuation and compilation

techniques; November 8-12, 2010 – on statistics for the service sector emphasizing coverage and classification, as well as on the

transition from BPM5 to BPM6; March 28-April 8, 2011 – this mission involved two experts and focused on estimation techniques for international

trade in services and goods, personal portfolio investment transfers, foreign direct investment, external debt and reserves. The mission also provided technical inputs specific to the BPM6.

November 14-22, 2011 – with emphasis on statistics of the services sector and international personal transfers (key areas covered included information sources, methodology, registration, and dissemination of information on services and personal transfers. In addition the mission provided advice on specific difficulties relating to the adoption of BPM6 standards;

March 15-22, 2012 – focusing on the financial account of the balance of payments, and on IIP; and August 27-September 7 2012 – providing advice on BoP and IIP statistics (mostly reserves sheet and external debt

statistics). In addition, a series of regional workshops was also organized, with assiduous attendance from Salvadorian technical officers. These seminars focused on technical areas including: (i) regional harmonization of statistics; (ii) remittances; (iii) the service sector; (iv) foreign direct investment; (v) external debt statistics; (vi) reserves; as well as (vii) issues relating to statistics for the BoP and for IIP.


63 | P a g e

Outputs The project has trained 11 technical officers from El Salvador in the range of technical areas mentioned above. In addition, enhanced systems have been adopted at BCRES for the compilation of external sector statistics. On the one hand, the central bank has been legally empowered to gather the necessary data. On the other, internal capacity has been strengthened, particularly as regards the standards set by BPM6. More specifically, the Central Bank has introduced annual and quarterly surveys for services and non-financial public sector, institutionalized agreements between private and public sector agencies and better systems for quarterly reconciliation among the balance-of-payments, IIP and external debt statistics. The Bank has also expanded the samples to improve coverage and accuracy. Outcomes In the short term the project contributes towards enhanced qualifications and training by BCRES staff as well as to the inception of new methodologies for compiling external sector statistics. In the medium term, expected outcomes include sustainable capacity building for improving the methodology so as to attain (i) greater accuracy and internal consistency among the balance-of-payments, IIP and external debt statistics; (ii) greater detail in the statistics complied, as well as (iii) wider dissemination through domestic and IMF’s web pages. In the long term, the objective is to produce a set of fully harmonized more reliable and more complete data on BoP, IIP, Foreign Debt, reserves and metadata for all member countries. We have constructed the project’s principal expected outcomes as summarized in the table below:18

Outcomes Short-term

(Up to 1 year) Medium-term


(4-10 years) Improvement action plan for external sector statistics adopted.

BCRES disseminates harmonized BoP statistics, IIP, reserves and foreign debt increasingly based on BPM6 guidelines.

BCRES has built up sustainable capacity to produce harmonized, reliable and complete data on BOP, IIP, Foreign Debt, reserves and metadata as per BPM6 standards.

Improvements in the (i) adjustments to the transactions register (incl. foreign assets, goods, & services); (ii) reclassifications among accounts (mainly between income & services, and within the financial account).

BCRES has (i) put in place improved systems for compilation in services & financial transactions (non-financial private sector); (ii) expanded coverage of external sector statistics; (iii) improved calculation of the BoP, IIP, and int’l reserves template by including new headings and valuing transactions & balances; and (iv) improved consistency of the BoP, IIP & external debt statistics.

Legal framework in place for BCRES to gather the necessary information to produce external sector statistics.

18 These were not produced at the outset of the project. They have been prepared ex post by the Evaluators to provide benchmarks for assessing outcomes.


64 | P a g e

Achievements The short-term outcomes have been largely attained and there is promising progress towards medium-term outcomes. More specifically, the legal framework is now in place and over 90 TA recommendations have been implemented, of which 25 were considered top priority. Most of these recommendations deal with improvements in the database and of the data analysis. About 10 percent of all recommendations concern data reclassification, institutional strengthening, software procurement, public relations, inter-agency cooperation, and other miscellaneous topics. External sector statistics in El Salvador have made significant progress in terms of quality, coverage, methodology and compilation practices. They are steadily converging towards more harmonized practices, as well as a higher degree of internal consistency. The fact that CAPTAC-DR’s TA is embedded in the multi-country PRAESE program has contributed significantly in moving decisively towards regional harmonization. Remaining Challenges El Salvador has not attained full compliance with the BPM6 standard. Reaching that goal constitutes the main challenge ahead. Also, achieving full harmonization at the regional level is probably a very remote prospect. Yet, there is still room for improvement in this regard Sustainability is often a key concern with respect to the stability of a well trained staff and the institutional capacity to provide quality training and guidance to staff. Sustainability of the TA is likely to be high in El Salvador because central bank staff is well remunerated and staff turnover is low. The bank routinely rotates staff, which is a powerful mechanism to ensure that the skill base is broadly distributed. As such, transitions tend to be smooth, when a particular officer leaves BCRES. Progress in this project has been strong. Almost two thirds of the priority recommendations have been implemented although a significant number of technical recommendations (13 of the priority ones plus a further eight lower priority ones) remain to be implemented. 5.6 PRINCIPAL CONCLUSIONS The Annex presents our detailed responses to the KEQs and the basis of arriving at our ratings for the various DAC criteria. In summary: We rate Relevance of the project as Excellent (score 3.6), reflecting its close alignment with CAPTAC-DR’s and national objectives. Strong ownership has been demonstrated by BCRES With respect to Effectiveness, our rating at the Outcomes level is also at the top level of the good category (3.4) reflecting the probability of attaining the project’s outcomes. About two thirds of the priority recommendations have already been implemented; we consider that the progress to full implementation is therefore strong enough to warrant a high score. The high quality of advice and the seamless flow of technical missions and regional workshops are reflected in a rating of Excellent (3.6) for Outputs. Efficiency is again rated as Good (3.2), reflecting solid organization. There is still room for improvement as regards cost efficiency in running and organization of workshops.


65 | P a g e

Sustainability is rated at the lower range of the Excellent category (3.6), as the majority of the advice delivered so far is likely to be retained within BCRES. Also, the new techniques introduced do not require additional resources. Their costs can therefore be met from existing sources. As project implementation continues, prospects for sustainability are likely to improve even further. 5.7 KEY FINDINGS AND RECOMMENDATIONS Findings

1. This type of project is particularly well suited to CAPTAC-DR’s delivery modality and to the IMF’s strengths. 2. The cost of workshops in the corresponding budget line underestimates the true cost of training as certain costs,

such as RA and administrative time, are not captured in this budget line. 3. BCRES provides good salaries to its staff and has the means to ensure long-term sustainability of this intervention. 4. The format of the project summary of recommendations has evolved through time and according to the expert who is

making them. Some experts, for example, included a column indicating the priority of their recommendations, whereas others do not. In addition, some experts presented the recommendations as discrete actions, such as “calculate accrued interests on the external public debt and include them in the balance of payments.” Others provided lengthier recommendations which included several actions. This lack of uniformity made it difficult to follow the progress of particular, discrete actions.

5. In addition, some recommendations were not implemented in time and, consequently, were repeated in subsequent reports, making it difficult, although not impossible, to distinguish between new and repeated recommendations. The problem was compounded by the fact that sometimes the phrasing changed from report to report.

Recommendations

1. This TA should also undertake hands-on learning based on real cases. 2. Development of in-house training does constitute a priority for BCRES. Ideally procedures should be recorded in a

comprehensive manual and an induction pack for new staff should be developed. 3. It is suggested to increase the duration of workshops by one more day. There may be room for more cost efficiency

in organizing these training events. 4. RBM should be used more effectively as a project design and monitoring tool. 5. Given the large number of recommendations, it is suggested to create a monitoring tool, such as a database, based

on RBM concepts to follow up the status of each item. It is recommended to indicate whether a recommendation commands low, medium or high priority. Recommendations should be both detailed and consistent through time and across the experts who make them.


66 | P a g e

5.8 ANNEX 5.8.1 SUGGESTED L OGICAL FR AMEW ORK A logical framework analysis / RBM has not been applied to this particular initiative, though many of its elements have been implicit. In line with other sections of this review, the table in the following page proposes a sketch of the basic suggested contents:

Project

Inputs / Activities (CAPTAC-

DR)






External Sector Statistics in El Salvador.


Activities: Missions; backstopping workshops; mentoring; training.

a) Diagnostic & action plan

b) Technical staff trained.

c) Enabling BCRES organic law & regulations enacted.

d) Adjustments/reclassifications proposed.

e) Surveys for services & non-financial private sector.

f) Information sources expanded.

g) Plan for institutional strengthening.

i) Broader sample of companies surveyed.

h) Inter-agency info sharing agreements.

Action plan adopted.

Adjustments to transactions register

Reclassifications among accounts

Legal framework for BCRES to gather information.

Better quarterly reconciliation

BCRES disseminates harmonized external sector statistics, increasingly based on BPM6 guidelines.

Improved systems in place for compilation; consistency, coverage; & more accurate valuations.

BCRES has built up sustainable capacity to produce harmonized, reliable and complete data on BOP, IIP, Foreign Debt, reserves & metadata as per BPM6 standards.


Indicators: Quarterly BP, IIP & total external debt figures in national & IMF

website. Publication of monthly figures of international reserves Report BP & IIP metadata on national & IMF website considered

technically sound. Percentage compliance with BoP and IIP to BPM6 standards Overall data deemed of better quality.


CAPTAC-DR & IMF reports (e.g. aide memoires, BTOS, briefing notes)

Workshop reports, workshop evaluations, surveys, interviews

CAPTAC-DR and IMF reports External Sector Accounts Interview with experts (& staff) and CCS records Expert opinion from CAPTAC-DR / IMF

Major risks & assumptions


BCRES is willing to implement project recommendations.

Sufficient resources available for BCRES.

Political stability & support. Sensible macroeconomic policy. Staff with the appropriate qualifications trained and retained. Information is gathered satisfactorily (adequate disclosure from

both private and public sector organizations). Effective legal systems and overall institutional infrastructure. Sufficient resources available. Absence of substantial external shocks


67 | P a g e

5.8.2 EVAL UATION K EQS AND RATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities


Activities have been thoroughly consistent with program document, CAPTAC-DR strategy and approved work plan.



The project is coherent with national and regional goals, although it is not linked to any particular strategy document.

o Extent to which activity is linked to and supports sector/topical reform strategy (if one exists);


BCRES is keen on the project and has made important efforts to implement its recommendations.


Improving the external account statistics is very important for a country that is highly dependent on exports and remittances. An analysis of the project’s results chain (see above) suggests that sequencing was carried out correctly. The achievement of desired outputs and outcomes at a relatively early stage, as well as the interviews held during this evaluation indicate good absorptive capacity among BCRES staff.


Consistency with IMF Headquarter/other activities Whether activity is appropriately focused in terms of subject area, taking into account the IMF’s expertise and integration with HQ and other Fund activity (e.g. other training and TA delivered including through the topical trust funds).

This intervention draws from a technical area where the IMF has particularly strong expertise. There is a good balance between focus and materialization of synergies stemming from the regional aspect of the PRAESE project.


The activity is mostly self-contained.

Context of donor/TA landscape of CAPTAC-DR TA Why did the TA recipient choose the CAPTAC-DR as the TA provider as opposed to other donors?

The IMF is a recognized world leader in this technical area.


N.A.

Score for Relevance Excellent 3.6 EFFECTIVENESS Impact Interventions’ expected Impact achieved or likely to be achieved (to the extent defined in the activity’s log frame).

There is a high probability that the intervention will achieve its expected impact. In fact, there are already signs that the quality, timeliness, and frequency of publication of the data are improving.

Outcomes Intervention’s Outcomes achieved or likely to be achieved (as defined in its log frame): Short, medium and long term Outcomes.

There is already a clear indication that the progress is advancing firmly towards achieving its outputs, including those, which are set for the long-term.


The PRAESE program further leverages this activity across the region.

Score for Outcomes Good 3.4


68 | P a g e

Outputs Intervention’s Outputs achieved or likely to be achieved (as defined in its log frame), including: o Timeliness of reports/workshop presentations; o Quality of reports/workshop presentations; o Appropriateness/applicability of advice, given


Outcomes.

The TA reports and recommendations are comprehensive enough in scope to produce the desired outcomes due to the fact that most of them were accepted and implemented by the BCRES. A large number of outputs have already been produced. There have been minor deviations in the timing of some workshops, but all of them have been delivered roughly as originally programmed.

Score for Outputs Excellent 3.6 EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; The workshops have been carried out in an efficient manner. Timeliness in executing the activity; Workshops and TA have been provided on time. Appropriateness/effectiveness of IMF’s internal management of the activity;

Appropriate


Appropriate.


Work was carried out at a high technical standard.


Workshops are run efficiently. In particular, participants appreciate the fact that there is continuity and follow-up.


Evident from backstopping correspondence/comments and their influence on the quality of the final TA report.


Staffing is of a high technical caliber.


The RA thought that backstopping was very useful, especially for those who never worked at the IMF. Division of labor appears appropriate.


Each year the work plan has been fully executed, with the budget leaving a modest surplus.

Whether activity, given its results, has been cost-effective compared to other TA delivery modes

It is difficult to compare costs with similar activities, as TA is highly specialized. Yet the costs are in line with prices for top level workshops of similar logistical characteristics in El Salvador.


The chosen venues have been five-star hotels. Participants suggested that more efficient use of resources could be attained if more modest venues were chosen and the savings used to lengthen training sessions.


A comprehensive monitoring matrix listing all key recommendations was developed and used appropriately.


No information reviewed. It appears that TAIMS is not widely used by technical officers at CAPTAC-DR.


RBM has recently been implemented. No evidence was reviewed of its intensive use as a project management tool.


No evidence was reviewed on formal extraction/usage of lessons learnt.

Score for Efficiency Good 3.2 SUSTAINABILITY Sustainability of TA activity Outcomes from intervention will last beyond/continue after completion of TA/training.

The intervention has strengthened BCRES as an institution. Possible staff turnover is a concern. The bank rotates its staff regularly, which is extremely positive to ensure a broad distribution of skills. There is room for improvement as


69 | P a g e

regards the establishment of in house training.

For TA/training designed to deliver Sustainability, local institutions/ capacities have been strengthened to sustain results beyond the life of the intervention: o E.g., absorptive capacity improved and developed to

sustainable level; o E.g., in-house training capacity built, where relevant.

Absorptive capacity at BCRES is high. Yet, strengthening in-house training should be a priority for the next stages of the project.


Because there are low marginal costs incurred in the improvement of the external statistics, the additional financial burden is low and the Bank can implement the improvements within its own resources restrictions.


No pertinent information reviewed


The PRAESE program, in which this activity is embedded, has developed capacity throughout the region.

Score for Sustainability Excellent 3.6 Rating Based on Previous Evaluation Methodology

DAC


Weighed Score

Weight Total score

Rating


3.80 60% 2.28


3.80 20% 0.76

Co-ordination with Development Partners 2.60 20% 0.52 3.56 32% 1.14 Excellent Effectiveness Impact - Outcomes: TA 3.4 30% 1.02 Outcomes: Regional capacity building 3.6 30% 1.08 Outputs 3.6 40% 1.44 3.54 28% 0.99 Good Efficiency Process and implementation efficiency 3.5 40% 1,40 Efficient use of resources (human & fin.)


Monitoring and reporting 3.0 20% 0.60 3.20 22% 0.70 Excellent Sustainability Sustainability of RTAC/TA activity 3.6 75% 2.66 Contribution to building sustainable

regional TA, implementation capability 3.6 25% 0.90

3.56 18% 0.64 Excellent

TOTAL 3.47 Good


70 | P a g e

DESKTOP REVIEWS


71 | P a g e

6. EL SALVADOR AND HONDURAS: LARGE TAXPAYERS UNITS 6.1 INTRODUCTION AND SUMMARY OF RESULTS

The above project was selected for “desktop sample” review, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s inception report. It has been assessed on the basis of: (i) project-related documentation available at CAPTAC-DR (Mission Reports, Annual Reports, website, etc.); (ii) a phone conversation with the CAPTAC-DR RA on tax issues; and (iv) survey feedback relating to the project. CAPTAC-DR’s intervention in this area has filled an extremely important void. In addition, this TA has initiated a strong reform process. Progress in El Salvador has been substantial; whereas Honduras has advanced at a steady, yet slower pace. Collaboration with other donors, particularly in El Salvador has been

noteworthy, providing an example of how to leverage CAPTAC-DR’s TA in order to achieve increased effectiveness. 6.2 BACKGROUND

World Bank data shows that, with a population of just over 6.2 million, El Salvador has made steady progress in important indicators such as life expectancy at birth (72 years in 2011) or per capita GDP (current US$ 3,702 in 2011). Nevertheless, the country still lags behind the Latin American averages of 74 years and US$9,754 in 2011. By 2005, the economy was growing steadily, with GDP growth at around 4 per cent per annum. However, this trend was interrupted by the 2008 global financial crisis, and has not recovered since. Honduras, with a slightly larger population of approximately 7.8 million has a higher life expectancy than its neighbor to the southwest (73 years). The country has a substantially lower per capita GDP of US$2,226 than El Salvador (albeit showing a marked improvement in the last ten-year period). These figures remain well below neighboring Costa Rica’s, with 79 years and US$8,676. The Honduran economy had been growing at well above 5 percent until a major dip 2008/9, but managed to recover to approximately 3 percent in 2010 and 2011. These countries show a rather unequal income distribution, with Gini coefficients of 0.48 and 0.57, roughly within the range of other Latin American countries. It is well documented that fiscal policy can be a powerful tool to tackle income inequality and to foster other critical aspects of human development, such as health and education. Efforts to continue improving growth, poverty, health, and important policy goals must therefore include an optimal utilization of fiscal tools. Unfortunately, tax policy was performing below potential for both countries. A number of well-documented reasons explain this situation. A limited number of taxpayers account for a large proportion of government revenues. For example, in El Salvador the top 300 taxpayers account for almost 54 percent of total tax receipts; whereas in Honduras the top 621 represent 80 percent of total revenues.


72 | P a g e

Tax administrations in both countries therefore focus on their top contributors as a priority. On the one hand, large taxpayers are highly sophisticated in managing their tax liabilities. On the other hand, the amounts involved are extremely high and represent several percentage points of GDP. For these reasons, the respective internal revenue administrations (Dirección General de Impuestos Internos (‘DGII’) and Dirección Ejecutiva de Ingresos (‘DEI’) respectively) need to develop their capabilities both to build good working relationships with large taxpayers and to exercise effective control over their tax affairs. CAPTAC-DR has been supporting them in meeting these objectives. At the outset of the project, both countries’ capacities in this area were deemed to be low. In the case of El Salvador, there were no specialized competences or processes dealing with type of taxpayer. The country had at that time a list of 3,519 taxpayers classified as large on the basis of their yearly tax payment. On the one hand, this criterion missed out on important aspects, such as a firms’ overall revenues (independently from their tax liabilities), and amount of purchasing activity carried out. On the other hand, the limited number of staff dedicated to dealing with these large taxpayers lacked effective control and support. The staff also lacked an appropriate work infrastructure to facilitate their proper execution of their tasks and carry out their important functions properly. The country’s treasury was thus losing out on substantial potential income due to inappropriate control mechanisms. Honduras, on the other hand, had a large taxpayer unit in the past, but it had been dismantled. As with El Salvador, there was no dedicated space, infrastructure, or staff support. In addition, there was no proper distribution of work between officers in Honduras and those in San Pedro Sula. Cross-checking of information with customs was almost inexistent. Because of this lack of effective control the Honduran treasury was forgoing income potentially amounting to several percentage points of GDP. Finally both countries’ treasuries also lost out on income because their tax inspector’s lacked sector-specific knowledge in critical industries, as well as having deficient ITC systems in place. 6.3 CAPTAC-DR TECHNICAL ASSISTANCE Since its inception, CAPTAC-DR has remained in active dialogue with both DGII and DEI, leading to concrete collaboration at the operational level. CAPTAC-DR supported the formulation of the DGII’s Strategic Plan for 2012-14, as well as a similar plan for Honduras. Both strategies are now adopted and published. In the case of El Salvador, the plan contains the necessary elements to build DGII’s capacity and focuses on seven strategic lines of action, namely: (i) new control audit model, (ii) registry and account management; (iii) use of internet as means for interaction; (iv) process reengineering, restructuring, and management control; (v) integrated human resources management; (vi) implementation of tax reform; as well as (vii) setting up a Large Taxpayer Unit (‘LTU’). Implementation of these plans has already begun. In fact the LTU has already been created in El Salvador, with a staff of 103 officers. Its organizational structure was approved and job descriptions formulated. CAPTAC-DR helped DGII and DEI draw optimized lists of large taxpayers (‘LTs’) by using key criteria not only covering past tax payments, but also overall sales and the level of purchasing / procurement by the individual companies. As a result, in both countries the original large taxpayer lists were reduced to 600 in El Salvador (accounting for about 58 percent of total tax collection) and to 621 in Honduras. The latter represents as much as 12 percent of Honduran GDP. In both countries, the volume of information available about these large companies is systematically growing. CAPTAC-DR has also supported DGII and DEI in implementing a number of key recommendations. They revolve around the determination of an optimal organizational structure and operational processes for the respective LTUs. Some of the key strategic concerns include:


73 | P a g e

Separating strategic and operational functions to optimize internal controls and to clarify responsibilities; Making suitable facilities available to unit staff and taxpayers alike; Using teamwork to discuss special / complex cases; Making the necessary IT support available (e-mail storage space, software, access to the internet); Checking various software packages for tax administration and giving their implementation high priority (e.g. FENIX

Live, Selex, MaxTax, and Auditax); Providing training on SAP, Accent and/or Excel (advanced); Standardization of “equivalent cases”, in relation to complexity and size, thus enabling comparisons among individual

inspectors and audit teams; and Generating an organizational culture to detect evasive practices. El Salvador has adopted a number of these recommendations and already begun implementation. As indicated earlier, the LTU has now been set up, and work processes are currently being optimized. A large taxpayer support center has been established, with adequate facilities and equipment, which has been funded through bilateral aid channels with the USA. Large taxpayers now have their dedicated space. The Honduran LTU is already operating in a new building with adequate overall facilities (e.g. computers).19 DEI has also made significant progress in updating its audit manual. Recommendations also touch on operational process for the LTU. Given the large amounts of tax receipts at stake, there is a strong need to bring both units rapidly to full operational levels. Some of the key suggestions provided by CAPTAC-DR’s STXs relate to: Control mechanisms, law enforcement, and tax collection; Services to taxpayers; Coordination with the national Treasury; Use of the internet (which in Honduras has recently allowed the detection of irregularities with a large taxpayer); Determination of the right value of business operations (e.g. payment of royalties or transfer of company stock); Detection and monitoring of key practices (such as mergers and acquisitions); and Set up of a system to identify new or emerging issues for priority auditing. CAPTAC-DR has placed a strong emphasis on developing in-depth knowledge of the key sectors being covered by the LTU. In the case of El Salvador, sector files have been developed for energy (oil) and communication companies in order to capture their industry specificities. Honduras has identified its initial focus for this work, namely the communications, mining and financial sectors. It is expected that work on developing files for LTs in these sectors will begin shortly and will allow for enhanced fiscal control.

19 Though they are part of the same team, units are operating out of both San Pedro Sula and Tegucigalpa.


74 | P a g e

Key aspects of this work include: Understanding the characteristics of the sector under study (structure, competencies, profitability, types of

investments, actors, origin of the funds); Gathering rules and regulations (both tax and otherwise) applying to the sector; Establishing both formal and informal contacts with regulating bodies, seeking to understand the nature of their work

and to exchange important information (revenues, costs, investments); Establishing collaborative relationships with both the individual companies and their sector-level associations; Determining the sector’s legal franchise as well as tax retention regimes; and Drawing up their business processes through flux diagrams. Finally, CAPTAC-DR remains in close touch with both DGII and DEI in order to monitor the advancement of work in their respective countries. TA support continues to be provided, mostly in the form of TA at regular intervals, while training is also scheduled for 2013. It is expected that the process of selecting and auditing large taxpayers will broadly follow a three-year cycle. If sustainability conditions are met as expected, Honduras and El Salvador will be updating their respective LT lists in around 2015. 6.4 SUGGESTED LOGICAL FRAMEWORK We found no direct evidence of a logical framework analysis / RBM applied to this particular initiative. The table below proposes a sketch of the basic suggested contents applicable for such a framework:

Project

Inputs / Activities

(CAPTAC-DR Only)






El Salvador & Honduras Large Taxpayer Units (LTUs)

I/P: LTX / STX Backstopping & consulting time; travel costs; overheads. Activities: 1. Missions; 2. Back-

stopping; 3. workshop; 4. mentoring; 5. training.

a) Diagnostic & recommendations

b) Strategic plan adopted

c) Staff recruited and trained

d) Work systems in place

e) List of LT published f) Adequate premises

for the unit operational

g) Sector-specific LT files populated and used

h) Online only tax returns by LT.

LTU starts operations. Indicators: a) LT Dedicated

Facilities open b) Dedicated tele-

phone support line operational

c) Website updated

d) LT List published

e) Expert opinion of CAPTAC-DR/IMF.

LTU improves audit of and support to large taxpayers. Indicators: a) Number of

irregularities detected

b) % of overall LT list audited yearly

c) Expert opinion of CAPTAC-DR/IMF

d) Number of taxpayers surveyed indicating improved support.

Improved govern-ment revenue (=> stronger human development support). Indicators: a) Tax collection as

% of GDP b) Total revenue

from LT c) Total tax receipts

from sectors identified as % of their revenues.

Sources of verifi-cation

a) CAPTAC-DR reports

b) Workshop reports.

a) CAPTAC-DR reports b) Staff lists c) Physical inspection d) DGII & DEI websites.

a) Physical inspection

b) Systems assessment

c) DGII & DEI websites.

a) CAPTAC-DR monitoring reports

b) DGII & DEI records c) Large taxpayer d) Interviews /

surveys.

a) Nat’l accounts b) DGII & DEI

statistics c) Treasury

statistics.


75 | P a g e

Project

Inputs / Activities

(CAPTAC-DR Only)






Major risks / assump-tions

Local imple-mentation conditions allow for the activities to take place as planned.

a) Both parties are willing to implement a LTU

b) Sufficient resources available for appropriate IT deployment & facilities upgrade.

a) Political support

b) IT systems in place and functioning correctly

c) Sufficient re-sources available.

a) Sufficient absorp-tive capacity at both organizations

b) Sufficient levels of staff with the appropriate qualifications trained and retained.

Enhanced control mechanisms (investigations, audits, LT sensitization) lead to increased tax revenue.

(*) Note: On the basis of the information reviewed, it is estimated that for both countries outputs a), b), e) and f) have already been completed whereas outputs c), d) g) and h) have been started but are not yet fully accomplished. El Salvador appears to be more advanced than Honduras in attaining project outputs and outcomes. 6.5 EVALUATION KEQS AND RATINGS The evaluation ratings below reflect the averages for El Salvador (which would have scored higher across the DAC criteria on its own) and Honduras (generally lower). Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities Consistency of activity with Program Document, CAPTAC-DR strategy, and approved work plan.

Fully consistent with the project document and CAPTAC-DR strategy.

Consistency of intervention(s) with national/regional priorities: Intervention identified as a Government priority. o Extent to which activity is linked to and supports

national/regional strategy Fully consistent with local strategies and plans (which CAPTAC-DR helped develop). Work has been carried out at the right institutional level. Strong evidence of resource allocation by both El Salvador and Honduras (e.g. to set up the large taxpayer support centers).




Sequencing is appropriate.


Local capacity appears adequate for this particular initiative, though further training has been identified as a key priority.

Consistency with IMF Headquarter/other activities Whether activity is appropriately focused in terms of subject area, taking into account the IMF’s expertise, and integration with HQ and other Fund activities

The project supports both countries in carrying out suggestions provided by TA from FAD/IMF.


Coordination with the USA made possible appropriate facilities in El Salvador. Collaboration with IADB was actively sought in Honduras. Dialogue is also ongoing with GIZ.

Context of donor/TA landscape of CAPTAC-DR TA Why did the TA recipient choose CAPTAC-DR as the TA provider as opposed to other donors?

This line of TA appears to have flown naturally from the dialogue between CAPTAC-DR and the tax administrations in both countries.


N.A.

Score for Relevance Excellent 3.7 EFFECTIVENESS


76 | P a g e

Key Evaluation Questions Comments Impact Intervention’s expected Impact achieved or likely to be achieved (to the extent defined in the activity’s logframe).

The intervention’s impact is likely to be achieved, although it may require longer-term support, particularly in Honduras.


Outcomes are likely (although their accomplishment may require longer than expected), although there is slower progress in Honduras.

Significance of CAPTAC-DR contribution/likely contribution to developing core economic functions and institution building in the country/region, through the activity.

DGI and DGSA appear to support this initiative at the appropriate levels. No information on the use of local expertise has been available for review.

Score for Outcomes Good 3.3 Outputs Intervention’s Outputs achieved or likely to be achieved (as defined in its logframe), including: o Timeliness of reports/workshop presentations; o Quality of reports/workshop presentations; o Appropriateness/applicability of advice, given existing

capacity/constraints; o Whether Outputs are likely to produce intended Outcomes.

Outputs are likely to be achieved, although CAPTAC-DR’s intervention is only responsible for a relatively small fraction of them. CAPTAC-DR’s highest value added appears to stem from its catalytic role in bringing about change. Assistance so far seems to have been of good quality. It is unclear to what extent both LTUs will build their capacity up to international best practice standards in the short-term. A longer-term approach seems more likely.

Score for Outputs Good 3.5 EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; N.A. Timeliness in executing the activity; The pace of progress appears appropriate, particularly in El Salvador. Appropriateness/effectiveness of IMF’s internal management of the activity;

Internal management appears appropriate.


Appropriate.


Not evident from documentation reviewed.

The efficiency of planning and executing the TA/training delivery. This appears to have been the case. Efficient use of resources (human and financial) and attention to cost effectiveness

Appropriateness of staffing composition for the intervention; Professionals appear to have the right qualification and caliber. Appropriateness of work allocation between HQ and CAPTAC-DR in relation to activity;

The relative work allocation appears appropriate.


No information available. Expenditure does not seem to be tracked in detail.


The activity appears cost effective. In some cases missions have covered a number of issues back-to-back, including that of this particular project. There are efficiency gains by working in two countries simultaneously. However, it is not possible to be definitive given the absence of cost information.


Not evident from information available.


Reporting appears patchy or at least not easily accessible.


RA and back-stoppers have no access to TAIMs. This solution is only used for administrative purposes.


RBM does not appear to have been extensively applied to this intervention as it wasn’t’ foreseen since its inception. Yet its key elements are implicit.


No evidence of application of lessons learned was made available to the evaluators, though the RA has the right qualifications and impressive experience.

Score for Efficiency Good 3.0 SUSTAINABILITY Sustainability of TA activity


77 | P a g e

Outcomes from intervention will last beyond/continue after completion of TA/training.

Outcomes are likely to persist if the project achieves its goals satisfactorily. However, DGII and DEI will need to maintain human & financial capacity.


It is expected that DGII and DEI will have built their capacity to continue information exchanges in the future.


Not applicable.

o E.g., in-house training capacity built, where relevant. Likely. For interventions designed to deliver sustainability, financial sustainability has been achieved

The activities are likely to be sustained on a self-supporting basis particularly if tax revenue rises as a result of the project.




Positive.

Score for Sustainability Good 3.1 Rating Based on Previous Evaluation Methodology

DAC


Weighted score

Weight Total score

Rating


3.9 60% 2.34


3.9

20% 0.78

Co-ordination with Development Partners 3.9 20% 0.78 3.90 32% 1.25 Excellent Effectiveness Impact - - Outcomes: TA 3.2 30% 0.96 Outcomes: Regional capacity building 2.6 30% 0.78 Outputs 3.1 40% 1.20 2.94 28% 0.82 Good Efficiency Process and implementation efficiency 3.2 40% 1.28 Efficient use of resources (human and

financial) & attention to cost effectiveness 3.0 40% 1.20

Monitoring and reporting 2.5 20% 0.50 2.98 22% 0.66 Good Sustainability Sustainability of TA activity 3.1 75% 2.32 Contribution to building sustainable


2.94 18% 0.53 Good

TOTAL 3.26 Good


78 | P a g e

6.6 KEY FINDINGS AND RECOMMENDATIONS 1. The rate of progress towards successfully establishing LTU in the region is variable. There seems to be scope for

enhanced collaboration among CAPTAC-DR member countries (notably El Salvador and Honduras for the case reviewed).

2. In the same vein it might be practical (subject to host country resource constraints) to support a secondment arrangement between El Salvador and Honduras.

3. Both El Salvador and Honduras appear to have appropriate absorptive capacity while political commitment seems strongest in El Salvador.

4. The use of RBM, and notably of logframes as programming and monitoring tools for the LTU project in each country, is strongly recommended.

5. The current budgeting system mostly provides information on mission costs (by person time of RAs or STXs). This process does not easily allow for building in the time of back-stoppers to particular projects and other ancillary costs.

6. It is also not clear if other costs are also involved (e.g. for the training events) or how to track actual versus budgeted costs for individual, multi-year, multi country projects. Reports aggregating all relevant project costs on a sub-topic by country including all relevant cost items would be a substantial improvement.

7. One objective is to automate analysis of LTU online returns to facilitate a risk based approach to detecting evasion and providing focus to tax audits. Since the IMF excludes funding for the IT investments, this objective might fail to be achieved totally without complementary TA from other donors and major financial commitment from the recipient countries.

8. It is unclear whether tax administrations have developed in-house training capacity that can build on the procedures manuals and training of existing staff and taxpayers provided as outputs of the LTU program.

9. The encouraging preliminary results from El Salvador and Honduras could be leveraged for sensitizing other governments in the region.

10. Different STXs were deployed at different points of time and in each country. Using the services of one person for the whole project in both countries might have yielded additional efficiency gains.

11. Although impressive progress has already been achieved, long-term support is likely to be required. In particular, developing sector-specific approaches, as already started in both countries will certainly benefit from continued TA.

12. In general, the detailed recommendations provided during CAPTAC-DR missions (and spelled out in the various reports reviewed) are technically sound and well suited for in-country conditions. CAPTAC-DR should ensure that TA support is provided to El Salvador and Honduras to implement the recommendations.


79 | P a g e

7. NICARAGUA: TAX-CUSTOMS INFORMATION SHARING 7.1 INTRODUCTION AND SUMMARY OF RESULTS

The above project was selected for desktop review, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s inception report. This review has been carried out on the basis of: (i) project-related documentation available at CAPTAC-DR (Mission Reports, Annual Reports, website, etc.); (ii) a phone conversation with the CAPTAC-DR RA on tax issues; and (iv) survey feedback relating to the project. This project constitutes a good example of the added value of CAPTAC-DR’s TA to its member countries. Despite its high importance to enhance government revenue, this information sharing would not be taking place without the Center’s intervention. Work has been carried out in a technically solid manner. It remains to

be seen whether Nicaragua can institutionalize the customs/internal revenue information sharing arrangements in the long term. 7.2 BACKGROUND According to the World Bank Nicaragua is the region’s second poorest country after Haiti. Nevertheless its social indicators, such as life expectancy at birth and poverty rates have shown sustained improvements in the past decade. GDP, in turn, has been growing above 4 percent since 2008/9 when the global financial crisis had a negative impact. As mentioned in CAPTAC-DR’s reporting, Nicaragua has complied with all quantitative criteria set out in the IMF country program. Despite these encouraging trends, Government revenue remains well below potential. The government has therefore recognized the need to reform the tax and customs administrations (Dirección General de Ingresos General Income (‘DGI’) and Dirección General de Servicios Aduaneros (‘DGSA’) respectively) in order to enhance resource mobilization and to improve control mechanisms. These goals will only be achieved, inter alia, through a substantial strengthening of institutional capacity at both organizations. In line with Government priorities, DGI has already embarked on a reform program, as set out in its Modernization Plan 2012-2025, in its Strategic Plan 2009-2012, as well as in its yearly operational plans. Key aspects of this strengthening involve securing an enhanced resource base, as well as using technology in order to leverage the effectiveness of its control functions. In fact DGI’s tax information system dates from 1993. Even though some of its modules are still relevant, the overall system requires a major overhaul, particularly in terms of programming and functionalities. The hardware supporting this information system is also becoming obsolete and is susceptible to breakdowns. In general both processing and storage capacities are limited, while the system is not scalable due to its specific technical configuration. DGSA, in turn, has also formulated a Modernization Plan for 2012-2015, as well as a Strategic plan for 2009-2012 (With CAPTAC-DR support). The organization seeks to improve its operational effectiveness by adopting international best practices in customs administration.


80 | P a g e

Within this process, DGSA has received support not only from CAPTAC-DR, but also from the IADB, the World Bank, the US Customs and Border Protection Agency, the Word Customs Organization, and the IMF (headquarters). DGA’s priorities include the updating of its IT system (to the “ASYCUDA World” solution); implementing a merchandise transit system; and strengthening rules and regulations. Despite DGI’s and DGSA’s efforts, fraud remains a matter of particular concern across both administrations; ranging from issues such as fake invoicing and bogus VAT refunds to misrepresentation of tax/duty obligations. As a response, the country has begun implementing a set of measures to stem fraudulent activities. Enhancing the operational collaboration between DGI and DGSA constitutes a cornerstone of these efforts. Such cooperation will improve the Government’s ability to detect suspicious activities and to leverage its law enforcement through joint DGI and DGSA operations. Such joint work will enable, inter alia, to reconcile import declarations and tax returns, which in turn leads to more focused tax audits. These enhanced investigations are expected to generate an increase in tax collection (e.g. through the detection of irregularities such as under-reporting of profits/ sales, to minimize VAT payments). In general, improving the operations of both DGI and DGSA will also generate enhanced taxpayer compliance (resulting from taxpayers’ own risk assessments that will be modified through broader sensitization and a grounded feeling of higher risk of detection). 7.3 CAPTAC-DR TECHNICAL ASSISTANCE Since its inception, CAPTAC-DR has maintained frequent contact with Nicaragua’s tax and customs administrations through a number of channels. Fruitful interaction has come about, inter alia, through the participation of Nicaraguan officials at CAPTAC-DR training events, including a national seminar on an “Integrated vision on tax control processes based on basic elements of the tax administration” which complemented a regional workshop entitled “Integrated vision on control processes and tax management”. Both events took place in 2011. In addition, contact was maintained through a series of short-term visits to Nicaragua by CAPTAC-DR staff and STXs. Within that context, CAPTAC-DR carried out a diagnostic of both DGI and DGSA. This exercise yielded practical recommendations identifying ways to optimize the performance of both institutions. In particular, this diagnostic identified the need to achieve concrete synergies between DGI and DGSA, through enhanced information sharing and joint operational collaboration. A roadmap was formulated, setting out the necessary steps to bring about the desired collaboration between DGI and DGSA. This agenda is based on a formal agreement/protocol signed in 2009 by both institutions, aimed at instituting such an information exchange system, as well as to enhancing collaboration at the operational level. In line with international best practice, it was decided to operationalize this information exchange by means of virtual private networks (‘VPNs’), in order to allow each institution to keep autonomous information platforms. As per this agreement, DGI was to provide (i) its taxpayer database; (ii) its economic activity database; (iii) information on fiscal solvency of key companies; (iv) tax information on companies that fall under the temporary admission regime; (v) annual reports of payments and levies on export/import operations; and (vi) information about ongoing administrative and other legal procedures. In turn, DGSA was to make available counterpart information including (i) listing of taxes paid in each declaration; (ii) information on specific exports and imports; (iii) details of all customs declarations; (iv) information linkages between export/import firms and freight companies; (v) details on cancelled customs declarations; and (vi) overall billing information. In addition DGSA agreed to collaborate with DGI on specific topics, such as VAT returns.


81 | P a g e

CAPTAC-DR’s advice was directed at solving specific issues appearing during the early implementation of the above mentioned agreement, such as dealing with extraordinary or anomalous situations; optimizing VPN communications between both institutions; and streamlining specific export/import procedure. In 2011, it was also agreed to revise the 2009 agreement through the formalization of an aide-memoire/MoU including a detailed work plan and sketching out the proposed solution to the operational issues identified. This update of the inter-institutional agreement appears not to have been executed yet. Nevertheless, it appears that inter-institutional information sharing is already underway. Finally, the various CAPTAC-DR missions have identified further training as a key need pursuant to the operationalization of this information exchange mechanism. Training is also required to ensure that the crosschecked information leads to an increase in targeted investigations, thus increasing the effectiveness of both DGI and DGSA. 7.4 SUGGESTED LOGICAL FRAMEWORK We found no direct evidence of a logical framework analysis / RBM applied to this particular initiative. The table below proposes a sketch of the basic contents of such a framework.

Project Inputs / Activities (CAPTAC-DR only)


Outcomes (Overall project) Short-term



(4-10 years) Nicaragua: info sharing between tax and customs administra-tions

I/P: LTX / STX / Backstopping & consulting time; travel costs; overheads; software & hardware.

Activities: Missions; backstopping; technical advice on IT systems; workshops; on-the-job training.

a) Diagnostic and recommendations.

b) Revised MoU between tax & customs administrations.

c) IT systems ready for information sharing.

d) Staff from DGI & DGSI trained on sharing and analyzing the info.

Customs & tax administrations share informa-tion as per the MoU.

Indicator: Information accessible at the IT systems of both organizations.

Customs & tax administrations reconcile the information leading to more focused audits.

Indicator: No. of investigations launched based on information crosschecks.

Improved revenue (allowing for stronger social & economic development support).

Indicators: tax collection as % of GDP. Customs receipts.

Sources of Verification

CAPTAC-DR mission & STX reports, workshop reports.

CAPTAC-DR mission & STX reports, workshop reports.

Examination of IT systems at DGI & DGSA

CAPTAC-DR monitoring reports; DGI & DGSA statistics; interviews with DGI & DGSA staff; nat’l accounts.

Major risks / assumptions

Local implementation conditions allow for the activities to take place as planned.

a) Both parties are ready to sign the MoU.

b) Enough resources available for appro-priate IT deployment.

c) Appropriate staff trained and retained.

a) DGI & DGSA are inclined to cooperate.

b) IT systems in place and functioning correctly.

Sufficient absorptive capacity at both organizations allows them to make good use of the shared info.

Enhanced control mechanisms (investigations, audits, taxpayer sensitization) lead to increased tax revenue.

(*) Note: On the basis of the information reviewed, it is estimated that output a) has already been completed whereas outputs b), c), and d) have been started but are not yet fully accomplished. There is also evidence of incipient achievement of short-term project outcomes.


82 | P a g e

7.5 EVALUATION RATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities Consistency of activity with Program Document, CAPTAC-DR strategy, and approved work plan.


Consistency of intervention(s) with national/regional priorities: Intervention identified as a Government priority. o Extent to which activity is linked to and supports

national/regional strategy Fully consistent with local strategies and plans, though depending on the goodwill of ministry staff.



Work has been carried out at the right institutional level. No evidence of resources being allocated for the project by Nicaragua itself.



o E.g., appropriateness of project expected outcomes given country/institutional absorptive & implementation capacity.

Local capacity appears adequate for this particular initiative, though further training has been identified as a key priority.

Consistency with IMF Headquarter/other activities Whether activity is appropriately focused in terms of subject area, taking into account the IMF’s expertise, and integration with HQ and other Fund activity

The initiative falls within both CAPTAC-DR’s and the IMF’s (HQ) areas of expertise. It results from inputs from both origins.


Good coordination appears to have taken place, particularly with the EU and GIZ.


This line of TA appears to have flown naturally from the dialogue between CAPTAC-DR and the tax administrations in both countries.

How do workshop participants compare with those provided by CAPTAC-DR with workshops from other donors?

N/A

Score for Relevance 3.6 Excellent EFFECTIVENESS Impact Intervention’s expected Impact achieved or likely to be achieved (to the extent defined in the activity’s logframe).

The project is likely to improve the Nicaraguan authorities’ effectiveness in enforcing its tax regime, if outcomes are achieved.


More specific and time bound work plans are necessary. There are also risks in achieving outcomes in the absence of an updated IT system. CAPTAC-DR has


DGI and DGSA appear to support this initiative at the appropriate levels.

Score for Outcomes 3.3 Good Outputs Intervention’s Outputs achieved or likely to be achieved (as defined in its logframe), including: o Timeliness of reports/workshop presentations; o Quality of reports/workshop presentations; o Appropriateness/applicability of advice, given existing


From the information readily available, the nature of the expected outputs is also unclear. Only recommendations and statements of purpose have been reviewed. These appear to have been of good quality

Score for Outputs 3.5 Good EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; N/A.


83 | P a g e

Key Evaluation Questions Comments Timeliness in executing the activity; So far progress appears slow, yet appropriate. Appropriateness/effectiveness of IMF’s internal management of the activity;

Appropriate, based on evidence reviewed


Right counterpart selected.


Appropriate.

Appropriateness of staffing composition for the intervention; Staff has the right qualifications. The efficiency of planning and executing the TA/training delivery. The TA has already made some achievements on the basis of limited

inputs. Efficient use of resources (human and financial) and attention to cost effectiveness

Appropriateness of work allocation between HQ and RTAC in relation to activity;

The relative work allocation appears appropriate and collaboration seems to be taking place.


No budget information available for this activity in particular.


Difficult to assess cost effectiveness in the absence of cost information relating to the project, and the absence of results given its early stage.


No information available on this.


No evidence seen of ongoing/regular monitoring of implementation of recommendations.


RAs have no direct access to TAIMS. It is only used for administrative purposes.


RBM has not been extensively applied to this intervention, though many of its elements are implicit.


No evidence of application of lessons learned available for review, though CAPTAC-DR RA does have strong experience, and backstopper support is available.

Score for Efficiency 3.0 Good SUSTAINABILITY Sustainability of TA activity Outcomes from intervention will last beyond/continue after completion of TA/training.

Outcomes are likely to persist if the right framework for institutional collaboration is achieved, as both institutions are to make concrete gains. In addition, IT upgrades are needed, which pose significant risks to sustainability going forward.


It is expected that DGI and DGSA will have built their capacity to continue information exchanges in the future.


Not applicable.

o E.g., in-house training capacity built, where relevant. Likely. For interventions designed to deliver sustainability, financial sustainability has been achieved (i.e., recipients will be implement the advice from own/other financial resources in a sustainable manner).

The activities should be sustained, as the required IT maintenance resources appear within the Government’s reach. One key concern is the continued will by both agencies to keep sharing sensitive information.


The project, if successful, is likely to showcase relevant achievements possible through TA at the regional level.





84 | P a g e


DAC criteria

Sub-criteria Score Weight Weighted

score Weight

Total score

Rating


3.7 60% 2.22


3.7

20% 0.74


financial) and attention to cost effectiveness

3.2 40% 1.28

Monitoring and reporting 2.0 20% 0.40 2.96 22% 0.65 Good Sustainability Sustainability of RTAC/TA activity 3.0 75% 2.25 Contribution to building sustainable


2.75 18% 0.50 Good

TOTAL 3.20 Good 7.6 KEY FINDINGS AND RECOMMENDATIONS 1. It would be worthwhile to draw lessons from how other tax and customs administrations in Central America share their

information. 2. Collaboration might be enhanced through an in-country secondment arrangement between DGI and DGSA staff. 3. Nicaragua appears to have appropriate absorptive capacity. Political will seems to have fluctuated through project

implementation. It is critical to ensure that project outcomes are achieved despite possible political fluctuations. 4. The use of RBM as a programming and monitoring tool for this project is strongly recommended. 5. Data collaboration requires an upgrade in DGI and DGSA’s IT systems. Since the IMF excludes funding for the IT

investments, this objective might fail to be achieved totally without complementary TA from other donors and major financial commitment from the recipient countries.

6. The encouraging results from Nicaragua could be leveraged for sensitizing other governments in the region. 7. Momentum should be maintained to ensure that the updated provisions of the DGI/DGSA MoU are signed, and other

options explored, in order to ensure the institutionalization of the information sharing arrangements. 8. Long-term support is likely to be required. If IT requirements are not met, the whole project could be undermined. 9. CAPTAC-DR support should continue until it is clear that DGI and DGSA have learned to use the shared information

in order to improve control practices. 10. In general, the detailed recommendations provided during CAPTAC-DR missions (and spelled out in the various

reports reviewed) are technically sound and well suited for in-country conditions. CAPTAC-DR should ensure that support is provided to Nicaragua seeking to allow DGSA and DGI to implement the recommendations.

11. Different STXs were deployed at different points of time. Using the services of one person for the whole project might have yielded additional efficiency gains.


85 | P a g e

8. REGIONAL: SINGLE TREASURY ACCOUNT 8.1 INTRODUCTION AND SUMMARY OF RESULTS

The above project was selected for desktop review, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s inception report. As per its methodology, this review has been carried out on the basis of readily available information. This assessment has revealed a solid TA project, providing sound advice on a matter of great importance regionally. In fact the advice provided has been of high quality and well-adapted to the needs of CAPTAC-DR member countries. A similar STA project was carried out in Costa Rica; the Phase II Program Document cites the success of this project as an example where Costa Rica could provide peer to peer guidance to other countries in the region that are

implementing STA. At the same time, it is extremely important for local stakeholders to understand that Single Treasury Accounts (‘STAs’) improve control and transparency over the use of government financial resources. They can also bring large financial gains to the treasury while also improving overall efficiency as regards financial management. CAPTAC-DR benefits in all of its projects by the support of FAD in discussing and approving the action plans (BPs) and mission reports (BTOs) as part of FAD’s back stopping function. As with other CAPTAC-DR TA, impact is affected by counterpart interest and ownership of project activities. 8.2 BACKGROUND Governments around the world handle substantial amounts of financial transactions. They make millions of payments each year, and receive a similar number of small or large cash inflows. In most of the countries covered by CAPTAC-DR, relatively small economies imply a lesser amount of financial transactions than, for example middle-sized emerging markets such as Colombia, South Africa or Thailand. Nevertheless, in proportion to the size of their economies, governments in Central America, Panama and the Dominican Republic still handle a vast amount of payments. Most CAPTAC-DR member countries, even those with modern treasury systems such as Guatemala, have traditionally kept multiple government accounts in the commercial banking sector. This practice has sought to separate funds according to their destination, having in mind orderly compliance with constitutional or other legal budgetary allocation requirements. To illustrate the complexity, a CAPTAC-DR STX surveyed over 1,300 government accounts in the case of Nicaragua only. In some cases, payments systems were rather rudimentary and required support from commercial banks to organize the payment arrangements. This type of cash management system has brought about substantial inefficiencies. Given the relative weight of government treasury operations, resulting losses or opportunity costs can be substantial. Countries such as the Dominican Republic, Guatemala, El Salvador, Nicaragua and Panama have suffered from this fragmentation of their banking transactions management in four different ways. First, multiple accounts mean multiple idle account balances, which fail to earn proper interest. By way of reference on the orders of magnitude, the Guatemalan Government held cash positions of almost US$2 billion as of Oct. 2011.


86 | P a g e

Second, individual accounts occasionally get overdrawn, which incur interest payments. In many occasions other official accounts, for example that of an autonomous government entity, may have a positive balance at the same time. Overdraft-related costs would have been avoided if surplus cash from other accounts had been used wisely. Third, commercial banks may well lend on the basis of the idle government deposits in their institution, thus generating additional liquidity. If the government does not wish to run an expansive monetary policy, it would need to mop up the additional liquidity, for example by issuing debt obligations. This solution also entails additional costs. Fourth, key government officials, confronted with fragmented official accounts in the banking sector, typically encounter difficulties when trying to assess the government’s cash position as a whole. This situation in turn may cause sub-optimal policies to be enacted as well as poor overall oversight. Finally, the complexity relating to controlling the correctness and the timeliness of the various payments is compounded by the existence of numerous bank accounts, some of them, like in the cases of Guatemala or Panama, falling outside of the Treasury’s oversight. The IMF, CAPTAC-DR management, its member countries, as well as its donors, recognize this problem. A viable solution is also widely acknowledged, namely instituting a centralized STA system. This issue has emerged as a priority within the Public Financial Management reform agenda regionally. The main underlying principle is that money is fungible and its flow can be centrally managed regardless of its final use. On the other hand, tracking expenditure does not require assigning different bank accounts to different government bodies. An appropriate accounting system accomplishes that function instead. Under this system, no government body may operate its own bank account outside the treasury’s oversight. In addition, the STA approach does require all funds to be linked to one “top” (STA main) account, usually at the central bank that determines a daily consolidated cash position. Yet, this arrangement still allows for ledger sub accounts or zero balance accounts at commercial banks. Furthermore, different types of regimes may be enacted, ranging from the highly centralized to the decentralized. This unified (structure of) accounts thus allows managing both borrowing and the return of temporary cash surplus. It also brings monetary policy, as well as government cash oversight more closely under central Government control. In other words, instituting a STA brings about better cash oversight, together with increased net revenues for the treasury. All countries in the region participate in the forum of national treasurers (FOTEGAL) that is a project supported by the IMF, IADB, and the World Bank. FOTEGAL organizes an annual meeting that discusses ways to strengthen the STAs as one of the main objectives. This sensitization of the treasurers to the issues has been positive to engage and increase their commitment to support such reforms. 8.3 CAPTAC-DR TECHNICAL ASSISTANCE CAPTAC-DR has supported the Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua and Panama in implementing STAs. No other donor (except perhaps for the World Bank in the Dominican Republic) appears to have provided support in this important technical area in such a comprehensive manner. In fact IADB is currently supporting Panama (mostly on equipment purchase issue), USAID has provided a STX to El Salvador and the EC plans to provide TA to the Dominican Republic. CAPTAC-DR hence fills an important strategic role.


87 | P a g e

TA has been provided at a national level, though following a similar pattern of activities across countries. In fact, delivery has relied on TA missions, which have often included a workshop element. One of the first activities has typically revolved around making a diagnosis and sensitizing stakeholders. As a next step, the countries have established concrete roadmaps to achieve STA implementation. These roadmaps show various degrees of advancement, as per the reports reviewed. In general, the Costa Rican model, developed using FAD TA, has been followed. On the one hand, it is well adapted to the realities of CAPTAC-DR member countries. On the other, a single model across the region allows for synergies typical of regional harmonization. More detailed information about the advancement per country follows: Dominican Republic The Dominican Republic has received sustained support from CAPTAC-DR in implementing its STA. FAD has also led many diagnostic missions with the participation of CAPTAC-DR. Stakeholder workshops have also been organized alongside those missions. Despite the changes in administrations, the Dominican Government has consistently lent high-level political support and continuity to this initiative. The country has already put in place a regulatory framework compatible with STA practices. In addition, Phase I of the STA has already been implemented and substantial progress has been achieved. In fact 42 entities are now funded directly from the STA. In addition, participating government bodies have confirmed positive experience so far. A number of limitations however persist. CAPTAC-DR STXs have provided the government with a series of detailed recommendations in order to overcome them. El Salvador El Salvador has received five TA missions from CAPTAC-DR on this issue during the past two years. Their national roadmap was enacted and shows highly satisfactory progress. For example, the national regulatory framework has been made compatible with an STA system through revised legislation that has already been enacted. Functional requirements have also been identified, and many of them have already been met. Technical manuals have already been produced. CAPTAC-DR has also supported the country in formulating a plan for gradually closing distributed official bank accounts. A number of workshops have taken place nationally. More importantly, El Salvador appears to have fully appropriated the project and inter-ministerial collaboration has been both concrete and highly active. The country initially planned to start implementing its STA as early as January 2013. Nevertheless, outstanding work is still needed to be carried out as of September 2012, namely (i) the validation of bank account standards, (ii) the standardization of payment protocols and instructions; (iii) the inception of new monitoring tools, and (iv) the implementation of enhanced IT support. Guatemala The country boasts a state-of-the-art government payments system. All transactions are now executed electronically, thus bringing about substantial savings (on costs and banking fees), as well as efficiency gains (due to increased speed and accuracy in the payments). The principle of an STA is already enshrined in the country’s constitution. Selected laws, however, are not fully compatible and need to be amended. Nevertheless, a number of parallel government accounts are currently in operation, falling outside the scope of treasury oversight. Many of these accounts follow specific legal requirements. It has been recommended to transform many of them into zero balance accounts, or at least to ensure that balances are reconciled quickly.


88 | P a g e

The country has placed high priority on the implementation of a STA system. A vision document is however necessary. In fact, all stakeholders still need to understand that an STA approach revolves around liquidity management and has no impact on the ultimate use of the resources. In general, CAPTAC-DR’s support has played an important role in keeping the momentum of the reform process. Implementation of the work plan is advancing steadily. Nicaragua Nicaragua, in turn, has also embarked on setting up a STA system, particularly in light of the multiple accounts (over 1,300 of them) held by official government bodies in various currencies. There was a diagnostic mission in June 2009 involving FAD, the World Bank and the IADB, which laid the ground for this project and initiatives carried out directly by the government. In fact, the government has already succeeded in centralizing substantial information as well as the management on many of its official cash positions. Despite this encouraging progress, the country still needs to (i) improve its management of surplus cash; (ii) close a myriad of remaining accounts; (iii) enhance accounting practices relating to the differentiation of entities in the system; (iv) formulate a conceptual model document; (v) enact complementary regulations as required by law; (vi) discontinue the use of checks as means of payment; (vii) standardize the execution of similar transactions (e.g. payment of salaries); and (viii) organize a payment calendar. Panama The government has decided to implement an STA. CAPTAC-DR has kept in touch with Panama on the issue, including through technical missions in June 2011, October 2011 and September 2012. The June 2011 mission was carried out by FAD and produced an action plan which CAPTAC-DR has started to provide implementation TA on in the two subsequent missions above. As explained elsewhere in the report, the process of diagnostic from the TA Departments at HQ followed by implementation TA from CAPTAC-DR represents the usual division of labor and expertise as between field and HQ, which has proved itself as a successful working model. This work produced a detailed diagnostic, as well as a number of concrete recommendations. Implementation of the project has already begun though the progress appears to be rather slow. So far a concept note, a vision statement and revised legislation, a project work plan as well as new operational protocols have been proposed. By and large, these steps have yet to be enacted. In fact during the last CAPTAC-DR mission, the Government had not assigned a full time project coordinator to the STA initiative. Detailed recommendations have been issued for the process to gather momentum.


89 | P a g e

8.4 SUGGESTED LOGICAL FRAMEWORK We found no direct evidence of a logical framework analysis / RBM applied to this particular initiative. The table below proposes a sketch of the basic suggested contents applicable for such a framework:

Project

Inputs / Activities

(CAPTAC-DR Only)






Single Treasury Account (Dominican Republic, Guatemala, El Salvador, Nicaragua, & Panama).

Inputs (‘I/P’): Long-term expert (‘LTX’) / Short-term experts (‘STX’) Backstopping & consulting time; travel costs; overheads. Activities: Missions; backstopping; workshops; mentoring; training.


b) Strategic plan / vision. c) Stakeholders sensitized. d) Project team set up. e) Applicable legal framework

assessed & amended. f) Nat’l payment system

assessed & adjusted. g) List of preliminary accounts

to close. h) STA put in place (incl.

ledger subaccounts or zero sum bank accounts).

i) Preliminary accounts closed & transactions derived to STA.

j) Key staff trained.

Treasury has started operating the STA. Indicators: Amount (in local currency) of STA account turnover (including sub-accounts if appropriate). Expert opinion of CAPTAC-DR/IMF.

There are no residual secondary accounts open. Indicators: Amount (in local currency) in government accounts not under Treasury’s oversight. Expert opinion of CAPTAC-DR/IMF.

The gov’t improves its cash mgm’t; makes fluid payments & grows net revenues from cash positions. Indicators: a) Payment times. b) Total revenue

from interest in cash positions.

c) Total decrease in borrowing costs (ceteris paribus).

d) Total overdraft costs.

e) Cash forecasting accuracy.


a) CAPTAC-DR reports.

b) Workshop reports.

a) CAPTAC-DR reports. b) Physical inspection. c) Interviews with treasury

officials.


b) CAPTAC-DR reports.


b) Treasury records.

a) Nat’l accounts b) Treasury records. c) Interviews with

treasury officials. Major risks / assump-tions

Local conditions allow smooth implement-tation.

a) Political will to make STA a government priority.

b) Sufficient resources available for project activities.

a) Payment systems adequate.

b) Sufficient resources.

c) Absorptive capacity at treasury

d) Relevant orgs collaborate.

Treasury actively and competently manages cash positions.

(*) Note: Output completion varies among individual countries. El Salvador and the Dominican Republic appear to be most advanced whereas Guatemala and Nicaragua show a satisfactory evolution. Progress in Panama is slow.


90 | P a g e

8.5 EVALUATION RATINGS The ratings below reflect the average performance across all the countries reviewed. Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities Consistency of activity with Program Document, CAPTAC-DR strategy, and approved work plan.


Consistency of intervention(s) with national/regional priorities: Intervention identified as a Government priority across the region. o Extent to which activity is linked to and supports

national/regional strategy Fully consistent with local strategies (these plans are formulated at the beginning of the process).



Work has been carried out at the right institutional level. SLV appears to be the example to follow. PAN seems to have a weak sense of ownership.



o E.g., appropriateness of project expected outcomes given absorptive and implementation capacity.

Local capacity appears adequate for this particular initiative.


The project supports all countries in carrying out suggestions as per IMF’s rendition of best practices.


USAID (El Salvador); IADB (Panama), and EC may start work in the Dominican Republic.


This line of TA appears to have flown naturally from the dialogue between CAPTAC-DR and the regional Government. Also, the IMF is widely regarded as the utmost technical authority in relation to STAs.


N/A.


The intervention’s impact is likely to be achieved at least in some countries, although it may require longer-term support


Good outcomes are likely (although their accomplishment may require longer than expected)


Most countries (except perhaps Panama) appear to support this initiative at the appropriate levels. Progress has been variable. No information on the use of local expertise has been made available for review



Outputs are likely to be achieved, although CAPTAC-DR’s intervention is directly responsible for a relatively small fraction. The Center’s highest value added rests in keeping the momentum going while making expertise available to solve complex issues. Assistance is of good quality. It is unclear to what extent treasury depts. in all countries will build their capacity in the short-term. A longer-term approach seems appropriate


91 | P a g e

Key Evaluation Questions Comments Score for Outputs 3.0 Good EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; Not applicable Timeliness in executing the activity; So far the pace of progress appears appropriate in all countries bar

Panama Appropriateness/effectiveness of IMF’s internal management of the activity;

Internal management appears appropriate.


Not applicable


Appropriate counterparts were selected

The efficiency of planning and executing the TA/training delivery. Perhaps planning was somewhat optimistic Efficient use of resources (human and financial) and attention to cost effectiveness

Appropriateness of staffing composition for the intervention; Appropriate Appropriateness of work allocation between HQ and RTAC in relation to activity;

The relative work allocation appears appropriate i.e. a diagnostic mission and action plan generated by FAD in the first mission followed by implementation TA from CAPTAC-DR.


No information available


The activity appears cost effective. There are efficiency gains in relation with organizing similar work in several countries simultaneously




No evidence available. Reporting appears patchy. An effective monitoring tool, tracking project recommendations is sent to counterparts regularly


No information available. TAIMS appears to be used only for administrative purposes. Resident Advisors (RAs) have no access to the program


RBM does not appear to have been extensively applied to this intervention


No evidence of application of lessons learned was made available to the evaluators


Outcomes are likely to persist if STA gains are attained. However, treasury departments will need to maintain high human & financial capacity.


It is expected that the respective treasuries will have built their capacity to continue running the system


Not applicable

o E.g., in-house training capacity built, where relevant. Likely For interventions designed to deliver sustainability, financial sustainability has been achieved

The activities are likely to be sustained on a self-supporting basis particularly if government sees savings as a result of better cash and short term debt management.







92 | P a g e


DAC criteria


score Weight

Total score

Rating


3.9 60% 2.34


3.9

20% 0.78


financial) & attention to cost effectiveness 2.9 40% 1.20

Monitoring and reporting 2.5 20% 0.50 2.86 22% 0.63 Good Sustainability Sustainability of TA activity 3.0 75% 2.25 Contribution to building sustainable


2.95 18% 0.53 Good

TOTAL 3.17 Good 8.6 KEY FINDINGS AND RECOMMENDATIONS 1. The rate of progress towards successfully establishing STA across the region is variable. Whereas most countries

(e.g. El Salvador or the Dominican Republic) are making strides forwards, Panama seems to be stalling. 2. Given this disparity, secondment arrangements would be encouraged to foster the exchange of lessons learned. 3. All countries seem to have adequate absorption capacity. However, converging toward a STA is a complex process,

which makes TA necessary. Nicaragua is an example of this complexity and the need to edge forward steadily. 4. The use of RBM as a programming and monitoring tool for the STA project in each country is strongly recommended. 5. CAPTAC-DR has produced a monitoring matrix with the advancement of project recommendations per country. This

document has been circulated regularly with country counterparts, thus helping to keep the momentum for reform. 6. It is critical not to relent in the sensitization for political stakeholders to understand that STAs are not about controlling

the final use of the funds. Fears about losing resources could prompt important actors to block the reform process. 7. It is unclear whether treasury ministries have developed in-house training capacity that can build on the procedures

manuals and training of existing staff and other government departments and public sector entities provided as outputs of the STA program.

8. The encouraging preliminary results from across the region could be leveraged for sensitizing countries which are less advanced in the area (e.g. Panama) or those not yet boasting a STA.

9. Employing one person for the whole project in all countries might have yielded additional efficiency gains. On the other hand, having more than one STX available provides needed back-up options.

10. Although impressive progress has already been achieved, a long-term support is likely to be required. In particular, support may be needed for fine-tuning the system and solving glitches which are likely to arise regularly.


93 | P a g e

11. In general, the detailed recommendations provided during CAPTAC-DR missions are technically sound and well suited for country conditions. CAPTAC-DR should ensure that implementation support is provided. Also, work should be discontinued in countries lacking the necessary political will to carry out such a complex technical process.


94 | P a g e

9. MULTI-COUNTRY: SUPPORT TO MONETARY OPERATIONS 9.1 INTRODUCTION AND SUMMARY OF RESULTS

CAPTAC-DR’s TA on supporting monetary operations (liquidity management) in Costa Rica, the Dominican Republic and Guatemala was selected for desktop review, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s inception report. This review has been carried out on the basis of: (i) project-related documentation available at CAPTAC-DR (Mission reports, annual reports, website, etc.); (ii) A phone conversation with the CAPTAC-DR RA on monetary operations issues; and (iii) survey feedback relating to the project. The Center has provided valuable TA in order to help counterparts (i) better

manage short-term liquidity, (ii) develop and use macroeconomic models for forecasting purposes, and (iii) strengthen the deepness and the overall functioning of foreign exchange markets. Given the complexity of all three technical aspects addressed, progress hinges on both counterpart institutional capacity and their commitment to carry out the necessary reforms. The Center has made a strong contribution toward maintaining momentum for reform. Achievements are already substantial, but a long-term approach will be necessary in order to attain such a deep and varied set of objectives and to ensure that gains are consolidated in the long-run. 9.2 BACKGROUND In the last ten years, inflation has represented a serious preoccupation for central bankers and politicians in the region. As the chart below shows, the increase of consumer prices in these three countries was consistently faster than in Latin America and Caribbean as a whole until 2009 (World Bank, 2012). 20 From 2003 until 2009 inflation in Costa Rica ranged around a relatively high ten percent. During that period, the Dominican Republic suffered from high volatility in its CPI, having increased by over 50 percent in 2004, though steeply receding in the following years. Guatemala, in turn, maintained more moderate inflation, though the country did reach double digit inflation in 2008, when inflation accelerated throughout the region.

20 World Bank (2012). World data Bank. Accessed through www.worldbank.org.


95 | P a g e

Since 2009, inflation in Costa Rica and Guatemala has somewhat converged towards the Latin American & Caribbean average, which reached approximately five percent in 2011. In the Dominican Republic, however, inflation has consistently remained above that benchmark, at about 8.5 percent in 2011. Nevertheless, this level is relatively moderate in relation to the country’s recent history. In fact, since 1980, the Dominican Republic saw eleven years with double-digit inflation, five of which exceeded 40 percent. Maintaining price stability should constitute a priority for countries that tend to suffer from high inflation. Traditionally, CAPTAC-DR members have had their options seriously curtailed as their monetary policy has focused primarily on exchange rate stability and on managing monetary aggregates. Many countries in the region have implemented a rigid peg to the US dollar (e.g. El Salvador and Panama); or a (broadly defined) crawling peg (e.g. Costa Rica, Nicaragua, and Honduras). Even those countries with floating regimes have kept exchange rate stability as a priority concern. No country in the region has traditionally had a truly floating currency. More recently, Costa Rica, the Dominican Republic and Guatemala have seen a change of the primary approach of their central banks. Following the example of a number of other countries in the region, such as Chile, Colombia, Peru or Uruguay, they have switched to targeting inflation instead. Through this system, the central bank publicly announces a medium-term inflation target (or a band with minimum or maximum values). The monetary authority thus sets price stability as its top priority; as opposed to other policy priorities such as achieving exchange rate stability, maintaining levels of monetary aggregates, or implementing discretionary policies (Bernanke, 2003; Hammond 2012).21 Each monetary regime has its advantages and disadvantages. Different types may therefore be better suited for particular in-country conditions. The best-known advantages of targeting inflation include: (i) short term flexibility; (ii) anchoring inflation expectations; (iii) policy predictability & transparency; (iv) institutional continuity; and (v) ease of understanding for the general public. (Bernanke, 2003; Hammond 2012). In addition, IMF research has found that inflation targeting in developing countries contributed towards diminished expectations, lower inflation, & decreased price volatility (IMF 2005).22 9.3 CAPTAC-DR TECHNICAL ASSISTANCE Overhauling monetary policy by switching to an inflation-targeting regime is no easy task, particularly for countries with limited resources and/or central bank capacities such as Costa Rica, the Dominican Republic and Guatemala. Further to its ongoing dialogue with these three countries, CAPTAC-DR has deployed a number of capacity building initiatives seeking to help their respective central banks cope with the new policy approach. In particular, the Center has provided TA in areas such as (i) managing short-term liquidity, (ii) using macroeconomic models, and (iii) strengthening foreign exchange markets. A summary of CAPTAC-DR support work carried out in each of the three countries covered by this assessment is provided below:

21

Bernanke, Ben (2003). A perspective on inflation targeting: why it seems to work. Business Economics. Vol. 38:3. . Hamond, G. (2012). State of the Art of Inflation targeting. Bank of England: CCBS Handbook No. 29. Accessed on January 9 at: http://www.bankofengland.co.uk/education/Documents/ccbs/handbooks/pdf/ccbshb29.pdf. 22 International Monetary Fund. IMF World Economic Outlook, Summer 2005.


96 | P a g e

Liquidity Management As part of its implementation of an inflation-targeting monetary framework, the Central Bank of Costa Rica (BCCR) has gradually floated the national currency, albeit initially within a pre-established exchange rate band. In order to manage this process in a way that does not harm the economy or erode the very effectiveness of monetary policy measure, a gradual process to reach accurate liquidity estimations is required while embarking in open market operations. With this objective in mind, an STX recruited by CAPTAC-DR visited the country in 2010 in order to provide assistance to BCCR in issues relating to both primary and money market liquidity forecasting. As a result, the central bank managed to develop a robust system. Later, in June 2011, BCCR, redefined its (overnight) monetary policy interest rate. The bank’s board also decided to implement its own money market (called MIL) for intervening through auction arrangements to inject or mop up liquidity from the system as appropriate. CAPTAC-DR organized a mission by a specialized STX in order to support the Central Bank’s efforts in liquidity projections, as well as in managing monetary instruments for liquidity management. The STX proposed a number of improvements seeking inter alia, to provide more timely and accurate information; monitoring banking financial institution; modifying legal requirements on banks’ cash positions; improving auction mechanisms, enhancing daily discussions among key decision-makers at the bank; optimizing task allocations among bank divisions; considering a change in liquidity balance strategies; and improving communication with actors from the financial system. Many of these recommendations have been implemented, with promising preliminary results. In the Dominican Republic, the Central Bank (‘BCRD’) has traditionally targeted monetary aggregates (albeit with mixed historical results), and in January 2012 announced its transition to Inflation Targeting. In order to support the full implementation of inflation targets, a CAPTAC-DR STX visited the country in 2012 to introduce daily liquidity management techniques. As part of this work, specific requirements were identified in order to help BCRD adapt its practices to the requirements of this novel monetary policy framework. One year later, another STX carried out a diagnosis and proposed adjustments for daily liquidity management and overall financial infrastructure. The recommendations included switching from a passive to a proactive liquidity management approach, inter alia by: (i) liquidity needs, and preparing detailed information on a daily basis; (ii) organizing technical teams on liquidity projections; (iii) setting a daily communication routine with bank treasuries to gather information on liquidity needs, (iv) seeking to inform the determination of auction amounts; (v) introducing repo rates; (vi) reducing overnight deposit rates; (vii) promoting collateral-based inter-bank lending and (viii) improving overall communication with banks. Encouraging progress has been made in furthering this agenda. Though Guatemala traditionally pursued monetarist policies, the country is relatively more experienced in managing inflation targeting, as it adopted the system as early as 2005. As expected, the country still finds itself in a process of fine-tuning its monetary approach. This type of monetary policy particularly requires effective daily management of liquidity. A CAPTAC-DR STX visited the country in late 2011 in order to carry out a diagnostic and propose recommendations to the Central Bank for steering short-term liquidity in the country. In order to improve liquidity management the STX has suggested, inter alia, to (i) use overnight rates as the target rate; (ii) follow inter-bank lending rates on real time; (iii) develop a money market for financial institutions; (iv) promote repo operations; (v) change the schedule/time of the day of liquidity auctions and (vi) gradually reduce the maturity of the instruments. In order to improve liquidity estimation a new methodology was proposed. It was also suggested to develop fluid communication channels among financial actors.


97 | P a g e

Finally, the STX suggested continuation of the process of securities dematerializaton. A number of these recommendations have been implemented, with excellent preliminary results. Other recommendations will require additional time because of their technical complexity. Macroeconomic Modeling and Forecasting Countries with explicit inflation targets require a series of quantitative tools to understand and to predict the behavior of key macroeconomic variables. In particular, several countries worldwide use Dynamic and Stochastic General Equilibrium (‘DSGE’) models as a tool for developing monetary policy scenarios. These models provide a detailed account of transmission mechanisms and therefore allow for a better understanding of the impact of economic shocks. They also allow estimating a number of variables such as natural unemployment rates. CAPTAC-DR TA found that the models formerly used at Costa Rica’s central bank were robust and useful, but did not account for a number of distinctive features of the country’s economy such as its high level of financial dollarization. An STX therefore supported BCCR in developing a more complete DSGE model, representing a small open economy with nominal rigidities (and including a number of characteristics specific to the country). This work involved formulating all equations, codifying and calibrating all parameters, and carrying out various simulations. In addition, a work plan was established to determine how robust the model predictions are, as well as to fine-tune the system further. BCCR staff was deemed technically solid. The Dominican Republic is also developing models with support of a CAPTAC-DR STX. This expert undertook two missions to Santo Domingo respectively in 2010 and 2011. He expressed concerns on the progress of recommendations relating to upgrading Central Bank human resources for developing a DSGE model. Therefore, it was decided to create a more adapted version (i.e. semi-structured), with support from the STX. This model is effective enough to suit practical applications and constitutes the departing point for developing more sophisticated tools. In this regard, further support is likely to be needed. Another CAPTAC-DR STX visited the BCRD in two occasions, seeking to help complete the country’s “toolbox” for implementing inflation targeting. The CAPTAC-DR intervention supported the development of the above mentioned semi-structural econometric model (along the lines of that used by the Colombian Central Bank). This model seeks to attain endogenous cyclical forecasting, not only to assess policy impact in a more accurate manner, but also to simulate the impact of various external shocks. As part of the visits, the STX made a diagnostic of modeling/programming capacity within BCRD, as well as concrete recommendations. The specifications of the model and the key parameters have been set. The STX has recommended a follow up missions to complete the development of this model before engaging in the testing phase as well as to migrate to a different platform. Guatemala also benefitted from CAPTAC-DR-RD’s support in the area of Macroeconomic modeling. An initial mission by a Chilean STX helped Banco de Guatemala (‘BANGUAT’) to delineate a Structural Macroeconomic Model (i.e. a DSGE-type). During a second visit of the consultant, models of various complexities were assessed. This work also provided the opportunity to build BANGUAT’s capacity in analyzing model results, and using them for economic forecasting. It was proposed to allocate resources to maintain model building/refining and usage on a permanent basis. This work would require designing a model development project with well-defined phases. The STX also suggested BANGUAT to formulate a research agenda on the basis of its modeling work. Though human resources did not appear to be a limitation, project


98 | P a g e

management and communications within the bank were perceived as areas in need of strengthening in order to optimize further progress. Furthermore, a sophisticated model used by the Colombian Central Bank was described in detail to Guatemalan counterparts. At the same time, structural macroeconomic models being used by BANGUAT were assessed and refined. In fact, problems were found with issues such as surveying calibration, aggregation, and specific equations. The CAPTAC-DR STX and BANGUAT officials jointly solved many of these problems and identified areas for further work and research (e.g. the fiscal sector, investments, etc.) The work on modeling was carried out in coordination with ECLAC/Mexico as part of a first phase and second of a program aiming at improving economic forecasting techniques at the region’s central banks. The models developed under this scheme are used for short-term forecasting and their results feed into the models supported by CAPTAC-DR. Exchange Rate Market Development In order to support Costa Rica’s transition from an exchange rate band to a fully floating modality (as part of the adoption of inflation targeting), a CAPTAC-DR STX advised BCCR in 2010 on ways to foster the development of the foreign exchange market, while strengthening its own intervention capacity. In particular, recommendations were provided in order to create a functioning wholesale market and to replace former currency band intervention, with a rule-base protocol tracking short-term volatility. In addition, suggestions were provided for promoting a derivative market and a single money market, as well as to improve liquidity monitoring (including the calculation of reference rates). Furthermore, CAPTAC-DR commissioned Professor R. MacDonald, a world-class authority in the subject, to produce a tailor-made study for Cost Rica on determining its equilibrium exchange rate against the backdrop of current practices at BCCR. This study not only informed Central Bank authorities about international best practice in this regard, but also provided concrete pointers as to ways for BCCR to improve its working methods. In particular, the report provides specific recommendations and methodologies in order to estimate the Behavior Equilibrium Exchange Rate (‘BEER’) and the Fundamental Equilibrium Exchange Rate (‘FEER’) for Costa Rica’s particular case. Guatemala has also been gradually implementing explicit inflation targeting. As part of this effort, the Central Bank of Guatemala has adopted a transparent intervention strategy seeking to tackle excess volatility. BANGUAT also received support from CAPTAC-DR in improving the efficiency of its interventions at the foreign exchange market. A STX assessed current practices in the country and suggested actions to move towards a more flexible exchange rate, while improving market depth and liquidity. This in turn would help reduce exchange rate volatility. Advice was also provided both on the development of derivative markets and for BANGUAT to take a more proactive overall stance.

9.4 SUGGESTED LOGICAL FRAMEWORK

Project

Inputs / Activities

(CAPTAC-DR Only)



Short-term (up to 1 year) Medium-term (2-3 years) Long-term (4-10 years)

Support to monetary operations in (CIR, DOM & GTM)

I/P: LTX / STX Backstopping & consulting time; travel costs; overheads Activities: • Missions; • Back-

stopping; • workshops; • mentoring; • training.

Liquidity Management (LM) a) Diagnostic & recommendations b) Enhanced liquidity forecasting capacity

and systems in place. c) Enhanced management systems in

place (e.g. auction times adjusted, interbank repo lending in place, etc.)

d) Improved communication sharing with banks & other actors.

Macroeconomic Modeling (MM) e) Diagnostic & recommendations. f) Staff trained/sensitized on model

formulation & use. g) New models, existing models refined. Exchange Rate Mgmt (ERM) h) Diagnostic and recommendations. i) Central bank staff trained BEER & FEER

methodologies. j) Strategy for forex market intervention.

LM: central banks (CBs) carry out daily liquidity forecasts which inform their open market operations. MM: CBs develop, update and use macro-economic models. ERM: CBs execute revised protocols for intervention in the forex market. Indicators: a) Forecasting carried out daily as per

suggested methodology. b) Open market operations carried out

as per STX’s suggestions. c) Central Banks refine own models

without need for outside support. d) Models used for scenarios

presented to bank decision-makers. e) Intervention protocols respected.

CB institutional capacity strengthened to manage liquidity, use macroeconomic models & intervene in forex markets Indicators: a) CB officials self- assessment. b) Expert opinion of CAPTAC-

DR/IMF. c) Opinion of financial sector.

a) Inflation rate and inflation volatility fall.

b) Exchange rate volatility is equal to or lower than Latin American averages.

Indicators: a) Consumer Price Index. b) Exchange rate.


STX & workshop reports

a) CAPTAC-DR reports b) Physical inspection c) Interviews with treasury officials

CAPTAC-DR reports

a) CAPTAC-DR reports b) Interviews/surveys of CB

officials + banks

a) Nat’l accounts. b) IMF statistics.


Local condi-tions allow for the activities to take place

a) Political will to make inflation targeting a government priority

b) Sufficient resources available for project activities

a) Adequate absorptive capacity b) Sufficient resources assigned c) Buy-in of CAPTAC-DR

recommendations d) Political support to intervention

protocols

a) Absorptive capacity b) Collaboration from financial

sector actors c) Resources available

a) No inflation inducing shocks.

b) Overall monetary policy managed efficiently.

c) CB independence.

(*) Note: On the basis of the information reviewed, it is estimated that output completion varies among individual countries. Costa Rica and Guatemala appear to be most advanced whereas the Dominican Republic shows a satisfactory evolution. In general work has advanced most on (i) liquidity management (with a series of STX recommendations already adopted and showing promising results) and on (ii) macroeconomic modeling (with a series of models built and/or refined and in use by the Central Banks). The work on exchange rate management does not cover the Dominican Republic.

9.5 EVALUATION RATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities Consistency of activity with Program Document, CAPTAC-DR strategy, and approved work plan.

Fully consistent with the project document and CAPTAC-DR strategy

Consistency of intervention(s) with national/regional priorities: Intervention identified as a Government priority across the region o Extent to which activity is linked to and supports

national/regional strategy Fully consistent with local monetary policy plans



Work has been carried out at the right institutional level. Guatemala and Costa Rica appear to have particularly strong ownership of the TA.


Sequencing is appropriate


Local capacity appears adequate for this particular initiative


Appropriate


Good coordination has been instituted with ECLAC/Mexico. No other donors appear to be active on monetary policy in the region.


No other donors appear to be active in this area. In addition, CAPTAC-DR maintains an excellent dialogue with central banks, both at the regional and national levels.


N/A


The intervention’s impact is likely to be achieved particularly on liquidity management and marcoeconomic modeling, although it may require longer-term support.


Outcomes are likely (although their accomplishment may require longer than expected). Outcomes are less likely as regards exchange rate management.


High



Overall, quality of advice and analysis has been very high.

Score for Outputs 3.4 Good EFFICIENCY


101 | P a g e

Key Evaluation Questions Comments Process and implementation efficiency TA/workshop design has been efficiently carried out; Not applicable Timeliness in executing the activity; So far the pace of progress appears appropriate in all countries Appropriateness/effectiveness of IMF’s internal management of the activity;

Appropriate


Right counterparts chosen


Good

The efficiency of planning and executing the TA/training delivery. No information available Efficient use of resources (human and financial) and attention to cost effectiveness

Appropriateness of staffing composition for the intervention; Appropriate human resources have been deployed Appropriateness of work allocation between HQ and RTAC in relation to activity;

The relative work allocation appears appropriate




The activity appears cost effective. There are efficiency gains in relation with organizing similar work in several countries simultaneously




Reporting appears patchy


RA and back-stopper officer have no access to TAIMS. This solution is used for administrative purposes only


RBM has been extensively applied to this intervention as it was not foreseen, by most of its elements are implicit

Use of RBM to manage activity; RBM does not appear to have been extensively applied to this intervention Incorporation of lessons learnt in project design and implementation.

Good use of letters announcing ToRs as monitoring mechanisms which also helps in internalizing lessons learned.


At least some of the outcomes are likely to persist given the good absorption capacity and the high relevance of the CAPTAC-DR intervention


Central bank officers in Costa Rica and Guatemala appear to have developed the capacity to transmit their knowledge at a high technical level


Likely

o E.g., in-house training capacity built, where relevant. Likely For interventions designed to deliver sustainability, financial sustainability has been achieved

N/A







102 | P a g e


DAC criteria


score Weight

Total score

Rating


3.9 60% 2.34


3.5

20% 0.70

Co-ordination with Development Partners 3.7 20% 0.74 3.78 32% 1.21 Excellent Effectiveness Impact - - Outcomes: TA 3.0 30% 0.90 Outcomes: Regional capacity building 3.1 30% 0.93 Outputs 3.0 40% 1.20 3.03 28% 0.85 Good Efficiency Process and implementation efficiency 3.3 40% 1.32 Efficient use of resources (human and financial)


Monitoring and reporting 3.0 20% 0.60 3.32 22% 0.73 Good Sustainability Sustainability of RTAC/TA activity 3.0 75% 2.25 Contribution to building sustainable regional TA,

implementation capability 3.0 25% 0.75

3.00 18% 0.54 Good

TOTAL 3.33 Good 9.6 KEY FINDINGS AND RECOMMENDATIONS 1. The rate of progress towards successfully achieving project goals varies according to countries and technical areas.

CAPTAC-DR has succeeded in adjusting its work to reflect the speed, political situation and absorption capacity of each country. This level of progress is commendable, given the relatively few resources mobilized under this project.

2. Given this disparity, secondment arrangements between countries would be highly encouraged as a way to foster the acquisition of lessons learned across the region.

3. The technical areas considered in this assessment have been grouped because of their relationship to inflation targeting. They have been run as “initiatives” rather than “projects”. The use of RBM, is strongly recommended. In all cases, detailed work plans should be prepared.

4. The relevance of all three projects is extremely high; particularly as few other development partners work with monetary policy in the region.

5. The levels of capacity and working facilities seems appropriate for modeling and liquidity management. Implementing high cost or high complexity recommendations must be evaluated in detail (e.g. establishing derivative markets).

6. It is unclear whether central banks have developed in-house training capacity that can build on the procedures manuals and training of existing staff and taxpayers provided as outputs

7. The encouraging preliminary results from across the region could be leveraged for sensitizing countries which are less advanced in the area (e.g. Panama).

8. Different STXs were deployed at different points of time and in each country. Using the services of one person for the whole project in both countries might have yielded additional efficiency gains.

9. Although impressive progress has already been achieved, a long-term support is likely to be required. In particular, support may be needed for fine-tuning the system and solving glitches which are likely to arise regularly.


103 | P a g e

10. In general, the detailed recommendations provided during CAPTAC-DR missions are technically sound. In some cases, they should be screened according to their suitability for in-country conditions (e.g. exchange rate). CAPTAC-DR should ensure that support is provided to member countries seeking to allow them to implement the recommendations.

11. The RA has produced a recommendations matrix, which is sent to the counterparts regularly. This practice helps maintain the momentum of the reform process and should be generalized to the extent possible.


104 | P a g e

10. REGIONAL: NATIONAL ACCOUNTS STATISTICS 10.1 INTRODUCTION AND SUMMARY OF RESULTS The above project was selected for desktop review, in accordance with the methodology set out both in our technical proposal and in the Evaluation’s inception report. This review has been carried out on the basis of: Project-related documentation available at CAPTAC-DR (Mission reports,

annual reports to the Steering Committee, CAPTAC-DR’s website, briefing notes, BTOs, etc.); and

Survey feedback relating to the project. This initiative has helped modernize the implementation of national accounts across CAPTAC-DR’s member countries. By introducing modern statistical methods, this TA has changed the way business is conducted in a number of central banks and/or statistic bureaus in the region. Through a series of advisory missions and training events, the Center has helped its counterparts produce higher quality National Accounts Statistics (‘NAS’) (both annual (‘ANA’) and quarterly(‘QNA’)), while also making them more comparable with those of other countries and at different points in time. The CAPTAC-DR TA has been extremely relevant, effective and efficient. Though prospects are also good, achieving medium- and long term-sustainability of this impressive TA program is a matter of high priority. 10.2 BACKGROUND In the aftermath of World War II, national accounts became the preeminent tools to measure a country’s economy. Since the relevant methodologies are both shared and consistent on a global scale, results are comparable among countries. National accounts statistics are also comparable between different points in time. Key indicators, such as GDP growth have become a universal and comparable measure of an economy’s evolution and its relative performance. Having emerged as the standard measurement tools, national accounts also have a direct impact in the real economy, for example by way of inflation-indexed salaries or GDP-linked government bonds. Even the public’s expectations of the evolution of these variables translate into aggregate economic decisions. For example, expectations of high GDP growth may bring about increased investments. Inflation is another example, as most economists agree that inflationary expectations constitute one of the key determinants of actual inflation rates. Release of ANAs usually involves a relatively long time lag (normally about six months or more after year-end). For this reason, ANAs are not ideal for capturing short-term fluctuations. Nevertheless, these indicators give insightful glimpses on the structure and the evolution of a national economy. They also allow for interesting international comparisons. As such, they are particularly useful for unveiling long-term economic trends. In El Salvador, for example, there has been a multi-agency effort to help the country improve its ANAs. This is of key importance as CAPTAC-DR detected that GDP figures could be overstated by as much as 20 percent. Such inaccuracies can have enormous impacts on issues such as debt coverage or government deficit estimation.


105 | P a g e

QNAs, in turn provide more timely, albeit less comprehensive measurements. For this reason, QNAs help understand the business cycle and are useful inputs for sharpening the timing of policy responses. In fact, the recent volatility of the global economy has underscored the need for timely and updated data in order to identify areas requiring policy action. QNA timeliness, coherence, and accuracy therefore become key preoccupations. Nicaragua, for example did not have a set of reliable QNA. Without them the conduct of macroeconomic policy could be totally misguided. Fortunately, CAPTAC-DR’s TA helped the country publish more robust quarterly statistics using 1994 as the base year. In addition the country’s central bank is empowered by a new law that provides a solid framework for collecting data from the private sector. Within this overall framework the estimation of the evolution of various types of prices indexes such as producer’s prices, export and import prices or consumer prices also constitutes a key issue. The Dominican Republic, for example needed to develop control mechanisms to identify atypical unit values of its Export-Import Unit Value Indexes. Solid national statistics are required in order to translate such strategic goals into concrete action in an effective manner. Inaccurate information will lead to erroneous macroeconomic policies, which again can have a deleterious impact on the real economy and on people’s livelihoods. Given the time lags between the adoption of a particular policy and its actual effects, timeliness is also of the essence. Panama, for example, has emerged as a regional financial hub. Yet, the government was making inaccurate measurements of this sector’s contribution to the economy. In fact, the National Statistics and Census Institute (‘INEC’) was not making full use of indirect information such as employment data or household surveys to crosscheck measures. By incorporating this readily available data the quality of national statistics can be substantially improved. Costa Rica was having similar programs, but in this case it fell short due to misclassification of data related to the financial sector. In such cases CAPTAC-DR provides valuable help to calibrate NAS. The Center thus addresses an extremely important need, particularly as its member countries move towards a more proactive policy stance. 10.3 CAPTAC-DR TECHNICAL ASSISTANCE Since its inception, CAPTAC-DR received a number of requests for TA on the System of National accounts (‘SNA’). The Center reacted by including this line of business in its overall portfolio, while simultaneously recruiting a specialized RA. The goals of this initiative are three-fold: First, this work seeks to improve the quality, reliability and consistence of national accounts and price statistics across CAPTAC-DR-member countries, in line with best international practice. Second, the TA aims at supporting regional harmonization as well. Third, the Center’s initiative also supports the reconciliation and harmonization of national accounts with other statistical systems. In order to achieve these goals, CAPTAC-DR has applied a number of TA approaches. On the one hand, the Center has made available detailed information to its member countries on this technical area. Furthermore, it has provided hands-on training focused on real-life situations, both through highly practical exercises or learning the use of highly specialized software such as BENCH or X-13 ARIMA.23 Throughout the implementation of these activities, technical advice has consistently followed the prescriptions of the System of National Accounts 2008 (‘2008 SNA’), the IMF QNA Manual, the IMF Producer Price Index (‘PPI’) Manual, as well as the IMF Export and Import Price Indices Manual. Finally, training events have been further enriched by sharing the experience of other countries in the region and beyond, particularly as regards current practices and lessons learned. 23 These programs respectively deal with benchmarking and seasonal adjustment and were developed by the Canadian & American treasuries.


106 | P a g e

A series of short-term missions constituted the main vehicle for the delivery of the above-mentioned TA. The missions provided detailed recommendations for improvement, while also monitoring progress towards achieving expected outcomes. The list of TA missions is provided below and indicates, between parentheses, the specific technical area in which CAPTAC-DR TA was provided. Fiscal Year 2011 Costa Rica: May 2011 (PPIs), April-May 2011 (basic data sources, general advisory, diagnostics); Guatemala: June 2011 (QNA); El Salvador: February 2011 (general advisory, diagnostics); Honduras: June 2011 (general advisory, diagnostics); Nicaragua: January 2011 (general advisory, diagnostics), & May 2011 (XMPI)24; Panama: June 2011 (QNA). Fiscal Year 2012 Costa Rica: August 2011 (basic data sources), November 2011 (XMPI), March 2012 (sources), & April 2012 (IMEA); Dominican Republic: August 2011 (general advisory, diagnostics), September 2011(XMPI), October 2011 (sources),

December 2011 (XMPI); February/March 2012 (QNA, IMEA, financial sector); & April 2012 (PPI, sources). El Salvador: July 2011 (QNAs), September 2011 (follow up to previous missions), March 2012 (PPI, financial service

sector), April 2012 (SMPI), & May 2012 (PPI); Guatemala: September 2011 (XMPI), October 2011 (sources), October-November 2011 (IMEA), March 2012 (general

follow up), May 2012 (XMPI), & June 2012 (PPI); Honduras: September 2011(XMPI), October 2011 (QNA), January 2012 (QNAs, IMEA, financial sector), & April 2012

(XMPI); Nicaragua: September 2011 (XMPI), October 2011 (QNA, cash flow), November-December 2011 (QNA), December

2011 (PPI, XMPI), March 2012 (QNA), & June 2012 (ANAs, IMEA); Panama: December 2011 (QNA, IMEA, financial sector), January 2012 (sources, PPI), March 2012 (QNA, IMEA), &

April 2012 (ANAs, financial sector). Fiscal Year 2013 Costa Rica: July 2012 (basic data sources), & October 2012 (QNAs, agricultural activity and financial services sector) Dominican Republic: December 2012 (XMPI), & January 2013 (QNAS); El Salvador: August 2012 (ANAs, QNAs, financial services), October 2012 (ANAs, QNAs), November 2012 (XMPI), &

January 2013 (general follow up); Guatemala: August 2012 (IMEA), November 2012 (ANAs & QNAs), December 2012 (PPI), & January 2013 (XMPI); Honduras: February 2013 (information not reviewed); Nicaragua: June-July 2012 (XMPI), July-August 2012 (QNA, IMEA, PPI), September 2012 (ANAs), & January 2013

(XMPI); as well as Panama: July 2012 (ANAs, financial sector)., August 2012 (XMPI), October 2012 (ANAs), November 2012 (XMPI), &

December 2012 (financial sector).

24 A previous IMF mission (from HQ) took place in January 2008.


107 | P a g e

In addition, the following training activities took place: Fiscal Year 2011 Regional Seminar on Implementation of the System of National Accounts 2008 in Central America, Panama, and the

Dominican Republic.

Fiscal Year 2012 Regional Course on Annual National Accounts Statistics in the framework of the 2008 SNA; Regional Seminar on Services Statistics for National Accounts in the Framework of the 2008 SNA; Regional Course on Quarterly National Accounts Statistics; and Regional Course on Price Statistics. Fiscal Year 2012 Regional Workshop on Techniques to Analyze Economic Data in National Accounts Statistics. Initially, TA focused on supporting member countries in collecting information in accordance with the statistical techniques and guidelines presented in the IMF’s 2001 QNA Manual. Later on, the center’s work sought to help improve the source data, and coverage methods, both for QNAs and for ANAs. Particular emphasis was given to key indices such as (i) PPI; (ii) the export and import price index (‘XMPI’); (iii) the Export-Import Unit Value Index (‘XMUVI’); and the monthly index of economic activity (MIEA). More specifically, In Costa Rica, Guatemala, the Dominican Republic, and Nicaragua, the Center provided assistance in reconciling various data samples and unifying the systems in place for gathering and stratifying information, particularly as regards enterprises directories and their classification by economic activity. In the Dominican Republic and Guatemala, the International Standard Industrial Classification of all Economic Activities (revision 4) and the Central Product Classification (version 2) were also implemented, ensuring the proper classification by function and purpose. In El Salvador, CAPTAC-DR helped reconcile (i) new GDP estimates, (ii) the supply and use table, (iii) the employment matrix and (iv) integrated economic accounts per sector. In turn, Costa Rica, Honduras, El Salvador, and Panama, finalized the evaluation of output and value added of financial and insurance services, using an improved methodology for calculating financial intermediation. Updated samples have also been developed for the non-financial private sector In Costa Rica. Progress made under the QNA includes the revaluation of financial assets and the losses for bad debts in Costa Rica and Nicaragua. In addition, Guatemala, Honduras, Nicaragua, Panama and the Dominican Republic made strides in (i) evaluating source data quality, (ii) benchmarking methods, as well as (iii) seasonal adjustment methodologies. Nicaragua improved its use of foreign trade indices and began compilation of new QNA series. Additional areas of wok included the compilation of export and import unit value indices, as well as PPIs in a number of countries. 10.4 SUGGESTED LOGICAL FRAMEWORK The evaluators found no direct evidence of a logical framework analysis / RBM applied extensively to this particular initiative. The table in the next page proposes a sketch of the basic suggested contents applicable for such a framework:


108 | P a g e

Project

Inputs / Activities (CAPTAC-DR Only)




Medium-term (2-3 years) Long-term (4-10 years)

National Account Statistics

I/P: LTX / STX Backstopping & consulting time; travel costs; overheads. Activities: missions back-

stopping Training.

j) Diagnostic, recommendations & action plans

k) Updated listings of enterprises l) New base-years. m) Methods for annual & quarterly data

compilation for output and value added in key sectors (e.g. agricultural work-in-progress, processing third-party inputs, financial services, insurance, etc.)

n) Updated basket composition and weighing for relevant indices (e.g. PPI, XMUVI, XMPI, & MIEA etc.)

o) Central bank/statistic bureaus staff recruited as appropriate and trained in relevant statistical techniques

p) Advisory on gather of annual employment and compensation information

q) Software training (e.g. BENCH, X-13 ARIMA).

Enhanced sources of basic data, coverage, and methods allow for improved compilation of QNA & ANA

Capacity strengthened in all members to (i) gather and compile information and (ii) to disseminate nat’l accounts statistics in a timely manner, as per best practices.

Nat’l account statistics in CAPTAC-DR member countries are accurate, meaningful, timely and harmonized.

Indicators: f) Adoption of 2008 SNA recommendations in national accounts. g) Output and value added estimation for key sectors in progress. h) Business registrar populated & entries classified. i) New/improved indices implemented. j) Software packages in use across the region. k) Annual employment and compensation matrix is compiled and used. l) Data is seasonally adjusted. m) Country officials’ self- assessment as well as Expert opinion of CAPTAC-DR/IMF. n) Time lag for publication of statistics.


STX & workshop reports

d) CAPTAC-DR reports e) Interviews with Central Bank / Statistic

Bureau staff f) Survey of training participants

a) CAPTAC-DR/IMF reports b) Interviews with Central Bank / Statistic Bureau staff c) CAPTAC-DR/IMF expert opinion d) Direct inspection of national accounts


Local condi-tions allow for the activities to take place

c) Political will to make improved national accounts government priority

d) Sufficient resources available at central banks / statistic bureaus for implementing project recommendations

e) Adequate absorptive capacity f) Buy-in of CAPTAC-DR recommendations

d) Central banks / statistic bureaus can hire and retain qualified staff e) Sufficient resources available to central banks/statistic bureaus f) In-house training arrangements put in place g) Institutional memory properly managed & institutional continuity h) Political commitment to sound and timely national accounts

Work plans vary according to individual country diagnoses and agreed action plans. Implementation is progressing at a sustained pace and outputs are being attained. There are also good indications of attainment of short (and, in some cases, medium-term) impacts already. Further time is however required in order to make a more definitive assessment. See next page for further details.

Costa Rica Development of updated samples for activities of the agricultural and the non-financial private sector. Preliminary review of costs in agriculture, livestock, aquaculture & fishery for estimating monthly work-in-progress. Guidelines to compile financial intermediation services indirectly measured (‘FISIM’), central bank’s services, and

insurance and pension services in accordance with (2008 SNA). Analysis of monthly FISIM for 2010 and 2011. Processing and classification of the financial statements by institutional sector. Progress in the compilation of the quarterly flow of funds table and the financial account for 2011. Guidelines to compile a new export and import price indices (‘EIPIs’) with 2012 as base year. El Salvador Reconciliation of new GDP estimates, the Supply and Use Table (SUT), the employment matrix, and the integrated

economic accounts by sector. Analysis of SUT results and guidelines for improved compilation (informal activities) Guidelines for compiling quarterly & annual production accounts at constant prices by activity for 2006-2010. Compilation of export and import unit value indices (‘EIUVIs’).

Guatemala

Selection of a sample of 312 establishments for the monthly price surveys and preparation of work plan for collection. Assessment of the new classifications of economic activities and products for new base year and recommendations.

Honduras Completion of the evaluation of output and value added of financial & insurance services at current & constant prices. Improved methodology for calculating FISIM, central bank’s services and insurance and pension services. Assessment of source data quality, benchmarking methods used, and seasonal adjustment methodology.

Nicaragua Guidelines to compile FISIM, central bank’s services, and insurance and pension services as per 2008 SNA. Procedure for compiling the new MIEA consistent with the new quarterly series and selection of the breakdown level. Processing and classification of the financial statements by institutional sectors were completed. Progress in the calculation of revaluation adjustments and of losses for bad debts. Panama Guidelines for the compilation of the employment and compensation matrix for the period 2007-2010. Assessment of alternative production estimates of household informal activities according to available source data. Guidelines on FISIM complication and insurance services for 2007-2009 as per the 2008 SNA. Suggestions for using the implicit interest rates on deposits and loans in local and foreign currency. Research on the legal framework of subsidies in order to record them as per international standards. Assessment of the establishment sample and price indices of the Colon Free Zone.


110 | P a g e

Dominican Republic Compilation of EIUVIs. All Countries Assessment of source data quality for QNAs & MIEA, of benchmarking & of seasonal adjustment methodologies. Progress in updating the directory of enterprises as well as internationally standardized classification methods. Change of base year (for QNA and/or for ANA) and dissemination of newly compiled data. Progress towards an improved MIEA, QNA compilation, and revised PPI. 10.5 EVALUATION RATINGS Key Evaluation Questions Comments RELEVANCE Consistency with Program Document and Government Priorities Consistency of activity with Program Document, CAPTAC-DR strategy, and approved work plan.

Consistent with CAPTAC-DR strategy.

Consistency of intervention(s) with national/regional priorities: Intervention identified as a priority across the region o Extent to which activity is linked to and supports

national/regional strategy Fully consistent with government policy across CAPTAC-DR member countries. In particular all seven countries have made explicit plans to change base years and to determine MIEAs. o Extent to which activity is linked to and supports

sector/topical reform strategy; o Extent to which strong country/institutional ownership of

activity has been demonstrated. Work has been carried out at the right institutional level. Overall ownership is good. It appears to vary across countries (as indicated for example by the efforts put in place to implement CAPTAC-DR recommendations).




Local capacity appears adequate for this particular initiative as seen by its relative success.


Appropriate. The Western Hemisphere Department at IMF has lent its support to project activities, as has its statistical department.


Dialogue with donors is ongoing. The documentation mentions coordination with ECLAC but further details were not found. IADB has provided a grant to Nicaragua pursuant to project recommendations.


Although the documentation does not give details on the reasons for this choice, CAPTAC-DR member countries requested this TA vehemently.


No information is available.


The intervention’s impact is likely to be achieved in most participating countries, as seen by the results of the TA so far.


The project is likely to attain its outcomes in several countries, though national governments will need to maintain their commitment in the long run


111 | P a g e

Key Evaluation Questions Comments Significance of CAPTAC-DR contribution/likely contribution to developing core economic functions and institution building in the country/region, through the activity.

High



Output lists vary according to countries. Certain countries, such as El Salvador, have been slower in attaining these outputs. Nevertheless the project has already made impressive progress across the region.

Score for Outputs 3.6 Excellent EFFICIENCY Process and implementation efficiency TA/workshop design has been efficiently carried out; The limited number of inputs deployed suggests a highly efficient

implementation of project outputs. Timeliness in executing the activity; So far the pace of progress appears appropriate in all countries Appropriateness/effectiveness of IMF’s internal management of the activity;

Appropriate


Appropriate



The efficiency of planning and executing the TA/training delivery. No information available Efficient use of resources (human and financial) and attention to cost effectiveness

Appropriateness of staffing composition for the intervention; Appropriate human resources have been deployed. Appropriateness of work allocation between HQ and RTAC in relation to activity;

The relative work allocation appears appropriate.


No evidence of analysis of variances.


The activity appears cost effective. There are efficiency gains in relation with organizing similar work in several countries simultaneously.




Monitoring appears appropriate. Project documentation has not been easily retrievable in the CAPTAC-DR website.


N/A.


RBM does not appear to have been extensively applied though some elements are implicit. RBM seems to be present in documentation for HQ (Statistics Department).

Use of RBM to manage activity; RBM does not appear to have been applied as a management tool Incorporation of lessons learnt in project design and implementation.

No evidence of application of lessons learned was made available to the evaluators. Activities have been ongoing for a limited period of time.


Many outcomes are likely to persist given the good absorption capacity and the high relevance of the CAPTAC-DR intervention. In fact the intervention has modified the way many counterparts carry out their daily activities.


It is still early to assess this KEQ though the outlook does appear favorable.


112 | P a g e

Key Evaluation Questions Comments o E.g., absorptive capacity improved and developed to

sustainable level; Likely

o E.g., in-house training capacity built, where relevant. No information available. For interventions designed to deliver sustainability, financial sustainability has been achieved

N/A




This work appears to be a collection of national initiatives. The contribution towards sustainable TA implementation capacity seems moderate.



DAC


Weighted score

Weight Total score

Rating


3.5 60% 2.10


3.8 20% 0.76

Co-ordination with Development Partners 3.0 20% 0.60 3.46 32% 1.11 Excellent Effectiveness Impact - Outcomes: TA 3.4 30% 1.02 Outcomes: Regional capacity building 3.3 30% 0.99 Outputs 3.5 40% 1.40 3.41 28% 0.95 Good Efficiency Process and implementation efficiency 3.3 40% 1.32 Efficient use of resources (human and


Monitoring and reporting 2.7 20% 0.54 3.26 22% 0.72 Good Sustainability Sustainability of RTAC/TA activity 3.0 75% 2.25 Contribution to building sustainable regional

TA, implementation capability 3.0 25% 0.75

3.00 18% 0.54 Good

TOTAL 3.32 Good 10.6 KEY FINDINGS AND RECOMMENDATIONS 1. The rate of progress is commendable, given the relatively low level resources mobilized under this project. 2. Given the heterogeneity among member countries, attachment arrangements between officials from the region could

be organized in order to facilitate the sharing of both technical knowledge and work practices. 3. It is not clear how to track actual versus budgeted costs for individual, multi-year, multi country projects. Reports

aggregating all relevant project costs on a sub-topic by country would be a substantial improvement. 4. The levels of capacity and working facilities seemed appropriate. Nevertheless CAPTAC-DR should monitor this issue

closely as it is key for project success in the long-term. In fact, building both in-house and regional training capacity appears to be an important priority in the years to come.

5. The encouraging preliminary results from across the region could be leveraged for sensitizing countries which are less advanced in the area (e.g. El Salvador).