m&e in ppps · 2014-01-17 · management center (pimac), korea development institute nov. 2013...

Public and Private Infrastructure Investment Management Center (PIMAC),

Korea Development Institute

PIMAC Nov. 2013

Korean Evaluation System - M&E in PPPs -

Soojin Park

2

The Role of PIMAC stipulated by the National Finance Act and its decrees

1) Evaluator and/or Government Agency in Public Investment Management (PIM) :

Carry out ex-ante evaluation such as PFS (preliminary feasibility study)

Carry out intermediate evaluations such as RSF (re-assessment study of feasibility) and RDF (re-assessment of demand forecast)

Carry out ex-post evaluation such as IEBP (in-depth evaluation of budgetary program)

2) Researcher

Support for new initiatives of better PIM

Policy studies on PIM

Missions of PIMAC (Fiscal investment)

Overview of Korean PPP Part-01

Amendment

Jan. 2005

Revision

Jan. 1999

PPP Act, which was introduced in 1994 has gone through several

revisions.

PPP Legal Framework in Korea (I)

Enactment

Aug. 1994

『The Private Capital Inducement Promotion Act』

『The Act on Private Participation in Infrastructure 』

· Unsolicited proposals, Minimum Revenue Guarantee

『The Act on Private Participation in Infrastructure 』

· Diversified PPP-eligible Facility Types (35 -> 44)

· Introduction of BTL Scheme

Unsolicited proposals and MRG in 1999

- MRG abolished (unsolicited in 2006/ solicited in 2009)

BTL in 2005 : promoted its use in educational facilities, military residences,

environmental facilities, etc.

4

5

Investments through PPP increased from 1995 to 2007, since then it

continues to decrease.

MRG was abolished in 2006, and financial crisis occurred in 2008

As considerable level of infrastructure are equipped, growth ratio of PPP investment

began to decrease.

Track Record of Korean PPP - History

bil USD Number

(*) All the figures are contract basis and thus real investment balance may be different

6

Track Record of Korean PPP – Snap Shot

As of 2012, Korea had 633 PPP Projects either in construction or in

operation status, total value being 85 bil USD.

209 number of BOT and BOO projects with 60.4 bil USD of investment

424 BTL projects with 24.5 bil USD of investment cost.

mil USD

BTL BTO & BOO

mil USD

60.4 bil USD 24.5 bil USD


7

PPP has alleviated the government’s burden on infrastructure

investment from 5% to 36% per year. ( Average: 21% per year )

The promotion of PPP has helped ease constraints on the government’s financial

resources, enabling it to secure resources for sectors other than SOC.

PPP’s Contribution to Fiscal Budget


bil USD % of PPP

Monitoring & Evaluation in PPP Part-02

9

Monitoring & Evaluation by Project Cycles

Feasibility Test

Vfm test

Revised Vfm test

Macro level Project level

Intermediate & Ex-post

management

BTO/ BTL Performance

evaluation

Comprehensive ex-post evaluation • Macro-lev. evaluation • Ex-post VFM • Satisfaction survey

Ensuring macro economic sustainability and better PPP policies

Continuous Reporting

& Improvement

Feedback

Infra-info DB

Feedback &

Input

Fiscal Soundness Management Part-03

11

Macroeconomic Fiscal Risk Management

PPPs can accelerate infrastructure provision by mitigating the

problem of limited public resources

Government can benefit from the efficiency of the private sector and

invest in other areas by saving the expenditure

However, it is not appropriate to increase PPPs without limit

What is the sustainable level of PPPs?

How to measure the government debt caused by PPPs?

How to manage a PPP liability ceiling?

– Who evaluates the ceiling?

– When and how often evaluated?

– How to report to and get approval from the National Assembly?

Key Questions

12


Necessity of setting up a safeguard ceiling for PPPs

– May have a negative impact on fiscal soundness and stability

– Debt which should be paid by next generations (no free lunch)

United Kingdom

– Annual government payment for PFI : about 2% of total government

expenditure

– PFI investment : 10% to 15% of total public capital investment

Korea : Setting a safeguard ceiling on government payment

– Annul government payment : 2% of total government budget

expenditure

– PPP investment : 10% to 15% of total public investment

13

Simulation 4

• Examines whether total amount of public financing of signed and planned BTO plus BTL

projects exceeds the 2% ceiling.

• In Scenario 1, still less than 2%, but, in Scenario 2, higher than 2% .

Forecast on Ratio of Public Financing of Signed and

Planned BTO plus BTL Projects: Scenario 1 (selected)

Assumptions: 1. Calculation of amount of public financing of BTO: Based on agreements+Planned 2. Estimated period of BTL projects in Scenario 1: 10

years (3.8 tri.(2005), 7.3 tri.(2006), 5.5 tri.(2007), 5.0 tri.(2008) to be invested each year in 2009-2015) 3. Estimated government expenditure = (247 trillion

won in 2007, adjusted for 4% inflation)

Forecast on Ratio of Public Financing of Signed and

Planned BTO plus BTL Projects: Scenario 2

0.7%

0.6%0.5%

0.4%0.3%

0.2%0.1% 0.1%0.0%

0.8%0.8%0.9%0.9%0.9%1.0%1.0%1.0%1.0%

1.5%

1.8%

2.2%

1.5%

1.3%

1.6%1.6% 1.6% 1.6%

1.1%1.1% 1.1%

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036

Ratio of Public Financing of BTO Ratio of Public Financing of BTL Combined Ratio of BTO and BTL

1.9%

1.7%1.5%

1.4%

1.2%

1.0%

0.7%

0.5%

0.3%

2.0%2.1%2.1%

2.2%2.2%2.3%

2.4%2.5%

2.5%

1.5%

1.8%

2.2%

1.5%

1.3%

1.7%

2.0%

2.3%

2.5%

2.2%

2.4%

2.6%

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

3.0%

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036

Ratio of Public Financing of BTO Ratio of Public Financing of BTL Combined Ratio of BTO anc BTL

Source: Jay-Hyung Kim (2008)


14

Simulation Results


The amount of public financing for signed and planned BTO projects

is not yet at a level that can affect the stability of fiscal management.

If the government carries out BTL projects for a limited amount of 5

trillion Won as in Scenario 1, there would be no fiscal problem.

However, if the government carries out BTL projects for more than a

decade or longer at a cost of annual 10 trillion won in Scenario 2, it

may harm the government fiscal sustainability.

Scenario 1 adopted due to recommendation and market condition

Feasibility & Vfm test Part-04

Phase 1: Feasibility study (Decision to Invest)

The cost- benefit analysis is conducted to determine feasibility of the project from a national economy perspective.

Phase 2: Value for Money Assessment (Decision on PFI)

The government payment of PSC (Public Sector Comparator) is compared against that of PFI (Private Finance Initiative) to decide whether the PFI achieves VfM.

Phase 3: Formulation of PFI alternatives

Based on the results of phase 2, an appropriate PFI alternatives is formulated.

The level of project cost, user fee, subsidy scale, etc. are suggested by the government.

Bonus points (10% Phase 4: Award bonus points to the initial proponent max.)

awarded to the initial proponent are estimated based on the results of VfM tests

and the quality of the proposal.

Feasibility & VFM Test

17

Economic Feasibility

Estimate Benefit and Cost

Discount to PV

Ratio Analysis

Bench Marks (Comparable)

Benefit Estimation Cost Estimation

PV(Benefit) PV(Cost)

Social Discount Rate

CBR NPV IRR Payback

Period

Guidelines for Pre-Feasibility Study and Value for Money (VFM) test

Guidelines for RFP preparation

Guidelines for Standard Output Specification by facility

Guidelines for Tender Evaluation

Guidelines for Standard Concession Agreement

Guidelines for Refinancing

Objectivity, Consistency and Independence

Implementations in Transparency, Objectivity and Consistency as well

as Professional expertise are key elements

Independent PIMAC has developed PPP Implementation Guidelines

and updated reflecting changes in market conditions

Project Performance Evaluation in

Regular Basis

Part-05

Performance evaluation

Issues in PPP facilities

Acceptable service quality

Operational and cost efficiency

Cooperation between competent authority and private investor

Evaluation procedure (SOP)

Evaluation Planning

Operation Data Collection

Data / Document Review

Field Evaluation User Satisfaction

Survey Reporting

& Feedback

Preliminary Evaluation Field Evaluation


Evaluation Criteria

Degree of User Satisfaction : Safety, Customer Service, Kindness, etc.

Cooperation with Competent Authority : Feedback and Improvement,

Timely Reporting, VMS information support

Cost and Operational Efficiency : Operation Cost Management,

Organization Management, Public Relationship

Cost Efficiency

User Satisfaction

Cooperation Between Competent Authority and Project Company

Balanced Evaluation considering Trade-offs

Performance Evaluation

For fair performance evaluation, the competent authority must form a performance evaluation committee consisting of government officials, the project company (SPC or operator), and experts of the relevant field.

The project company should be allowed to first submit a self-evaluation report, which is reviewed by the competent authority.

The performance evaluation committee can decide whether to conduct an additional independent evaluation by a third party.

Performance Check List


User Satisfaction Survey

The satisfaction survey is conducted by each project company and submitted to the competent authorities.

It is taken yearly or quarterly as specified in the standard performance quality requirement.

Each evaluation item is given an evaluation grade(e.g., grade A-D) and then a score according to the grade. Weights are given to evaluation items to calculate final evaluation result.

Evaluation Criteria


Comprehensive Ex-post evaluation

-Macro level evaluation

-Ex-post VFM

-Stakeholder satisfaction survey

Part-04

Welfare Assessment (Based on 14 road construction projects)

It is noted that it took 2 years less to complete 14 road construction projects

using PPP procurement compared to those with TIPs (Traditional Investment

Procurements) as of 2006.

Socio-economic benefits (i.e. travel time reductions) caused by early completion

of the projects are estimated about 1.45 billion USD as of 2006.

Early Realization of Socio-economic Benefits

(W billion)

Comprehensive Ex-post evaluation ; Macro-level evaluation

As of 2006 1 year delay 2 year delay

3 year delay

4 year delay

623 1,455 2,472 3,301

Economic Growth Impact Assessment

The analysis of the ripple effects that the promotion of PPP projects could have

on the economy was conducted using KDI Macroeconomic Model.

Given that the government could have used the saved budget in other areas

thanks to the promotion of PPP projects, it is expected to have had the effect of

increasing the GDP by 0.198% in 2008.

Estimation of Growth Impact of PPP using the KDI Macroeconomic Model

Comprehensive Ex-post evaluation ; Macro-level evaluation

Ex-post VFM Test (BTO projects)

Ex-post VFM tests for BTO projects, for which concession agreements were

signed as of 2008, show that ex-post VFM has increased compared to ex-ante

VFM due to savings in construction and operational costs.

Realized VFM Increase in BTO using Ex-post VFM Test (%)

In the case of 11

BTO projects, the

difference between

the ex-ante VFM

and ex-post VFM

figures shows an

additional VFM

increase of 16.32%

(W 142.5 billion).

Comprehensive Ex-post Evaluation ; Ex-post VFM Test

Outline of survey (BTO projects)

In 2006, PIMAC of KDI began a survey of major stakeholders of BTO road projects currently in operation, such as competent authorities, project companies, and experts, about user satisfaction, project performance, and other issues.

To find out the level of user satisfaction, interviews and e-mail survey were conducted with considerable amount of users, and a face-to face survey was conducted with numerable public officials, project company employees, and experts related to BTO projects.

Outline of Survey on BTO Projects

Comprehensive Ex-post Evaluation ; Satisfaction Survey

Perception about Infrastructure Expansion by Utilizing Private Capital

Do BTO roads shorten travel time compared to alternative roads?

Are the Toll Levels on BTO roads Appropriate?

Results of User Survey

The results of the survey show that different groups of stakeholders have different perceptions about the performance of BTO projects.

In the user survey, those who use BTO roads were found to be largely satisfied with the services despite the high tolls.

Although the BTO roads provide shorter travel time compared with alternative roads, the reduction of tolls appeared to be the most important task to increase user satisfaction levels.

Comprehensive Ex-post Evaluation ; Satisfaction Survey

Satisfaction Level with the Performance

Results of survey of the project companies, competent authorities, and experts

While project companies and experts have a very positive perception of the results of BTO projects, competent authorities have a somewhat negative perception.

This can be attributed to the financial burdens caused by subsidies and MRGs and the additional administrative burdens from higher tolls and civil complaints.

Did the BTO projects improve VFM?

Are the Risks and Responsibilities properly distributed? Are BTO projects Attaining the Expected Benefits?

Comprehensive Ex-post Evaluation ; Stakeholder Satisfaction Survey

Infrainfo DB System Part-11

Infrainfo DB System

Development of DB System

The integrated DB system is essential for monitoring and evaluation.

Without the integrated DB system, the PPP data was managed by each

procuring authorities or municipal governments.

PIMAC began to develop integrated data base system (Infrainfo DB system) in

2011, and initiated the system in 2012.

Min. of Strategy & Finance

PIMAC

Min. of Land, Transport & Maritime Affairs

Min. of Culture, Sports & Tourism

Min. of Education, Science & Technology

Min. of Environment

Min. of Health & Welfare

Min. of Natl. Defense, etc.

Procuring Ministries

DB system support

In the line of continuous monitoring process, all the status changes of PPP

projects should be reported to MOSF and the information be delivered to

information center for updates to system (Infrainfo System).

Infrainfo DB System

Ex-post management

Infra-info System

Project Implementation

Reporting

Information delivery

Information input

Support with expertise

Feedback

Comprehensive Ex-post Evaluation

Feedback

Continuous monitoring & improvement

Intermediate & Ex-post Management Part-12

Even with successful institutional settings of PPP, Korea is now

facing new controversial issues and challenges that need to be

resolved in order to move forward to advanced stage.

So far, most of the government's efforts focused on improving the PPP

procurement process from project initiation to the construction stage.

Relatively little attention has been paid to the ex-post operational phase.

PPP Projects typically involve significant investment by the private

sector over a long period of time, and substantial changes of the

business environment or policy objectives may cause dispute.

Disputes are costly, time-consuming and lead to a breakdown of the

project.

It is important to monitor and resolve conflict as quickly, efficiently and formatively

as possible.

Intermediate & ex-post management

BTO BTL Q&A Total

2007 20 69 45 134

2008 29 170 101 300

2009 35 180 118 333

2010 44 104 143 291

Number of Inquiries and Consultancy by PIMAC

PIMAC is acting as an help desk for inquiry and consultancy;

Scope of Supporting Work : Reviews for request for proposals, concession

agreement, financial model, refinancing, Q&A


PPP projects undergo concession change for various reasons

throughout the project period.

Main reasons for change of the concession agreement include:

(1) Change in toll fee

(2) Change in total project cost

(3) Change in corporate tax

(4) Change incurred by alteration of the PPP Act

(5) Change in investor and refinancing

(6) Change concerning port labor union policy

(7) Recalculation of government payment (by adjusting to macroeconomic shocks)


Building capacity of government officials through training and education

to reduce occurrence of disputes

Training provides sound knowledge of PPP rules and regulations, comprehensive

and thorough understanding of standard. It improves communications with

contractors, stakeholders/users.

Revise Standard Concession Agreement (“SCA”) to provide detailed

guidelines for initial concession design. Fairness and transparency in

PPP projects, leading to reduce the disputes, can be enhanced by

developing SCA

Weakness of concession agreement result from their deficient design

Create a formal PPP dispute resolution committee within the government

to manage and assist operational PPP projects

The Committee must be neutral, unbiased and independent.

Preemptive efforts to monitor and reduce disputes


A PPP Dispute Resolution Committee has been newly established

under the Minister of Strategy and Finance as a means to resolve any

dispute involving PPP projects.

Newly created as Article 44 of the revised PPP Act (November 2011)

The Committee must be neutral, unbiased and independent in order to administer mediation, viable alternatives to transnational arbitration or litigation.

The Committee is composed of no more than nine members including one chairperson, of whom one or more shall represent: ① the government, ② the project company, ③ the public interest.

The Committee shall submit a written draft of mediation within 90 days

from the date of request for dispute resolution. This may further extend

up to 60 days.

Dispute resolution committee


Conclusion & Lessons learned Part-13

Vfm test

Comprehensive ex-post evaluation

Macroeconomic fiscal risk management

Continuous Reporting

& Improvement

Monitoring and evaluation results are fed back to enhance the quality

of service and to ensure the macroeconomic sustainability.

Plays a very important role in achieving development goals!

BTO/ BTL Performance

evaluation

Infra-info DB

Intermediate & Ex-post

management

Conclusion & Lessons learned

Project level

DB setup and update

Macro level


Developing monitoring and evaluation system is not an one-off job! The system development itself is a never-ending journey.

In the early stages, the effort should focus on project level evaluation and monitoring.

As number of PPP projects accumulates and project level evaluation system matures, the focus may move forward to comprehensive (macro) level.

The project information should continuously be reported to control tower and updated to a database, which is an important asset for next step.

The journey of monitoring and

evaluation system development

• Feedback and system improvement • Dispute resolution by cases


Trade-off attributes should be carefully considered to achieve the goal

of development when designing evaluation and monitoring criteria

The quality of facility can be represented by user satisfaction survey

The cost efficiency can be measured by ex-post VFM

VFM

User

Satisfaction

Evaluation drive direction 1

1) Choice with more priority to quality

2) Choice with more priority to cost savings

Evaluation drive direction 2

Institution with independence and capable expertise is a prerequisite

for monitoring and evaluation Professional expertise is needed to setup Standard Operation Procedure of

evaluation and monitoring

Operational experience and fiscal policy direction should be reflected in developing Evaluation Criteria, which can’t be the same for all PPP facilities, and thus properly revised for the practical application

The evaluation report should be reliable and independent so that third party audience can trust

PPP is not a free gift! It may shift burden to next generation.

The liability should be properly measured and controlled to keep fiscal soundness.

Monitoring and evaluation itself is not enough! The results should be duly reflected into future projects or PPP system.

With monitoring, various field issues and disputes should be properly and timely resolved to make projects moving forward.



Preventive efforts for continuous field monitoring and problem

solving are by far more important than ex-post evaluation.

Continuous field monitoring system can help solving problems in a timely manner.

Every stakeholder should recognize themselves as project improver through monitoring and suggestion.

To achieve this goal, education is one of most critical factors for success.

Field monitoring

Reporting Ex-post

Evaluation

Feedback with incentive or penalty

Problem Solving

Education

m&e in ppps · 2014-01-17 · management center (pimac), korea development institute nov. 2013...

Documents