aea ji hyun park

Developing Key Performance Indicators Developing Key Performance Indicators Developing Key Performance Indicators Developing Key Performance Indicators

for the R&D Programs using Logic Model for the R&D Programs using Logic Model for the R&D Programs using Logic Model for the R&D Programs using Logic Model

and Matrix Mapand Matrix Mapand Matrix Mapand Matrix Map

JiJi hyunhyun ParkPark, Researcher (, Researcher ([email protected]@kistep.re.kr))

Sang Sang kiki JeongJeong, Director (, Director ([email protected]@kistep.re.kr))

Dong Dong hoonhoon Oh, Research Fellow (Oh, Research Fellow ([email protected]@kistep.re.kr))

November 3 2006November 3 2006

R&D Industry & Infrastructure Evaluation Team of R&D Evaluation Center

Korea Institute of Science and Technology Evaluation and Planning (KISTEP)

Contents

II

IIII

IIIIII Challenges

Background for developing Standard Performance Indicators for the

evaluation of R&D programs

- Status of R&D program evaluation in Korea

- Necessity of performance indicators

Methodologies for developing Performance indicators and Key

Performance Indicators reflecting R&D characteristics

- Case studies for the development of performance

indicators using Logic Models

- Setting up Key Performance indicators on the Matrix

History of Research & Development program evaluation

1999Program evaluation was started by NSTC*

2006

2005

Performance based evaluation of R&D programs was

implemented with ‘Act on performance evaluation and

management of national R&D program’.

Program evaluation was output oriented.

It was a monitoring system to evaluate R&D programs

in a short term every year.

* NSTC : National Science and Technology Council

Status of national R&D evaluation in Korea

Ministries/

Research Councils

NSTC

(National Science and

Technology Council)

R&D

Centers

Funded

By Gov.

(R1)

Project

(R2)Project

R&D Program

Policy/Planning

Project

Technical Output

Program

Outcome & Impact

Project

(R3)

Social & economic

effect

Evaluation system of national R&D programs

Performance based evaluation of R&D programs

Meta EvaluationSpecific Evaluation Self Evaluation

Subject NSTC NSTC Ministries

Periods Feb ~ July Feb ~ AprMay

Evaluation for the achievement of

program objectives using

performance indicators

Summary In-depth evaluation issue

oriented and comparative

analysis

Evaluation for the results of self

evaluation

VisionVision

National policy level National policy level

goalsgoalsgoalsgoalsgoalsgoalsgoalsgoalsDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategiesDistribute key strategies

Develop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic modelsDevelop logic models

Program planningProgram planning

Ministrial program levelMinistrial program level

ProgrmsProgrms

AA BB . . .. . .

100%100% 50%50% %%

InitiativesInitiatives

Key programsKey programs

aaaaaaaa

bbbbbbbb

cccccccc........

MOSTMOSTMOSTMOST

....

....

Program PIProgram PI

Individual project Level Individual project Level

Main projectsMain projectsPrivate projectsPrivate projects

Performance indicators at the projects levelsPerformance indicators at the projects levelsPerformance indicators at the projects levelsPerformance indicators at the projects levels

. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .

Future

image

Future

image Policy goalPolicy goal Generic LMGeneric LM Standard PI(SPI)

Standard PI(SPI)Strategic plansStrategic plans Key strategies

(National plans of each areas)

Key strategies(National plans of each areas)

Mutual

Agreement

Overview of the national R&D performance measure system

Standard national R&D program classifications

• Basic Knowledge

- Pure basic, Specific basic

• Industrial Technology

- Short term, Mid-Long term

• Public Services/Region/etc.- Public service, Bio·Agricul·Envir, Region

Result utilization

Program objectives

• Research & Development

• Development of Human Resources

• International Cooperation

• Utility/Equipment. Etc.

• Guideline for the self-evaluation provides Standard Performance Indicators with R&D program classification.

National master plans of S&T in Korea

2003~2007

2002~2006

2000~2006

2005~2007

2004~2008

2006~2010

Periods

Environment

Bio·Health

Agriculture & Forestry

Region

Industrial technology

Basic research

Area

Ministry of Science &Technology, Ministry of Commerce, Industry and Energy etc.

Improving regional science & technology

Ministry of Agriculture & Forestry, Ministry of Maritime affairs & fisheries, Forest government Information Agency etc.

improving Agriculture & Forestry

Ministry of Health & Welfare, Ministry of Science &Technology, etc.

Fostering Bioscience

Improving Basic research

Ministry of Environment, Ministry of Science &Technology etc.

Development of Environmental technology

Ministry of Commerce, Industry and Energy

Ministry of Science &Technology, Ministry of Education & Human Resources Development

Innovation of industrial technology

Involved ministriesPlans

Example : Developing standard logic model from the national plans

(Bio area)

Back

ground

Input

Activity

Output

Out

come

Increasing

concerns on

bioscience

Weak

competitiveness

in technology

Securing genetic

resources and promoting utilization

Increasing

interests in bio

safety

Weak industry-academic cooperation

and information network

Gov.

investment

Private

investment

Human resource

in Bioscience

Facilities and

information system

Related institution and acts Real time analysis

and monitoring on

invest portfolio

국가적 차원의 종합

조정

Expanding R&D areas and

concentrating on selected areas

Strengthen

ing HRD

in BT

Supplying

research

facilities

Global

networking

System

maintenance

Coordination

& allocation Establishing

policies

Expanding R&D

in New tech and

fusion tech area

Developing HR in BT area

Construction of research centers & systems

Constructi-on of

systematic infra

Construct-ion of

internatio-nal

linkages

Fostering

venture,

small co.

Promoting

industrializ

ation

External

factors

Strengthening

cooperation

between co.

Active

network

Increase of

government

investment

Concentrating on new tech and mixed tech development

Building R&D hubs for global

networking

Creating high values

and enforcing bio

resources

Upgrade of policy

analysis and

evaluation system

Fostering industries

strategically

Strengthen public and

R&D infrastructure

Concentrating on selected BT

areas

Intensively fostering Bio

industry as a strategic area

Policy

Issues

Government

Policies

Uncertainty

of R&D

Social

interests

-Biotechnology promotion law

-Biotechnology promotion plan-5,1620 billion -9392(’00yr)

Strong

specialty

-7,7455 billion

Overall Coordination

Arranging

program

system

Fostering

intensively BT

industry

R&D Industrialization Infrastructure

Establish-ment of

systematic infra

Policy for

industriali

zation

R&D priority

setting and

coordination

Example : Performance indicators from the logic model (Bio area)

·Social cognition

about importance

of Bio-tech

·International

competitiveness

·Promoting application of future genetic resources

·Public concerns about

national security and

bioethics etc. related

with Bio tech

·Status of industry-

academic cooperation and

information network

·Gov/private investment

·Portion of investment of BT

in a whole R&D investment

·Human resources involved

in BT research area

·Ratio of high level HR

·Research

facilities and

information

systems

· Related system and laws ·Ratio of R&D budget

depending on various

standards

·Development high, new

and fusion technologies

(papers, patent etc.)

·Development of HR and research institutions in BT area·Construction of research centers/bio-complex·Building R&D hubs for international cooperation and attracting international co.·Protection of system maintenance such as national safety, bio ethics and intellectual property rights·Application of bio resources

·Fostering ventures and

small co.

·Utilization and

commercialization of BT

·Revitalization of tech transactions

business to business

·Success rates of utilization and

commercialization

·Rates of government

investment

·Market share by creating

high values

·Analyzing in a view of

industrial economy

·Utilizing systems such as

evaluation system

·Extending the share of the market (Effect of industry development)·Improving productivity (Effect of reducing costs)

·Informing about Bio tech to people

·Establishing c-GMP and bioplants

·Strengthening international basis for clinical study

·Certifying safety of LMOs

·Internal bio tech degree comparing with

developed countries

·Advanced bio technology

·Ratio of BT industry among the main

strategic industries

·Important strategic tech in BT industry

·Minimizing redundancy and

duplication

·Establishing related policies

·Upgrade of R&D capacity

·Diffusion of research results

(Public relations etc.)

·Improvement of specialties in BT area·Utilization of network such as information & facilities·Construction of international cooperation system·Standardization and internationalization of BT·Preservation of bio resources for the national benefits

Back

ground

Input

Activity

Output

Out

come

External

factors

Infrastructure

Policy

Issues

Government

Policies

Uncertainty

of R&D

Social

interests

국가적 차원의 종합

조정

Expanding R&D areas and

concentrating on selected areas

Strengthen

ing HRD

in BT

Supplying

research

facilities

Global

networking

System

maintenance

Overall Coordination

Arranging

program

system

Fostering

intensively BT

industry

R&D Industrialization Infrastructure

R&D priority

setting and

coordination

-Biotechnology promotion law

-Biotechnology promotion plan

Positioning R&D programs in a

group of area on the Matrix

Setting up key performance indicators

of each area and utilizing other area’s

indicators as indirect performance

indicators

Setting up Key Performance Indicators on the Matrix

Need for analysis of specific characteristics in different areas

Area

-Too concentrated on public & basic research

-Benefits are invisible in spite of investment

-Long-term planning is necessary

Aerospace & astronomy

-Influenced on international restriction such as national energy safety & the Climatic Change Convention

-Too strong on public research Energy

-Development of industry is very fast

-Progressed rapidly fusion interlaboratory

-Nongorvernment lead area

Information & Communication

-Too strong on public goods

-Influenced on the law, system and restriction

-A technology intensive industry

Environment

-No key trends due to various area involved

-low level infrastructureAgriculture & Forestry

-Many beneficiaries should be satisfied in a short time

-Public area needs for research on aging society

-Ongoing information industrialization is necessary to open health care information

Health care

-Basic industrial technology

-A technology intensive industry in various area

-Outputs are produced in short term

Machinery & Materials

-Government lead mass programs

-Strongly influenced by government on decision making such as construction of new town and highway

-Too concentrated on centralization of capital

Construction & Transportation

Technological & Industrial Characteristics

Example : Key Performance Indicators

• Performance indicators of Bio·Health area

• Technology (know-how) transfer• Strengthening health care technology• Increasing competitiveness of health industries • Information industrialization Information industrialization Information industrialization Information industrialization • Improve customer satisfaction for public service Improve customer satisfaction for public service Improve customer satisfaction for public service Improve customer satisfaction for public service

• Papers• Patents• Effect on making social & economic profits • Advances in technology development• Public based technology • Reinvigorate local economy• Productivity increase• Development of human resources in bio·health area• International cooperative research • Construction and utilization of research facilities

Key Performance

Indicators

Indirect Performance

Indicators

Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & Analysis of technical & industrial industrial industrial industrial industrial industrial industrial industrial

characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D characteristics of R&D areaareaareaareaareaareaareaarea

Key Key Key Key Performance Performance Performance Performance Indicators Indicators Indicators Indicators

Key Key Key Key Key Key Key Key Performance Performance Performance Performance Performance Performance Performance Performance Indicators Indicators Indicators Indicators Indicators Indicators Indicators Indicators

Setting up

effective

performance

indicators of each

R&D program

Setting up Setting up

effective effective

performance performance

indicators of each indicators of each

R&D programR&D program

Process of developing R&D program performance indicators

Logic model of Logic model of Logic model of Logic model of Logic model of Logic model of Logic model of Logic model of individual programindividual programindividual programindividual programindividual programindividual programindividual programindividual program

Standard Standard Standard Standard Performance Performance Performance Performance IndicatorsIndicatorsIndicatorsIndicators

Standard Standard Standard Standard Standard Standard Standard Standard Performance Performance Performance Performance Performance Performance Performance Performance IndicatorsIndicatorsIndicatorsIndicatorsIndicatorsIndicatorsIndicatorsIndicators

Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Logic model from R&D Policy & National Policy & National Policy & National Policy & National Policy & National Policy & National Policy & National Policy & National

master planmaster planmaster planmaster planmaster planmaster planmaster planmaster plan

Introduction of ‘R&D commercialization for Genome’ program

• Objectives :

- Establish infrastructure for national resources and high tech medical technology

- Promote advanced clinical technology through disease research

- Realize genomic medicine by the predictive and tailored medicine

• Period : 2001∼2010

• Budget : 1,220 billion

• Flow of activities

1111st st st st (01~03) 2222nd nd nd nd (04~07) 3th 3th 3th 3th (08~10)Commercialization of genomic

research

4th(11~15)

Simulation of commercialized

techniques

Genetic analysis related life and

disease

• Program contents

- Planning to analyze Korean genome

- Genome based R&D commercialization

- Development of applied and predictive technology

- Analysis of genes which are related with life cycle and diseases

Establishment of infrastructure for genomic research

Background InputInputInputInput ActivityActivityActivityActivity OutputOutputOutputOutput OutcomeOutcomeOutcomeOutcome

Government invest 1,220billion

National resourcesProblems

related aging society

Increasing average span of human’s life

Accomplishment of Genome projects

Diseases

Development of resource genetic informationEstablishment of base tech system

Forming cooperation networks

Accomplishing genetic information delivery

Developing disease related genetic markersAnalyzing genes generating disease from animals

Systematization of disease knowledge

Utilization of predictive tech

Construction of cooperation for diagnosis and medical treatment

Development of public health policy/program

Realization of utilizing techProtection disease in advance

InfraInfraInfraInfra R&DR&DR&DR&D UtilizationUtilizationUtilizationUtilizationStabilizationStabilizationStabilizationStabilization

Preservation of bio resources

Construction of information

Research on gene function

Results of disease research

Development of High /Predictive tech

Healthcare/disease management program

Logic model of ‘R&D commercialization for Genome’ program

National long-term plans (Genetic research fostering plan)

▶ Selecting important indicators among the performance indicators generated from program logic model and

deciding final program KPI by mutual agreement between the NSTC and program managers.

Genomics

Disease research

Development of high tech medical technology

Genomic medicine (Predictive, tailored medicine)

Performance indicators of ‘Genome based R&D commercialization program’

ActivityActivityActivityActivity

OutputOutputOutputOutput

OutcomeOutcomeOutcomeOutcome

National resources

GenomicsDisease research

Development of high tech medical technology

Genomic medicine (Predictive, tailored medicine)

•No. of developing genomic characters•No. of constructing information DB

•Results of research on gene function-papers, patents

•Heath care program management

•Rates of guaranteeing bio resources

•Rates of genetic information DB

•Rates of developing human resources-Specialized HR -Teaching materials

•Construction of international cooperation network

•Analysis of Korea-specific and target diseases genetic characteristics

•Accomplishment of genetic information delivery map

•Development of gene markers related specific disease

•Mapping of specific diseases related genes

•Degree of establishing basis for genetic information system

•Utilization of high/predictive tech-Developing program for disease prediction-Covering information system for direct response

•Rates of cooperating for developing diagnosis and medical treatment

•Degree of advance in public health policy/program comparing with developed countries

•Degree of utilizing medical tech

•Rates of early diagnosis of diseases

•Results of research on diseases-papers, patents

•Results of developing high/predictive tech papers, patents

• Logic Model was used to develop performance indicators of R&D

programs.

• Performance indicators of R&D program should be developed through

mutual agreement between government and each ministry to decide

whether program’s performance indicators reflect the national master plan.

• R&D programs were evaluated output oriented but now on it would be

evaluated focusing on the performance such as outcome and impact. So,

further research would be developing methodologies how to measure the

impact of R&D programs.

• Peer review and survey are necessary to verify the key performance

indicators of specific area and to decide allocation of them.

Challenges

aea ji hyun park

Documents

performance evaluation

key performance indicators

rd program evaluation

technology evaluation

research fellow

monitoring system

comparative analysisevaluation

korea necessity