SAI’s role in development and use of key indicators for R&D evaluation

Timo Oksanen 3.4.2012

2

About SAI’s role in indicator development

Depending on the national mandates, the SAI’s role can be active or passive – or something in between

However, an active role in indicator development can endanger SAI’s independency and objectiveness

The NAO of Finland has not participated in Finland’s KNI development

Therefore, we have kept an outsider’s view to Finnish KNI-system

Findicator: The place to find up-to-date information on:

demographic developments in Finland

international crisis management

public expenditure

obesity among Finns

income differences

fish catches

Available to all at www.findicator.fi

3

Findicator is a comprehensive databank

4

Includes approximately 100 indicators of social progress

Up-to-date and relevant information on important societal issues

Indicators selected in consultation with user groups and data providers

Grouped thematically and by policy issue in line with Government Programme

5

R&D key indicators

In the Finnish KNI collection (findicator.fi), there are only two indicators that measure R&D outcome and effort:

Time series of patent applications

Time series of R&D expenditures, by sector

6

Patent applications in Finland, 1972 - 2010

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

1970 1975 1980 1985 1990 1995 2000 2005 2010

nu

mb

er o

f p

aten

t ap

pli

cati

on

s Foreign

Finnish private

Finnish businesses

7

R&D expenditures by sector, 1971 - 2010

0

1000

2000

3000

4000

5000

6000

7000

8000

1971

1973

1975

1977

1979

1981

1983

1985

1987

1989

1991

1993

1995

1997

1999

2001

2003

2005

2007

2009

mil

lio

n e

uro

s

universities

government+npo

business

8

On the basis of our R&D audits it can be said that…

1. A lot of work has been done (Slides 7-9)

http://www.tekes.fi/u/Better_results_more_value.pdf

2. Some things are just happening (slide 10)

3. A lot of things has still to be done….(slides 11-18). To put it shortly:

”Systemic change is a highly topical issue both in Finland and internationally. Climate change, aging of the population, advancing sustainable development as well as structural transformations in the economy and the related need to find new growth areas are current grand challenges. Responding to these challenges is not possible through individual small-scale reforms but calls for system-level changes”. http://www.tekes.fi/u/systeemisen_muutoksen_haasteet.pdf

9

Indicators related to economy and economic renewal

Phenomenom Indicators

National prosperity GDP per capita

Overall productivity of the economy Total Factor Productivity TFP

Productivity renewal indicator

Foreign Direct Investments Share of Foreign Direct Investments per GDP

Strengthening of intangible assets Share of new innovative products and services from business turnover.

Volume and share of intangible investments

Position in global value-networks Exports of knowledge-intensive sectors

Continuous improvement of competitiveness

Development of turnover in knowledge intensive sectors (or alternatively in KI jobs)

Collaboration, networks and knowledge flows

Share of public and private organisations having collaborated in innovation projects

Capability to innovate Development of patenting, registered trademarks and designs (EPO / USPTO / TRIAD)

Investments in R&D&I Share of R&D&I expenditure in business turnover

Government direct & indirect support to business R&D

Foreign direct investments in Finnish R&D&I

Human resources for R&D&I Availability of highly educated workforce

General conditions and incentives for R&D&I

GDP share of VC investments at different growth stages

9

10

Indicators related to well-being

Phenomenom Indicators

Health and quality of life Life expectancy

Well being in working life Share of 25 to 64-year-olds very or fairly satisfied with their current job

Healthy and safe living environment No indicator selected yet

New knowledge and competence associated with well-being

Scientific publications among the top 10% most cited publications worldwide as % of total scientific publications of the country

Innovations and systemic changes supporting well-being

New products, processes, services and social innovations

High-quality and innovative well being services Productivity of the social and health services of municipalities and federations of municipalities

Quality and extent of R&D&I activities directed towards well-being

Share of public organisations involved in health and well-being related R&D&I activities

Interorganisational collaboration related to well-being in value networks and the strengthening of flows of know-how

Mobility of researchers in the fields of health and well-being

R&D&I investments on well being Private and public R&D expenditure on well-being, health care and working life

Knowledge and human resources No indicator at present

Supportive operational environment Health and social care costs10

11

Indicators related to knowledge, education and culturePhenomenom Indicators

Competences and opportunities for life-long learning Education level of population

Education and active citizenship Interest in science, research and technology

Active and diverse cultural life Value added in the cultural sector

Openness, diversity and networks Share of foreign nationals in the human resources of science and technology

The quality and efficiency of the educational system OECD international student assessment - PISAThe quality and efficiency of higher education and research

Scientific publications within 10% of the most cited publications worldwide as % of total scientific publications of the country

Knowledge as a resource for the economy and society Patent applications by institutes of higher education and public R&D-institutes

Citizen participation Participation of population aged 18+ in lifelong learningActive and vital cultural life No indicator selected at present

Internationalisation and openness in research activities No indicator selected at present

Scientific research and education Share of doctors of the Human Resources in Science and Technology

Disseminating research information to citizens and the use of society

No indicator selected at present

Research and innovation activities related to culture No indicator selected at present

International mobility and cooperation in research Researcher mobility (inwards and outwards)

Investments in competences and human resources Investments in R&D activities in the public sector and in the higher education sector

Investments in competences and human resources Research personnel's share of workforce

Investments in general education and adult education Costs from adult education

Investments in the culture related to research and innovation

The Government R&D funding based on societal objective: culture

Investments in international cooperation and networking The share R&D expenditure from abroad in the Higher Education and Government sectors

11

12

Levels of the implementation

12

• Visual level = Graphics and visualisation of indicators

• Activity level = e.g. availability of the information behind the indicators

• Technical level = platform and connections, technical implementation of administration

• Definition level = Detailed indicator definition

• Data level = sourcing and updating of data

• Administrative level covers all the levels above defining the responsibilities and funding

Visual level

Activity level

Technical level

Data level

Adm

inistrative level

Data

Indicators

Definition level

13

Concluding remarks (1)

There is a clear need for discussion on research and innovation and the related societal development and challenges

The framework should allow for the decision makers to quickly assess the Finnish research and innovation development against the main international trends and key benchmark nations

The question of phenomena selection contains a political aspect

The set of indicators (including the impacts chains) needs continuous update on the basis of newest knowledge

As well, it should adapt to the changes in societal priorities and goals

International cooperation is needed to make existing and forthcoming efforts comparable

More knowledge needs to be gained on analyzing the impacts of research and innovation on well-being and the society 13

14

Concluding remarks (2)

Factors that need to be considered particularly when setting target values to indicators include:

Accumulation factor: how much emphasis is put on the current level of standing, what has been achieved and built over the past years or decades

Improvement and development factor: as compared to relation to peers or selected benchmarks

Systemic factor: how much emphasis is given to the existence and functioning of the research and innovation system (or ecosystem) as a whole, instead of its elements alone.

Global factor: how much emphasis is given to Finnish position and impact in global value chains, EU frameworks or in addressing global challenges, as compared to our success and impact within the country

Relevance factor: how well is the current research and innovation activity tuned for, prepared or adaptive to address the societal challenges, such as ageing of population, economic recession, etc.

14

15

A step to wider scope: where are we now?(Fred Gault: Social impacts of the development of the STI Indicators, 2011)

Indicator development is a dynamic activity. The OECD Innovation Strategy included a Measurement Agenda which is now being implemented. It includes intentions to:

i. Improve the measurement of broader innovation and its link to macro-economic performance;

ii. Invest in high-quality and more comprehensive data infrastructure to measure the determinants and impacts of innovation;

iii. Recognize the role of innovation in the public sector and promote its measurement;

iv. Promote the design of new statistical methods and interdisciplinary approaches to data collection; and,

v. Promote the measurement of social goals and social impacts of innovation”.

16

Special issues…(1)

Policy impacts (Gault, 2011)

As the indicators expand and policy makers recognize that innovation is not an isolated event, more attention is being given to the framework conditions and the policy mix that helps the system to work better. As more micro data analysis is done, the important result that the propensity to innovate in firms is higher than the propensity to do R&D will have more influence on policy.

17

Special issues…(2)

Funding of Finnish universities 2012 The model comprises in three main parts: education, research, and other education and science policy objectives. A total of 75% of the core funding will be allocated on the basis of a formula for education and research, of which 41% is based on educational factors and 34% on research factors. The remaining 25% of the core funding is based on education and science policy objectives.

Education-based funding criteria (as % of core funding): Master’s degrees awarded by the university (15 %), Bachelor’s degrees (9 %), the number of students completing a minimum of 55 credits (11 %, of which 3 % based on data produced by the student feedback system from 2015), credits completed in open university and non-degree studies (2 %), the number of degrees awarded to foreigners by the university (1 %), incoming and outgoing international student exchanges in the university (2 %) and the number of job-holding graduates (1 %).

18

… universities…(3).

The research-based funding criteria: doctoral degrees awarded (9 %), publications (13 %, of which 10 % international refereed publications and 3% other scientific publications - from 2015 the number of the Finnish Publication Forum classification levels 2 and 3 publications instead of international refereed publications and the number of level 1 publications instead of other scientific publications); competed research funding (9 %, of which 3 % international competed research funding and 6% other competed research funding); doctoral degrees awarded to foreigners (1 %); and foreign teaching and research personnel (2 %).

19

SAI’s role in development…?

Attribution/contribution/independence/capacity-building/resources:

So far the NAO of Finland has not participated in Finland’s KNI development

The Future: it is difficult or even impossible to separate the impact of SAI from other inputs and activities.

20

The Role of the SAI with the STI-indicators? i. besserwisser?

ii. Facilitator?

iii. Risk Manager?

iv. Measurement Technician?

v. Capacity

Builder?

(workshops 2011,

brainstorms 2012)

vi. Problem solver?

vi. Link between

R&D actors

and desicion

making?

- from measurement of results for contributor of the knowledge management of the whole R&D

- from peaces to the whole (the life cycle of the knowledge and knowledge management)

- from analyses to synthesises,

- from outputs to processes and capabilities of actors

- from mechanical synthesises to real problem solving

- …by co-operation with the R&D actors

The last steps at 2012: Findicators: An official network to coordinate the development of indicators (1)

to make propositions about developing the services and infrastructures of indicators (and knowledge management) in the government of Finland

to follow-up international development in the field

to co-operate with other developmental projects in this field, etc.

21

The spesific lines of developing R&D indicators in near future 2012 (2)

http://www.tekes.fi/u/Better_results_more_value.pdf

The Academy of Finland and Tekes will report until September 2012 to the Research and Innovation Council of Finland about the main lines of implementing the Framework (slides 9-11 above) (resources, administrative responsibilities and task, pilots (focusing single indicators) etc.

are politics able to utilise this kind of information?

how? when? why? where? how much it costs?

22