composite indicators. methodology and applications to r&d...

64
1 Composite indicators. Methodology and Applications to R&D indicators by Andrea Saltelli EC, JRC [email protected] Workshop on Entrepreneurship Indicators: R&D and Technology Copenhagen, 30 and 31 October 2008

Upload: others

Post on 01-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

1

Composite indicators.Methodology and Applications to

R&D indicators

by Andrea SaltelliEC, JRC

[email protected]

Workshop on Entrepreneurship Indicators:

R&D and Technology

Copenhagen, 30 and 31 October 2008

Page 2: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

2

Joint Research

Centre

Page 3: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

3

Whereabouts

Page 4: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

4

Page 5: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

5

Methodology from: Joint OECD-JRC

handbook.

•5 years of preparation,

•2 rounds of consultation with OECD high level statistical committee,

•finally endorsed March 2008 with one abstention

Page 6: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

6

• Composite indicators’ fortune

Page 7: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

7

How many of them are around?

Searching “composite indicators”

Scholar Google October 2005 992

June 2006 1,440

May 2007 1,900

October 2008 3,030

Page 8: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

8

Use: an example, from the Economist web edition, October 15

Business travel ranking (The most and least attractive destinations for business travellers)Distance-learning MBAs(A ranking of the top distance-learning MBA programmes)Democracy index (The world's most and least democracticcountries)Operational risk ratings (Rating the risk of doing business in 150 markets)E-readiness (The best countries for conducting electronic business)Global house-price indices (Tracking house-price inflation around the world)

Page 9: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

9

Another example, about Mauritius, Economist October 16

THE 1.3m people of Mauritius love to prove famous people wrong. On independence from Britain in 1968, pundits such as a Nobel prize-winning economist, James Meade, and a novelist, V.S. Naipaul, did not give much of a chance to this tiny, isolated Indian Ocean island 1,800km (1,100 miles) off the coast of east Africa. Its people depended on a sugar economy and enjoyed a GDP per person of

only $200. Yet the island now boasts a GDP per person of $7,000, and very few of its people live in absolute poverty. It

once again ranks first in the latest annual Mo Ibrahim index, which measures governance in

Africa. And it bagged 24th spot in the World Bank’s global ranking for ease of doing business—the only African country in the top 30, ahead of countries such as Germany and France. How does it pull it off?

Page 10: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

10

Ideally, a composite indicator should be based on:

‐Solid theoretical framework,

‐Underlying data of good quality,

‐Tested methodology. 

When these conditions are met, it can be used for advocacy and analysis.

Saltelli, A., 2007, Composite indicators  between analysis and advocacy, Social Indicators Research, 81 , 65‐77. 

Page 11: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

11

• The steps to build CIs

Page 12: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

12

A good technical preparation for a CI can make it more robust (to uncertainties in data, weights,…) more resilient (remain relevant over time), more defensible (in dialogue with stakeholders…)

Page 13: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

13

Step 1. Developing a solid theoretical framework

What is badly defined is likely to be badly measured …

Examples …

The challenges are:• To integrate a broad set of (probably conflicting) points of view while keeping within a manageable construct. A model.  a storyteller a representation/narrative meaningful to a community of peers

Page 14: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

14

E‐business readiness (EC) 

Definition: Capability of a company to engage in electronic transactions with the objective of increasing business competitiveness.

Aim: evaluation of a composite indicator to monitor country progress in the implementation of the i2010 initiative...

Page 15: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

15

33 countries: EU‐27, US, JP,CH, IS, NO, TR

Summary innovation Index 

Innovation drivers, to measure the structural conditions required for innovation potential

Innovation & entrepreneurship, to measure the efforts towards innovation at the microeconomic level

Knowledge creation, to measure the investments on human factors and on R&D activities, considered as the key elements for a successful knowledge-based economy

Input Indicator

Innovation Drivers Knowledge CreationInnovation & Enterpreneurship

A theoretical framework consisting of 5 groups was discussed and agreed.

Page 16: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

16

Application, to measure the performance, expressed in terms of labour and business activities, and their value added in innovative sectors

Intellectual property, to measure the achieved results in terms of successful know how, especially referred to high-tech sectors

33 countries: EU‐27, US, JP,CH, IS, NO, TR

Output Indicator

Application Intellectual Property

Summary innovation Index 

Page 17: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

17

Step 1. Developing a solid theoretical framework

After Step 1. the developer should have…• A clear understanding and definition of the multidimensional phenomenon to be measured. • A nested structure of the various domains. • A list of selection criteria for the underlying indicators, e.g., input, process, output.

Page 18: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

18

Step 2. Selecting indicators

A composite indicator is above all the sum of its parts…

Excerpt: The strength of a composite indicator can largely depend on the quality of the underlying data. […]. The theoretical framework should guide the choice of the underlying indicators. The selection process can be quite subjectiveand therefore should involve stakeholders. 

Page 19: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

19

a1 - % of firms that use Interneta2 - % of firms that have web/home pagea3 - % of firms using ≥ 2 security facilitiesa4 - % of employees using computera5 - % of firms with broadband connectiona6 - % of firms with LAN

b1 - % of firms purchasing products/serv. via Internetb2 - % of firms receiving orders via Internetb3 - % of firms with IT linked with other internal ITb4 - % of firms with IT linked with external ITb5 - % of firms using Internet for bankingb6 - % of firms selling products via Internet

E-business readiness composite indicator

Adoption of ICT Use of ICT

Enterprise survey 2007 (Eurostat), yearly since 2003, Adoption and Use composite indicators

Page 20: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

20

a1 - % of firms that use Interneta2 - % of firms that have web/home pagea3 - % of firms using ≥ 2 security facilitiesa4 - % of employees using computera5 - % of firms with broadband connectiona6 - % of firms with LAN

Adoption of ICT

Page 21: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

21

b1 - % of firms purchasing products/serv. via Internetb2 - % of firms receiving orders via Internetb3 - % of firms with IT linked with other internal ITb4 - % of firms with IT linked with external ITb5 - % of firms using Internet for bankingb6 - % of firms selling products via Internet

Use of ICT

Page 22: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

22

Summary Innovation Index• Data from Eurostat Science and Technology Indicators and

Community Innovation Survey (CIS).

• A set of 53 indicators analyzed (FA)

• A reduced list of 26 indicators was discussed by developers and members of GSO (an expert group) with DG ENTR.

Page 23: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

23

Step 2. Selecting indicators

After Step 2. the developer should have…

• Checked the quality of the available data.• Discussed the strengths and weaknesses of each selected indicator.• Metadata

Page 24: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

24

Step 3. Multivariate analysis

Analysing the underlying structure of the data is an art …

a1 1 a2 0.85 1 a3 0.83 0.87 1 a4 0.77 0.81 0.89 1 a5 0.56 0.55 0.72 0.72 1 a6 0.64 0.61 0.66 0.73 0.56 1 b1 0.50 0.72 0.69 0.63 0.40 0.54 1 b2 0.61 0.80 0.75 0.74 0.62 0.54 0.86 1 b3 0.43 0.39 0.54 0.54 0.39 0.48 0.10 0.29 1 b4 0.35 0.26 0.41 0.30 0.31 0.37 0.16 0.24 0.75 1 b5 0.70 0.49 0.48 0.45 0.60 0.31 0.09 0.38 0.21 0.13 1 b6 0.55 0.75 0.78 0.77 0.59 0.57 0.77 0.83 0.44 0.24 0.29 a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5

Component Initial Eigenvalues

Total % of variance Cumulative % 1 7.242 60.35 60.35 2 1.523 12.69 73.04 3 1.178 9.82 82.86 4 0.554 4.61 87.47 5 0.512 4.26 91.73 6 0.385 3.20 94.94 7 0.242 2.01 96.95 8 0.131 1.09 98.04 9 0.098 0.82 98.04 10 0.064 0.54 99.40 11 0.043 0.36 99.76 12 0.029 0.24 100.00

0

1

2

3

4

5

6

7

8

0 2 4 6 8 10 12 14

Component number

Eige

nval

ues

Page 25: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

25

Step 4. Imputation of missing data.

The idea of imputation could be both seductive and dangerous …Almost all datasets contain missing data.

E-businessCountry a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6

AT 0.97 0.78 0.84 0.53 0.72 0.34 0.42 0.18 0.49 0.19 0.85 0.02

BE 0.97 0.72 0.82 0.59 0.86 0.45 0.43 0.18 0.49 0.18 0.87 0.02

BG 0.75 0.31 0.54 0.20 0.61 0.16 0.03 0.01 0.17 0.05 0.44 0.01

CY 0.88 0.47 0.67 0.42 0.69 0.20 0.12 0.07 0.44 0.06 0.54 0.01

CZ 0.95 0.71 0.77 0.40 0.77 0.31 0.22 0.09 0.31 0.08 0.87 0.01

DE 0.95 0.78 0.90 0.61 0.80 0.47 0.52 0.24 0.52 0.19 0.76 0.05

DK 0.97 0.84 0.93 0.80 0.39 0.36 0.33 0.62 0.21 0.93 0.04

EE 0.94 0.62 0.74 0.39 0.78 0.26 0.13 0.07 0.24 0.11 0.93 0.03

EL 0.93 0.60 0.52 0.37 0.72 0.35 0.08 0.06 0.71 0.01

ES 0.94 0.49 0.76 0.49 0.90 0.27 0.16 0.08 0.30 0.11 0.81 0.02

EU27 0.93 0.63 0.80 0.49 0.77 0.34 0.29 0.15 0.41 0.14 0.77 0.03

FI 0.99 0.81 0.97 0.70 0.91 0.47 0.19 0.15 0.53 0.15 0.91 0.03

FR 0.96 0.57 0.82 0.52 0.89 0.35 0.43 0.11 0.76 0.02

Page 26: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

26

Three common approaches to deal with missing data:

•case deletion (removes either country or indicator from the analysis)

•single imputation (e.g. Mean/Median substitution, Regression, etc.)

•multiple imputation (e.g. Markov Chain Monte Carlo algorithms).

Page 27: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

27

Step 5. Normalisation of data

Avoid adding up apples and oranges …

RankingStandardizationRe-scaling Distance to reference countryCategorical scales

Page 28: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

28

Step 6. Weighting and aggregation

The relative importance of the indicators can become the substance of a negotiation …

Weights based on statistical modelsPrincipal component/Factor analysis Data envelopment analysis Regression approach Unobserved components models

Page 29: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

29

Step 6. Weighting and aggregation

Weights based on opinions: participatory methods

Budget allocation Public opinionAnalytic hierarchy process Conjoint analysis

Page 30: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

30

Step 6. Weighting and aggregation

Weights based on Multi Criteria Analysis

To know more buy:

Page 31: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

31

Space of alternatives

Including/excluding variables

Normalisation

...

ImputationWeights

Aggregation

Performance index

Italy GreeceSpain

10

20

30

40

50

60

Step 7. Robustness and sensitivity

Uncertainty analysis can be used to assess the robustness of composite indicators …

Page 32: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

32

Monitoring e-Business Readiness of European enterprises

Sensitivity analysis of countries position

Page 33: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

33

To know more buy:

Or read: Saisana M., Saltelli A., TarantolaS. (2005) Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, Journal of the Royal Statistical Society - A, 168(2), 307-323.

Page 34: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

34

y = 0.7691x + 20.249R2 = 0.6979

40

50

60

70

80

90

100

40 50 60 70 80 90 100

Composite Le arning Inde x

Econ

omic

and

Soc

ial W

ell-b

eing

Step 8. Links to other variablesComposite indicators can be linked to other variables and measures

Comparing effectively complex dimensions: Canadian welfare and Canadian Composite Learning Index

Page 35: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

35

From LudgerWößmann, Contribution of Education and Training to Innovation and Growth, see:

www.education-economics.org

Page 36: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

36

Step 9. Back to the details

De‐constructing composite indicators can help extend the analysis …

Construction (sec F)

Wholesale and retail trade (sec G)

Provision of short-stay accommodation (sec H)

Motion picture, video, radio and television activities (sec O)

Transport, storage and communication

(sec I)

Real estate, Renting and Business activities (sec K)

Manufacturing (sec D)

All sectors

0.15

0.20

0.25

0.30

0.35

0.40

0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95

ICT Adoption

ICT

Use

All countries by sector 

Page 37: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

37

Monitoring e-Business Readiness of European enterprises

Enterprise survey 2007 (Eurostat), yearly since 2003, Adoption and Use composite indicators

2006Use Adoption

Page 38: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

38

AU

IL

SKEL LT

PT

LU

ATIEFR

NLBE

UKUS DE

JP

CH

DK

HU

CYES MT

EEIT

CZSI

IS

CA

HRPL

LVRO

BG

SE

FI

NO

TR

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0

Average grow th rate of SII (2003-2007)

2007

Sum

mar

y In

nova

tion

Inde

x

Sw eden Innovation leaders Innovation follow ersModerate innovators Catching-up countries Turkey

Dotted lines show EU performance.

THE 2007 SUMMARY INNOVATION INDEX (SII)

Page 39: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

39

Step 10. Presentation and dissemination

Page 40: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

40

COMPARATIVE ANALYSIS OF INNOVATION PERFORMANCE of THE

EUROPEAN COUNTRIES

0.08

0.23 0.23 0.24 0.25

0.25 0.26

0.26 0.27 0.29 0.

31 0.33 0.33 0.

350.

360.

360.

36 0.37

0.44 0.

47 0.47 0.48

0.48 0.49 0.50 0.

53 0.55 0.57 0.59 0.60

0.61 0.62 0.64 0.

670.

73

0.18 0.19

0.45

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

TR RO LV B GHR P L SK P T EL HU LT M TES CY IT SI CZ NO A U EE CA EU B E FR NL A T IE IS LU US UK DE JP DK IL FI CH SE

THE 2007 SUMMARY INNOVATION INDEX (SII)

Page 41: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

41

Why the ten steps?

To combat lack of transparency, which makes them and the associated the policy inference, arbitrary and objectionable.

The three pillars of a well-designed CI are:Solid theoretical framework,Underlying data of good quality,Tested methodology .

Methodology should include assessment of the CI robustness.

CI can be refuted by practitioners(examples of WEF GCI and Ecological Footprint)

Page 42: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

42

•No more methodology. The KEI study 

Page 43: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

43

Giuseppe Munda and Michaela Saisana

Page 44: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

44

Main research questions:

• Is it possible to measure the knowledge economy?• What are the drivers of the knowledge economy?• How does knowledge economy relate to other

complex dimensions?• Is it possible to reduce the total number of individual

indicators of KEI conceptual framework without loosing any relevant information?

• Are rankings useful at all for deriving policy suggestions?

Page 45: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

45

Figure 1. KEI Conceptual framework of a Knowledge-based Economy

Knowledge-based Economy Index

A1. Production and

diffusion of information and communication

technology (ICT)

A2. Human resources, skills and creativity

A3. Knowledge

production and diffusion

B2. Social performance

A4. Innovation,

entrepreneurship and creative destruction

B1. Economic Outputs

C1. Internationalization

Economic impact of ICT (A1a) Internet use by firms (A1b)

Internet use by individuals (A1c) Government ICT (A1d)

General education (A2a) Human resource in S&T education (A2b)

Skills (A2c) Creativity (A2d)

Mobility (A2e)

Research and experimental development (A3a) Patents (A3b)

Bibliometrics (A3c) Knowledge flows (A3d)

Total investment in intangibles (A3e)

Entrepreneurship (A4a) Demand for innovative products (A4b)

Financing of innovation (A4c) Market innovation outputs (A4d)

Organisational indicators (A4e)

Income (B1a) Productivity (B1b) Employment (B1c)

Environmental (B2a) Employment and economic welfare (B2b)

Quality of life indicators (B2c)

Trade (C1a) Knowledge production and diffusion (C1b)

Economic structure (C1c) Human resources (C1d)

Page 46: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

46

1. Production and diffusion of ICT

2. Human resources, skills and creativity

3. Knowledge production and diffusion

4. Innovation, entrepreneurship and creative destruction

5. Economic outputs

6. Social performance

7. Internationalisation

Page 47: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

47

We do not trust one model to build the Knowledge Economy Index, thus we opt for a “multi-modellingprinciple”

Page 48: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

48

Methodological scenarios for the development of the KEI composite

Aggregation function Additive

(linear) Multiplicative (geometric averaging)

Non-compensatory multi-criteria analysis

All (total 29) • • • Sub-dimensions included One-at-a-time excluded • • •

All (total 7) • • • Dimensions included One-at-a-time excluded • • •

Preserved • • • Pillar Structure Not preserved • • • z-scores • • Min-max • • Normalisation Raw data • Factor analysis • • • Equal weighting • • • Weighting Data envelopment analysis •

The frequency matrix of a country’s rank in each of the sevendimensions and the overall KEI was calculated across the ~2,000 scenarios~2,000 scenarios.

Page 49: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

49

Ran

k 1

Ran

k 2

Ran

k 3

Ran

k 4

Ran

k 5

Ran

k 6

Ran

k 7

Ran

k 8

Ran

k 9

Ran

k 10

Ran

k 11

Ran

k 12

Ran

k 13

Ran

k 14

Ran

k 15

Ran

k 16

Ran

k 17

Ran

k 18

Ran

k 19

Ran

k 20

Ran

k 21

Ran

k 22

Ran

k 23

Ran

k 24

Ran

k 25

Ran

k 26

Ran

k 27

Ran

k 28

Ran

k 29

Sweden 54 46Denmark 55 30 14Luxembourg 36 4 14 25 4 7 7 4Finland 18 23 29 9 11 11USA 11 32 2 4 39 9 4Japan 4 7 18 32 36 4UK 2 5 16 38 39Netherlands 86 4 4 7Ireland 4 61 14 4 9 9Austria 18 50 18 7 7Belgium 11 4 11 57 16 2France 4 14 18 11 54EU15 4 57 39EU25 4 4 14 32 39 7Germany 7 79 4 7 4Slovenia 7 41 38 14Estonia 4 36 25 21 11 4Malta 7 13 9 21 23 27Cyprus 36 7 4 23 23 7Spain 4 4 32 25 29 7Czech. Rep. 4 7 30 39 5 7 7Latvia 20 36 11 21 7 5Italy 29 18 9 29 9 7Greece 4 4 4 29 18 21 7 14Lithuania 4 41 13 32 11Hungary 2 13 13 57 2 14Portugal 4 4 7 11 61 14Slovakia 4 7 18 71Poland 100Legend:Frequency lower 15%Frequency between 15 and 30%Frequency between 30 and 50%Frequency greater than 50%

Knowledge Economy Index

Page 50: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

50

Median and associated 5th and 95th percentiles for the rank distribution

3 34

56 6

89

1011

1213 13

15

1718 18

19 19

22 2223

24 2426

2728

29

10

4

8

12

16

20

24

28

Swed

enDe

nmar

kLu

xem

bour

gFi

nlan

dUS

AJa

pan

Unite

d Ki

ngdo

mNe

ther

land

sIre

land

Aust

riaBe

lgiu

mFr

ance

EU15

EU27

Ger

man

ySl

oven

iaEs

toni

aM

alta

Cypr

usSp

ain

Czec

h Re

p.La

tvia

Italy

Gre

ece

Lith

ouan

iaHu

ngar

yPo

rtuga

lSl

ovak

iaPo

land

2004

Kno

wle

dge

Econ

omy

Inde

x (ra

nk)

Same results as before, different way of presenting them

Page 51: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

51

Spearman rank correlation coefficients between KEI and its seven dimensions (median ranks across ~2,000 simulations)

Prod

uctio

n an

d di

ffus

ion

of IC

T

Hum

an re

sour

ces,

skill

s and

cr

eativ

ity

Kno

wle

dge

prod

uctio

n an

d di

ffus

ion

Inno

vatio

n, e

ntre

pren

eurs

hip

and

crea

tive

dest

ruct

ion

Econ

omic

out

puts

Soci

al p

erfo

rman

ce

Inte

rnat

iona

lisat

ion

KEI 0.90 0.95 0.91 0.40 0.68 0.78 0.63 Production and diffusion of ICT 0.87 0.88 0.42 0.54 0.58 0.41 Human resources, skills and creativity 0.90 0.44 0.60 0.68 0.53 Knowledge production and diffusion 0.37 0.42 0.69 0.55 Innovation, entrepreneurship and creative destruction 0.17 0.06 0.04 Economic outputs 0.59 0.48 Social performance 0.61 Internationalisation 1.00 All coefficients are significant ( 29,05.0 =< np ).

All dimensions play a role but surely the less influential seems to be the ‘innovation, entrepreneurship and creative destruction dimension’.

Page 52: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

52

atbe

cycz

dedk

ee

esEU15EU25

fifr

gr

hu

ie

it jp

lt

lu

lv

mt

nl

pl

pt

se

si

sk

uk

0

50

100

150

200

250

1357911131517192123252729

KEI (median rank)

Gro

ss d

omes

tic p

rodu

ct p

er c

apita

in

purc

hasi

ng p

ower

sta

ndar

ds (P

PS)

Is KE relevant at all for a good overall economic performance?

The answer seems to be “YES”. Except Germany, Italy and Spain which have a high level of GDP per capita without any particular good performance on a KE.All the other high level GDP countries seem to perform well in the KEI composite (Luxembourg probably an outlier)

Page 53: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

53

at

be

cy

cz

de

dkeees

EU15EU25

fi

frgr

hu

ie

it

jp

lt

lu

lv mt

nlpl

pt

se

sisk

uk

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

1357911131517192123252729

KEI (median rank)

New

PhD

s (p

er th

ousa

nd p

opul

atio

n ag

ed 2

5-34

)

at

be

cy

cz

de dk

ee

es

EU15EU25

fi

fr

gr

hu

ie

it

jp

lt

lu

lv

mtnl

pl

pt

se

si

sk

uk

0.0

0.1

0.1

0.2

0.2

0.3

0.3

1357911131517192123252729

KEI (median rank)

Per

cent

age

of w

orki

ng a

ge p

opul

atio

n (1

5-64

) w

ith a

tert

iary

edu

catio

n (IS

CED

5 a

nd a

bove

)

at

becy

czde

dk

eees

EU15EU25

fi

fr

gr

hu

ieit

jp

lt

lulv

mt

nl

pl pt

si

sk

uk

0.0

0.1

0.1

0.2

0.2

0.3

0.3

1357911131517192123252729

KEI (median rank)

Par

ticip

atio

n in

life

long

lear

ning

per

wor

king

ag

e po

pula

tion

(25-

64)

While the number of Ph.Ds seems to play a role in explaining the success of a KE (all countries with an high number of Ph.Ds, except Portugal, are top countries in the KEI median ranking), the same result does not apply to percentage of working population with a tertiary education (very clear the case of Italy where the number of working population with a tertiary education is extremely high, but the number of Ph.Ds is small). Participation to life long learning seems also to be a success factor, although not for all top countries.

In sum, the human capital theory seems to be corroborated by the KEI results roughly.

PhD

LLL

Tertiary Ed.

Page 54: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

54

at

becy

cz

de dkee

es

EU15EU25

fi

fr

gr

hu

ieit

jp

ltlu

lv

mtnl

plpt

se

si

sk

uk

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

1357911131517192123252729

KEI (median rank)

Gro

ss d

omes

tic e

xpen

ditu

re o

n re

sear

ch a

nd

expe

rimen

tal d

evel

opm

ent (

GER

D) t

o gr

oss

dom

estic

pro

duct

(GD

P)

Countries which invest in research are top countries in a KBE. Research is a key driver for a KBE surely, thus the endogenous growth idea (the Schumpeterian theory) seems also corroborated.

Checking the Schumpeterian theory:

‘Gross domestic expenditure on research and experimental development’relates with KEI median.

Page 55: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

55

at

be

cycz de

dk

eees

EU15

EU25fi

frgr

hu

ie

it jp

ltlu

lv

mt

nl

pl

pt

se

si

skuk

40

45

50

55

60

1357911131517192123252729

KEI (median rank)

Env

ironm

enta

l Per

fora

mnc

e (B

2A, s

core

)

These arguments seem corroborated by the KEI measure. In fact, as one can see, no clear relationship between environmental performance and a KBE exists.

Is a KE an eco-efficient one?The Jevons’ paradoxteaches us that an ▲ in efficiency in using a resource leads, in the medium to long term, to an ▲ in consumption of that resource (rather than a ▼). No Kuznets curve …

Page 56: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

56

at

be

cy

cz

de

dk

ee

es EU15EU25

fi

fr

gr

hu

ie

it

jp

lt

lu

lv

mt

nl

pl

pt

se

si

sk

uk

0.00

0.02

0.04

0.06

0.08

0.10

0.12

1357911131517192123252729

KEI (median rank)

Lon

g te

rm u

nem

ploy

men

t rat

e

Does a KE influence unemployment?

In the short term technological progress and job creation are conflicting objectives (Kok report on Europe) but they might be compatible in the long period if a right balance (i.e. compromise) between flexibility and employment security is found.

Compatibility between technological progress and job creation. All top countries in the KEI measure are presenting an extremely low long term unemployment rate.

Page 57: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

57

atbe

cycz

de

dk

ee

es

fi

fr

gr

hu

ie

it

jp

lt

lu

lv

mt

nl

pl

pt

se

si

sk

uk

1

5

9

13

17

21

25

29

33

37

41

45

49

1357911131517192123252729

KEI (median rank)

Hum

an D

evel

opm

ent I

ndex

External benchmark:Human Development Index.

The relationship found seems again a corroboration of the human capital theories, in fact the correlation between KEI and the HDI is extremely high (though possibly spurious!).

Page 58: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

58

•JRC‐IPTS (Seville) work on companies  R&D expenditures scoreboard. 

Page 59: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical
Page 60: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

60

- The EU Industrial R&D Investment Scoreboard: analysis of 1000 EU and 1000 non-EU top investing companies in R&D

- The EU Survey of Business R&D- Economic and policy analysis of corporate R&D.

Industrial Research and Innovation at JRC - Seville http://iri.jrc.ec.europa.eu/

Page 61: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

61

Some Results (1): Nature of the R&D investment gap

88 % 12 %EU - US

Sectoral composition effect Underinvestment effect

Total: -1.8%

Breakdown of EU-US gap in R&D intensity into sectoral composition and underinvestment effects.

Note: In this figure, only companies of similar R&D size are considered, i.e. the top EU (391) and the US (563) with R&D investment above a common threshold (€23 million in the latest year).Source: JRC-IPTS calculations (2008) - Analysis of 2007 EU Industrial R&D Investment Scoreboard

- EU's R&D intensity deficit is largely explained by the different industrial structure (sectoral composition effect).

Industrial Research and Innovationhttp://iri.jrc.ec.europa.eu/

Page 62: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

62

- R&D rises productivity much more in high-tech sectors than in low-tech ones

Some Results (2): Econometrics of R&D & firm productivity

Industrial Research and Innovation at JRC Seville

-6-4

-20

-6-4

-20

-8 -6 -4 -2 0 -8 -6 -4 -2 0

High Low

Medium Total

Pro

duct

ivity

/Em

ploy

ee

R&D stock/employeeGraphs by R&D Intensity sector groups

R&D Stock/Employee vs. Productivity/Employee

Source: European Commission, JRC –IPTS (2008) Analysis of 2007 EU Industrial R&D Investment Scoreboard

Page 63: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

63

Industrial Research and Innovationhttp://iri.jrc.ec.europa.eu/

More info on the R&D investment scoreboard from

[email protected]

Page 64: Composite indicators. Methodology and Applications to R&D ...rosted.nu/attachments/File/2008/Indicators... · Ideally, a composite indicator should be based on: ‐Solid theoretical

64

More reading at (just google composite indicators):

http://composite-indicators.jrc.ec.europa.eu