testing the robustness of the global competitiveness index · the underlying concepts and framework...

- 1-

JRC STATISTICAL AUDIT OF THE GLOBAL

TALENT COMPETITIVENESS INDEX 2018

Michaela Saisana, Marcos Domínguez-Torreiro, and William Becker

European Commission Joint Research Centre

The Global Talent Competitiveness Index (GTCI) aims to summarise complex and versatile

concepts related to human capital and talent competitiveness at the national scale in 119

countries worldwide. In so doing, it raises some conceptual and practical challenges, which are

discussed in the GTCI 2018 report. This chapter focuses on the practical challenges related to

the data quality and the methodological choices made in the grouping of 68 variables into 14

sub-pillars, six pillars, two sub-indices, and an overall index.

The GTCI 2018 has a very high statistical reliability (it has a Cronbach’s alpha value of 0.97) and

captures the single latent phenomenon underlying the six main dimensions of the GTCI

conceptual framework. Country ranks are also robust to methodological choices related to the

treatment of missing values, weighting, and aggregation rule (with a shift less than or equal to

2 positions with respect to the simulated median in 95% of the countries). The added value of

the GTCI lies in its ability to summarise different aspects of talent competitiveness in a more

efficient and parsimonious manner than is possible with the variables and pillars taken

separately. In fact, the overall ranking differs from any of the six pillar rankings by 10 positions

or more in at least one-third of the countries included in this year’s GTCI.

This audit represents the fifth analysis of the GTCI performed by the European Commission’s

Competence Centre on Composite Indicators and Scoreboards at the Joint Research Centre

(JRC). The previous two audits identified a few minor issues concerning variables that had little

correlation with the output, but those remaining in the 2017 index have largely been addressed

in the 2018 edition. Overall, the JRC concluded that the GTCI 2018 is robust and reliable, with a

statistically coherent and balanced multi-level structure. The analysis has been performed in

order to ensure the transparency and reliability of the GTCI and thus to enable policymakers to

- 2-

derive more accurate and meaningful conclusions, and potentially to guide their choices on

priority setting and policy formulation.

As in the previous audits, the present JRC assessment of the GTCI 2018 focuses on two main

issues: (1) the statistical coherence of the structure and (2) the impact of key modelling

assumptions on the GTCI scores and ranks.1 The JRC analysis complements the reported country

rankings for the GTCI, and for the Input and Output sub-indices, with confidence intervals in

order to better appreciate the robustness of these ranks to the computation methodology (in

particular, the missing data estimation, weights, and aggregation formula). Furthermore, the JRC

analysis includes an assessment of the added value of the GTCI and a comparison with other

global measures of competitiveness and innovation. Its main conclusions can be summarised as

follows: the version of the GTCI model presented in 2018 is coherent, balanced, and robust,

displaying strong associations between the underlying variables and the GTCI sub-pillars, pillars,

and sub-indices, and hence offers a sound basis for policy interpretations. Some minor issues,

which are outlined in this chapter, are also recommended for examination in the next version of

the GTCI.

The practical items addressed in this chapter relate to the statistical soundness of the GTCI,

which should be considered to be a necessary (though not necessarily sufficient) condition for a

sound index. Given that the present statistical analysis of the GTCI will mostly, though not

exclusively, be based on correlations, the correspondence of the GTCI to a real-world

phenomenon needs to be critically addressed because ‘correlations need not necessarily

represent the real influence of the individual indicators on the phenomenon being measured’.2

The point is that the validity of the GTCI relies on the combination of both statistical and

conceptual soundness. In this respect, the GTCI has been developed following an iterative

process that went back and forth between the theoretical understanding of human capital and

talent competitiveness on the one hand, and empirical observations on the other.

[a]Statistical Coherence in the GTCI Framework

An initial assessment of the GTCI 2018 was undertaken by the JRC in July 2017. The latest GTCI

model provided by the development team largely incorporated the issues identified and

discussed in the previous edition, in particular full normalisation of the data in order to scale all

- 3-

variables onto the same scale. No critical issues were identified in the 2018 model during this

preliminary phase of the audit.

The underlying concepts and framework used to describe global talent competitiveness in the

GTCI 2018 have remained essentially the same as those in the GTCI 2017. However, with the aim

of improving the quality of the data, several variables have been removed and several others

have been added to the model. As a result of the deletion/replacement of some of the

indicators, the total number of variables in the GTCI 2018 is now 68, three more than the 65

used in the 2017 version.

Five new variables coming from the World Economic Forum’s Executive Opinion Survey have

been added to enhance the conceptual framework of the GTCI 2018. Active labour market

policies has been included in the Business and Labour Landscape sub-pillar, while Collaboration

within organisations and Collaboration across organisations provide significant added value to

the Access to Growth Opportunities sub-pillar. The variable Social protection has been included

in the Sustainability sub-pillar. Finally, two new indicators, Skills matching with secondary

education and Skills matching with tertiary education have been added to the Employability sub-

pillar.

Two variables have also been re-allocated to a better-fitting sub-pillar. The Scientific journal

articles indicator has been moved to the Talent Impact sub-pillar, for both conceptual and

statistical reasons. And the Availability of scientists and engineers indicator has been reallocated

to the High-Level Skills sub-pillar.

Two variables—Skills gap as a major constraint and Taxation—have been deleted entirely from

the framework since, as pointed out by the JRC last year, they did not have a solid showing in

the correlations matrix.

Following the iterative process during which the index has been fine-tuned, the current

assessment of the statistical coherence in this final version of the GTCI 2018 followed four steps:

[b]Step 1: Relevance

Candidate variables were selected for their relevance to a specific pillar on the basis of the

literature review, expert opinion, country coverage, and timeliness. To represent a fair picture of

- 4-

country differences, variables were scaled either at the source or by the GTCI team as

appropriate and where needed.

[b]Step 2: Data Checks

The most recently released data were used for each country. The cut-off year was changed from

2002 to 2005, thus affecting country coverage figures. Countries were included if data

availability was at least 80% at the index level and at least 40% at the sub-pillar level. As a result,

the GTCI 2018 data set comprises 119 countries and 68 variables. Consequently, data availability

is at least 88% at the Input sub-index level and 63% at the Output sub-index level. Potentially

problematic variables that could bias the overall results were identified by the GTCI

development team as those having absolute skewness greater than 2 and kurtosis greater than

3.5,3 and were treated either by Winsorisation or by taking the natural logarithm (in the case of

five or more outliers). For variables with five outliers or more, a log transformation is used (see

the Technical Notes of the main GTCI report for details). These criteria follow the WIPO-INSEAD

Global Innovation Index practice (formulated with the JRC in 2011). Data checks confirm that no

outliers or problematic indicators are present in the normalised data set as facilitated by the

development team.

[b]Step 3: Statistical Coherence

This section presents the JRC’s analysis of the statistical coherence of the GTCI 2018, which

consists of a principal components analysis to analyse the structure of the data, a multi-level

analysis of the correlations of variables, and a comparison of GTCI rankings with its pillars and

with other similar composite indicators. This latter investigation demonstrates the added value

of the GTCI both against its component pillars and against other similar indexes.

[c]1. Principal Components Analysis and Reliability Analysis

Principal component analysis (PCA) was used to assess the extent to which the conceptual

framework is compatible with statistical properties of the data. PCA confirms the presence of a

single statistical dimension (i.e., no more than one principal component with an eigenvalue

significantly greater than 1.0) in the great majority (11) of the 14 sub-pillars, which captures 53%

(Formal Education) to 83% (Employability) of the total variance in the underlying variables.4 A

- 5-

more detailed analysis of the correlation structure within and across the six pillars confirms the

expectation that the sub-pillars are more correlated to their own pillar than to any other, and all

correlations within a pillar are positive, strong, and similar and well above 0.7 (see Table 1).

These results suggest that the conceptual grouping of sub-pillars into pillars is statistically

confirmed and that the six pillars are statistically well balanced in the underlying sub-pillars.

Table 1

Statistical coherence in the GTCI: Correlations between sub-pillars and pillars

Enable Attract Grow Retain

Vocational and Technical Skills

Global Knowledge Skills

Inp

ut

1.1 Regulatory Landscape 0.96 0.88 0.85 0.86 0.82 0.73

1.2 Market Landscape 0.94 0.77 0.90 0.88 0.88 0.84

1.3 Business and Labour Landscape

0.91 0.80 0.74 0.71 0.74 0.62

2.1 External Openness 0.80 0.93 0.69 0.64 0.69 0.56

2.2 Internal Openness 0.80 0.90 0.77 0.71 0.70 0.56

3.1 Formal Education 0.70 0.51 0.87 0.82 0.79 0.85

3.2 Lifelong Learning 0.84 0.81 0.94 0.73 0.81 0.68

3.3 Access to Growth Opportunities

0.90 0.84 0.94 0.83 0.85 0.78

4.1 Sustainability 0.91 0.82 0.88 0.94 0.87 0.79

4.2 Lifestyle 0.70 0.54 0.72 0.92 0.81 0.80

Ou

tpu

t

5.1 Mid-Level Skills 0.68 0.53 0.69 0.83 0.86 0.73

5.2 Employability 0.79 0.76 0.81 0.68 0.82 0.65

6.1 High-Level Skills 0.76 0.61 0.80 0.83 0.83 0.95

6.2 Talent Impact 0.72 0.55 0.79 0.78 0.73 0.94

Source: European Commission, Joint Research Centre (2018).

Note: The values are the bivariate Pearson correlation coefficients; values greater than 0.70 are desirable as they

imply that the pillar captures at least 50% (≈ 0.70 × 0.70) of the variation in the underlying sub-pillars and vice-versa.

The six pillars also share a single statistical dimension that summarises 86% of the total variance,

and the six loadings (correlation coefficients) are quite high and very similar to each other,

ranging from 0.85 to 0.95. The latter suggests that the six pillars contribute in a similar way to

the variation of the GTCI scores, as envisaged by the development team: all six pillars are

assigned equal weights. The reliability of the GTCI, measured by the Cronbach’s alpha value, is

very high at 0.97—well above the 0.7 threshold for a reliable aggregate.5

- 6-

An important part of the analysis relates to clarifying the importance of the Input and Output

sub-indices with respect to the variation of the GTCI scores. As mentioned above, the GTCI is

built as the simple arithmetic average of the four Input sub-pillars and the two Output sub-

pillars, which implies that the Input sub-index has a weight of 4/6 versus a weight of 2/6 for the

Output sub-index. Yet this does not imply that the Input aspect is twice as important as the

Output aspect in determining the variation of the GTCI scores. In fact, the correlation coefficient

between the GTCI scores and the Input or Output sub-index is 0.99 and 0.96, respectively, which

suggests that the sub-indices are effectively placed on an equal footing. Overall, the tests so far

show that the grouping of variables into sub-pillars, pillars, and an overall index is statistically

coherent, and that the GTCI has a balanced structure, whereby all six pillars are equally

important in determining the variation in the GTCI scores.

[c]2. Importance of the Variables in the GTCI Framework

The GTCI and its components are simple arithmetic averages of the underlying variables.

Developers and users of composite indicators often consider that the weights assigned to the

variables coincide with the variables’ importance in the index. However, in practice, the

correlation structure of the variables and their different variances do not always allow the

weights assigned to the variables to be considered equivalent to their importance.

This section assesses the importance of all 68 variables at the various levels of aggregation in the

GTCI structure. As a statistical measure of the importance of variables in an index we use the

squared Pearson correlation coefficient (otherwise known as the coefficient of determination

R2).6 The importance of the selected variables is taken to be equivalent to the contribution of

those variables to the variation of the aggregate scores, be those sub-pillars, pillars, sub-indices,

or the overall GTCI. The overarching consideration made by the GTCI development team was

that all variables should be important at all levels of aggregation. The results of our analysis

appear in Table 2. Examining the importance measures of the 68 variables, we see that almost

all variables are important at the various levels of aggregation. For example, country variations

in 1.1.1 Government effectiveness scores can capture 91% of the variance in the respective sub-

pillar scores (Regulatory Landscape), 90% of the variance in the respective pillar (Enable), and

91% both in the Input sub-index and overall GTCI scores. Similarly, country variations in 2.1.1

- 7-

Foreign direct investment (FDI) and technology transfer scores can capture 62%, 57%, 56%, and

52% of the variance in the External Openness, Attract, Input, and GTCI scores, respectively. In

the 2018 data set, there are five variables that have a very low impact on the GTCI variance (less

than 10%): 1.3.1 Ease of hiring, 2.2.5 Gender earnings gap, 3.1.3 Tertiary education expenditure,

3.2.2 Prevalence of training in firms, and 6.2.3 New product entrepreneurial activity. Although

conceptually enriching the current GTCI framework and despite the sufficient though modest

statistical relevance (ranging between 12% and 24%) of four of them to last year’s framework,

these variables are not found to be important at the overall index level in the 2018 data

framework. In fact, 1.3.1 Ease of hiring has consistently been a low-impact variable in the overall

index and has been flagged in the JRC’s audits since 2014. Accordingly, the GTCI development

team should monitor closely how the statistical relevance of all five of these low-impact

variables evolve over time in next year’s release.

Table 2

Importance measures for the variables at the various levels of the GTCI structure

Pill

ar

Sub-pillar Variable name Sub-pillar Pillar Input/ Output

sub-index

GTCI Index

1. E

NA

BLE

1.1 Regulatory Landscape

1.1.1 Government effectiveness

91% 90% 91% 91%

1.1.2 Business-government relations

45% 43% 32% 26%

1.1.3 Political stability 69% 52% 54% 48%

1.1.4 Regulatory quality 88% 84% 83% 81%

1.1.5 Corruption 90% 82% 84% 82%

1.2 Market Landscape

1.2.1 Competition intensity

56% 47% 40% 38%

1.2.2 Ease of doing business

65% 65% 62% 64%

1.2.3 Cluster development

65% 61% 56% 53%

1.2.4 R&D expenditure 67% 50% 49% 53%

1.2.5 ICT infrastructure 72% 63% 74% 80%

1.2.6 Technology utilisation

82% 79% 79% 76%

1.3 Business and Labour Landscape

1.3.1 Ease of hiring 39% 17% 9% 8%

1.3.2 Ease of redundancy

31% 18% 13% 11%

1.3.3 Active labour market policies

62% 63% 62% 61%

1.3.4 Labour-employer cooperation

67% 61% 53% 49%

1.3.5 Professional management

66% 77% 76% 72%

- 8-

1.3.6 Relationship of pay to productivity

79% 71% 63% 62%

2. A

TTR

AC

T

2.1 External Openness

2.1.1 FDI and technology transfer

62% 57% 56% 52%

2.1.2 Prevalence of foreign ownership

55% 54% 45% 39%

2.1.3 Migrant stock 55% 43% 31% 30%

2.1.4 International students

69% 57% 40% 38%

2.1.5 Brain gain 70% 63% 44% 38%

2.2 Internal Openness

2.2.1 Tolerance of minorities

61% 44% 43% 39%

2.2.2 Tolerance of immigrants

32% 30% 21% 17%

2.2.3 Social mobility 64% 75% 72% 66%

2.2.4 Female graduates 17% 8% 13% 14%

2.2.5 Gender earnings gap

30% 14% 9% 7%

2.2.6 Leadership opportunities for women

51% 44% 35% 31%

3. G

RO

W

3.1 Formal Education

3.1.1 Vocational enrolment

44% 27% 19% 21%

3.1.2 Tertiary enrolment 71% 44% 38% 45%

3.1.3 Tertiary education expenditure

18% 12% 8% 7%

3.1.4 Reading, maths, and science

72% 52% 50% 54%

3.1.5 University ranking 65% 63% 51% 54%

3.2 Lifelong Learning

3.2.1 Quality of management schools

78% 72% 65% 65%

3.2.2 Prevalence of training in firms

45% 28% 9% 8%

3.2.3 Employee development

80% 73% 75% 71%

3.3 Access to Growth Opportunities

3.3.1 Delegation of authority

82% 75% 73% 69%

3.3.2 Personal rights 46% 38% 34% 33%

3.3.3 Use of virtual social networks

61% 49% 56% 57%

3.3.4 Use of virtual professional networks

73% 67% 67% 67%

3.3.5 Collaboration within organisations

76% 69% 71% 69%

3.3.6 Collaboration across organisations

52% 49% 46% 45%

4. R

ETA

IN

4.1 Sustainability

4.1.1 Pension system 65% 80% 56% 63%

4.1.2 Social protection 86% 69% 78% 74%

4.1.3 Brain retention 56% 31% 55% 49%

4.2 Lifestyle

4.2.1 Environmental performance

81% 69% 54% 59%

4.2.2 Personal safety 51% 57% 50% 51%

4.2.3 Physician density 79% 63% 38% 44%

4.2.4 Sanitation 77% 61% 39% 44%

5. V O CA TI O N AL A N D

TE CH NI

CA L SK ILL S

5.1 Mid-level Skills

5.1.1 Workforce with secondary education

67% 31% 28% 19%

- 9-

5.1.2 Population with secondary education

69% 32% 29% 19%

5.1.3 Technicians and associate professionals

79% 73% 73% 67%

5.1.4 Labour productivity per employee

59% 66% 60% 65%

5.2 Employability

5.2.1 Ease of finding skilled employees

83% 56% 51% 56%

5.2.2 Relevance of education system to the economy

86% 58% 52% 59%

5.2.3 Skills matching with secondary education

82% 61% 53% 56%

5.2.4 Skills matching with tertiary education

81% 47% 38% 46%

6. G

LOB

AL

KN

OW

LED

GE

SKIL

LS

6.1 Higher-Level Skills

6.1.1 Workforce with tertiary education

84% 72% 69% 56%

6.1.2 Population with tertiary education

61% 45% 39% 29%

6.1.3 Professionals 75% 74% 73% 66%

6.1.4 Researchers 76% 75% 75% 71%

6.1.5 Senior officials and managers

49% 45% 37% 27%

6.1.6 Availability of scientists and engineers

58% 53% 60% 57%

6.2 Talent impact

6.2.1 Innovation output 75% 77% 77% 74%

6.2.2 High-value exports 38% 35% 31% 28%

6.2.3 New product entrepreneurial activity

18% 9% 6% 4%

6.2.4 New business density

39% 31% 23% 23%

6.2.5 Scientific journal articles

65% 60% 55% 46%

Source: European Commission Joint Research Centre (2018). Note: The values are the squared Pearson correlation coefficients, expressed as percentages.

[c]3. Added Value of the GTCI

A very high statistical reliability among the main components of an index can be the result of

redundancy of information. This is not the case in the GTCI. In fact, the overall GTCI 2018

ranking differs from any of the six pillar rankings by 10 positions or more in at least one-third of

the 119 countries included in the 2018 edition, peaking at two-thirds in the Attract pillar (see

Table 3). This is a desired outcome because it evidences the added value of the GTCI ranking,

which helps to highlight other components of human capital and talent competitiveness that do

not emerge directly by looking into the six pillars separately. At the same time, this result also

- 10-

points towards the value of duly taking into account the individual pillars, sub-pillars, and

variables on their own merit. By doing so, country-specific strengths and bottlenecks in human

capital and talent competitiveness can be identified and serve as an input for evidence-based

policymaking.

Table 3

Distribution of differences between pillar and GTCI rankings

GTCI Input Sub-Index GTCI Output Sub-Index

Shifts with respect to the overall GTCI rank Enable Attract Grow Retain

Vocational and Technical

Skills

Global Knowledge

Skills

More than 30 positions 5% 18% 6% 2% 2% 7%

20 to 29 positions 7% 13% 10% 7% 8% 18%

10 to 19 positions 26% 30% 17% 27% 29% 24%

More than 10 positions 38% 61% 33% 35% 39% 48%

5 to 9 positions 27% 18% 26% 27% 31% 29%

Less than 5 positions 29% 18% 34% 34% 26% 23%

0 positions 6% 3% 8% 4% 3% 1%

Total 100% 100% 100% 100% 100% 100%

Source: European Commission Joint Research Centre (2018).

In addition, we compared the GTCI 2018 with both the World Economic Forum’s 2016–2017

Global Competitiveness Index and Cornell University, INSEAD, and WIPO’s 2017 Global

Innovation Index. After having extracted data from both projects’ websites, we find that the

rank correlation between GTCI 2018 with both indices is substantially high (correlation ≈ 0.9),

which suggests that the GTCI has many aspects in common with both these two indices. Looking

at the shifts in rankings (see Table 4), we nevertheless find that 41% and 44% out of the

countries included in the GTCI 2018 that feature in the other two indices differ in ranking by

more than 10 positions when comparing the GTCI 2018 with, respectively, the 2016–2017 Global

Competitiveness Index and the 2017 Global Innovation Index. This indicates that the GTCI 2018

offers additional insights into nations’ human capital and competitiveness compared to the two

other international indices.

- 11-

Table 4

Distribution of differences between the GTCI 2018 and other international rankings

Shifts with respect to the GTCI 2018

2017Global Innovation Index (Cornell, INSEAD, and WIPO)

2016–2017 Global Competitiveness Index (World Economic Forum)

More than 30 positions 4% 8%

20 to 29 positions 13% 10%


More than 10 positions 41% 44%


Less than 5 positions 29% 25%

0 positions 4% 9%

Total 100% 100%


[b]Step 4: Qualitative Review

Finally, the GTCI results, including overall country classifications and relative performances in

terms of the Input and Output sub-indices, were evaluated by the development team and

external experts to verify that the overall results are, to a great extent, consistent with current

evidence, existing research, or prevailing theory.

Notwithstanding these statistical tests and the positive outcomes regarding the statistical

soundness of the GTCI, it is important to mention that the GTCI has to remain open to future

improvements as better data, more comprehensive surveys and assessments, and new relevant

research studies become available.

[A]Impact of Modelling Assumptions on the GTCI Results

Every country score on the overall GTCI and its two sub-indices depends on modelling choices:

the six-pillar structure, the selected variables, the imputation or not of missing data, , and the

weights and aggregation method, among other elements. These choices are based on expert

opinion (e.g., selection of variables), or common practice (e.g., min-max normalisation in the

[0,100] range), driven by statistical analysis (e.g., treatment of outliers) or simplicity (e.g., no

imputation of missing data). The robustness analysis is aimed at assessing the simultaneous and

- 12-

joint impact of these modelling choices on the rankings. The data are assumed to be error-free

since potential outliers and any errors and typos were corrected during the computation phase.

As suggested in the relevant literature on composite indicators,7 the robustness assessment of

the GTCI was based on a combination of a Monte Carlo experiment and a multi-modelling

approach that dealt with three issues: pillar weights, missing data, and the aggregation formula.

In general, the uncertainty analysis aims to respond to some extent to possible criticisms that

the country scores associated with aggregate measures are generally not calculated under

conditions of certainty, even though they are frequently presented as such.

While the term multi-modelling refers to testing alternative assumptions—that is, alternative

aggregation methods and missing data estimation methods—the Monte Carlo simulation

explored the issue of weighting and comprised 1,000 runs, each corresponding to a different set

of weights for the six pillars, randomly sampled from uniform continuous distributions centred

in the reference values. The choice of the range for the weights’ variation was driven by two

opposite needs: to ensure a wide enough interval to have meaningful robustness checks, and to

respect the rationale of the GTCI that places equal importance on all six pillars. Given these

considerations, limit values of uncertainty intervals for the pillar weights are 15% to 35% for the

four Input pillars for the calculation of the Input sub-index, and 40% to 60% for the two Output

pillars for the calculation of the Output sub-index (see Table 5). For the calculation of the GTCI,

the limit values of uncertainty intervals for all six pillar weights are 12% to 20%. In all

simulations, sampled weights are rescaled so that they always sum to 1.

The GTCI development team, for transparency and replicability, opted not to estimate the

missing data (only 4.7% of data were missing in the data set of 119 countries for all 68

variables). The ‘no imputation’ choice, which is common in similar contexts, might encourage

countries not to report low data values. To overcome this limitation, the JRC also estimated

missing data using the Expectation Maximisation (EM) algorithm.

Regarding the aggregation formula, decision-theory practitioners have challenged the use of

simple arithmetic averages because of their fully compensatory nature, in which a comparatively

high advantage for a few variables can compensate for a comparative disadvantage for many

variables. Despite the arithmetic averaging formula receiving statistical support for the

development of the GTCI, as discussed in the previous section, the geometric average was

- 13-

considered as a possible alternative. This is a partially compensatory approach that rewards

countries with similar performance in all pillars; it motivates those countries with uneven

performance to improve in those pillars in which they perform poorly, and not just in any pillar.

Table 5

Uncertainty analysis for the GTCI 2018: Missing data, aggregation, and pillar weights

Reference Alternative

I. Uncertainty in the treatment of missing values

No estimation of missing data

Expectation Maximization (EM)

II. Uncertainty in the aggregation formula at pillar level

Arithmetic average Geometric average

III. Uncertainty in pillar weights

GTCI sub-index

Pillar Reference value for the weight

Distribution assigned for robustness analysis

Input

Enable 0.25 U[0.15,0.35]

Attract 0.25 U[0.15,0.35]

Grow 0.25 U[0.15,0.35]

Retain 0.25 U[0.15,0.35]

Output

Vocational and Technical Skills

0.50 U[0.40,0.60]

Global Knowledge Skills

0.50 U[0.40,0.60]


Four models were tested based on the combination of no imputation versus EM imputation, and

arithmetic versus geometric average, combined with 1,000 simulations per model (random

weights versus fixed weights), for a total of 4,000 simulations for the GTCI and each of the two

sub-indices (see Table 5 for a summary of the uncertainties considered in the GTCI 2018).

[b]Uncertainty Analysis Results

The main results of the robustness analysis are shown in Figures 1a-1c, with median ranks and

90% confidence intervals computed across the 4,000 Monte Carlo simulations for the GTCI and

the two sub-indices. Countries are ordered from best to worst according to their reference rank

(black line), the dot being the simulated median rank. Error bars represent, for each country, the

90% interval across all simulations. Table 6 reports the published rankings and the 90%

confidence intervals that account for uncertainties in the missing data estimation, the pillar

weights, and the aggregation formula. All published country ranks lay within the simulated

- 14-

intervals, and these are narrow enough for most countries (less than or equal to 10 positions) to

allow for meaningful inferences to be drawn.

GTCI ranks are shown to be both representative of a plurality of scenarios and robust to changes

in the imputation method, the pillar weights, and the aggregation formula. If one considers the

median rank across the simulated scenarios as being representative of these scenarios, then the

fact that the GTCI rank is close to the median rank (differing by two positions or less) for 95% of

the countries suggests that the GTCI is a suitable summary measure. Furthermore, the narrow

confidence intervals for the majority of the countries’ ranks (less than or equal to 10 positions

for 95% of the countries) imply that the GTCI ranks are also, for most countries, robust to

changes in the pillar weights, the imputation method, and the aggregation formula.

Results for the Input and Output sub-index are also robust and representative of the plurality of

scenarios considered. The Input rank is close to the median rank (less than or equal to two

positions away) for 97% of the countries and the rank intervals are less than or equal to 10

positions for 88% of the countries. Similarly, the Output rank is close to the median rank (less

than or equal to two positions away) for 81% of the countries, and the rank intervals are less

than or equal to 10 positions for 87% of the countries.

Overall, country ranks in the GTCI and its two sub-indices are fairly robust to changes in the

pillar weights, the imputation method, and the aggregation formula for the majority of the

countries considered. For full transparency and information, Table 6 reports the GTCI country

ranks (and those of the sub-indices) together with the simulated intervals (90% of the 4,000

scenarios) in order to better appreciate the robustness of these ranks to the computation

methodology.

- 15-

Figure 1a

Robustness analysis (GTCI rank vs. median rank, 90% confidence intervals)

1

11

21

31

41

51

61

71

81

91

101

111

GTC

I 20

18

ran

ks a

nd

inte

rval

s o

f si

mu

late

d r

anks

Countries

Median rank

GTCI 2018rank

Figure 1b

Robustness analysis (Input rank vs. median rank, 90% confidence intervals)

1

11

21

31

41

51

61

71

81

91

101

111GTC

I 20

18

ran

ks a

nd

inte

rval

s o

f si

mu

late

d r

anks

Countries

Median rank

GTCI 2018Input rank

- 16-

Figure 1c

Robustness analysis (Output rank vs. median rank, 90% confidence intervals)

1

11

21

31

41

51

61

71

81

91

101

111GTC

I 20

18

ran

ks a

nd

inte

rval

s o

f si

mu

late

d r

anks

Countries

Median rank

GTCI 2018Output rank

Source: European Commission Joint Research Centre (2018). Notes: The Spearman rank correlation between the median rank and the GTCI 2018 rank is 0.999; between the median rank and the GTCI 2018 Input rank is 0.999; and between the median rank and the GTCI 2018 Output rank is 0.998. Median ranks and intervals are calculated over 4,000 simulated scenarios combining random weights, imputation versus no imputation of missing values, and geometric versus arithmetic average at the pillar level.

Table 6 Country ranks and 90% confidence intervals for the GTCI 2018 and its Input/Output sub-indices

COUNTRY

GTCI 2018 INPUT SUB-INDEX OUTPUT SUB-INDEX

RANK INTERVAL RANK INTERVAL RANK INTERVAL

Switzerland 1 [1, 1] 1 [1, 1] 3 [3, 3]

Singapore 2 [2, 2] 2 [2, 4] 2 [2, 2]

United States of America 3 [3, 3] 11 [9, 13] 1 [1, 1]

Norway 4 [4, 6] 3 [2, 5] 7 [5, 12]

Sweden 5 [4, 5] 4 [2, 5] 6 [5, 7]

Finland 6 [6, 7] 8 [7, 10] 4 [4, 5]

Denmark 7 [6, 7] 5 [4, 7] 8 [7, 9]

United Kingdom 8 [6, 9] 6 [5, 7] 12 [9, 13]

Netherlands 9 [8, 11] 9 [7, 11] 11 [9, 14]

Luxembourg 10 [9, 14] 7 [3, 13] 20 [18, 20]

Australia 11 [10, 13] 10 [9, 12] 14 [11, 15]

New Zealand 12 [9, 13] 12 [8, 13] 13 [12, 14]

Ireland 13 [10, 13] 13 [11, 13] 10 [10, 11]

- 17-

Iceland 14 [13, 15] 17 [17, 18] 5 [4, 7]

Canada 15 [14, 16] 15 [14, 16] 16 [15, 17]

Belgium 16 [15, 16] 14 [14, 16] 18 [16, 18]

United Arab Emirates 17 [16, 21] 18 [15, 19] 15 [12, 28]

Austria 18 [17, 18] 16 [14, 17] 22 [19, 23]

Germany 19 [18, 19] 19 [18, 20] 17 [14, 20]

Japan 20 [20, 23] 21 [20, 24] 23 [21, 23]

France 21 [20, 23] 22 [21, 23] 21 [19, 21]

Estonia 22 [20, 23] 24 [22, 24] 19 [16, 21]

Qatar 23 [20, 27] 20 [19, 21] 32 [32, 46]

Israel 24 [19, 24] 27 [25, 28] 9 [7, 11]

Czech Republic 25 [24, 25] 23 [22, 24] 27 [25, 27]

Malta 26 [25, 27] 26 [25, 27] 26 [25, 27]

Malaysia 27 [26, 27] 25 [24, 27] 29 [28, 30]

Slovenia 28 [28, 29] 33 [30, 34] 25 [24, 26]

Portugal 29 [28, 30] 28 [27, 28] 34 [32, 35]

Korea, Rep. 30 [28, 31] 35 [32, 37] 24 [21, 24]

Spain 31 [30, 31] 29 [29, 30] 38 [35, 38]

Lithuania 32 [32, 33] 31 [30, 34] 39 [36, 40]

Chile 33 [32, 34] 34 [30, 34] 42 [40, 42]

Latvia 34 [33, 34] 37 [36, 37] 30 [29, 31]

Costa Rica 35 [35, 38] 32 [30, 34] 52 [50, 56]

Italy 36 [35, 38] 42 [39, 45] 31 [30, 31]

Cyprus 37 [35, 39] 45 [41, 46] 28 [27, 29]

Bahrain 38 [36, 43] 30 [29, 35] 62 [58, 69]

Poland 39 [37, 40] 41 [40, 43] 33 [32, 39]

Slovakia 40 [38, 40] 40 [38, 41] 43 [42, 45]

Saudi Arabia 41 [40, 43] 38 [38, 41] 48 [46, 54]

Greece 42 [41, 44] 47 [44, 50] 37 [33, 43]

China 43 [40, 44] 46 [43, 49] 40 [33, 43]

Uruguay 44 [42, 47] 36 [35, 36] 73 [70, 75]

Panama 45 [44, 48] 43 [40, 46] 58 [55, 58]

Mauritius 46 [44, 56] 39 [38, 42] 67 [63, 76]

Bulgaria 47 [44, 47] 48 [48, 53] 46 [44, 47]

Croatia 48 [45, 48] 56 [50, 62] 45 [42, 45]

Argentina 49 [48, 54] 50 [47, 59] 51 [50, 52]

Jordan 50 [49, 54] 49 [47, 56] 55 [53, 56]

Kazakhstan 51 [50, 53] 54 [50, 57] 53 [51, 53]

Hungary 52 [51, 55] 59 [55, 64] 50 [48, 50]

Russian Federation 53 [47, 56] 66 [61, 71] 36 [32, 38]

Philippines 54 [48, 58] 55 [51, 61] 54 [49, 56]

Trinidad and Tobago 55 [53, 58] 52 [48, 54] 57 [57, 64]

Oman 56 [50, 64] 44 [43, 48] 75 [70, 83]

Azerbaijan 57 [52, 60] 57 [52, 63] 56 [52, 63]

Montenegro 58 [54, 59] 69 [62, 70] 41 [39, 47]

Macedonia, FYR 59 [57, 60] 51 [50, 59] 63 [59, 64]

Lebanon 60 [57, 66] 82 [77, 84] 35 [34, 40]

Ukraine 61 [56, 64] 78 [72, 85] 44 [35, 44]

Botswana 62 [61, 68] 53 [47, 59] 79 [77, 83]

South Africa 63 [60, 68] 61 [49, 64] 72 [71, 72]

Romania 64 [61, 66] 63 [60, 67] 64 [60, 64]

Kuwait 65 [61, 72] 58 [51, 62] 78 [74, 82]

Armenia 66 [61, 68] 77 [73, 85] 47 [45, 48]

Colombia 67 [65, 69] 64 [59, 67] 68 [66, 69]

Turkey 68 [63, 69] 71 [67, 77] 59 [57, 61]

Serbia 69 [63, 71] 84 [80, 88] 49 [48, 54]

- 18-

Thailand 70 [68, 70] 62 [61, 65] 77 [71, 80]

Mexico 71 [70, 73] 68 [65, 71] 76 [73, 79]

Georgia 72 [71, 75] 72 [71, 81] 69 [66, 69]

Brazil 73 [72, 74] 67 [63, 68] 82 [77, 85]

Peru 74 [73, 79] 74 [71, 78] 71 [70, 82]

Mongolia 75 [70, 78] 79 [74, 80] 65 [62, 72]

Rwanda 76 [72, 87] 60 [52, 66] 99 [97, 100]

Indonesia 77 [75, 78] 75 [72, 77] 74 [72, 84]

Albania 78 [75, 86] 70 [67, 71] 91 [86, 96]

Dominican Republic 79 [78, 85] 73 [71, 80] 88 [84, 90]

Namibia 80 [78, 85] 65 [59, 72] 97 [95, 100]

India 81 [75, 82] 88 [83, 92] 66 [65, 67]

Sri Lanka 82 [76, 83] 80 [76, 82] 81 [76, 85]

Tunisia 83 [76, 85] 94 [88, 95] 60 [58, 62]

Guatemala 84 [82, 86] 76 [71, 81] 93 [91, 94]

Ecuador 85 [83, 88] 83 [75, 85] 89 [88, 93]

Moldova, Rep. 86 [80, 87] 92 [87, 94] 70 [63, 71]

Viet Nam 87 [81, 88] 87 [85, 89] 83 [75, 90]

Kenya 88 [84, 89] 89 [85, 95] 86 [76, 87]

Bosnia and Herzegovina 89 [88, 91] 96 [89, 98] 80 [79, 89]

Ghana 90 [89, 94] 86 [79, 91] 98 [97, 100]

Bhutan 91 [89, 99] 81 [77, 87] 106 [101, 111]

Honduras 92 [91, 95] 91 [86, 98] 96 [94, 98]

Kyrgyzstan 93 [90, 93] 98 [95, 99] 84 [82, 93]

Iran, Islamic Rep. 94 [90, 99] 105 [101, 108] 61 [59, 69]

Lao PDR 95 [94, 102] 85 [84, 97] 108 [103, 113]

Gambia 96 [93, 106] 90 [84, 96] 104 [99, 117]

Senegal 97 [95, 100] 93 [88, 96] 101 [101, 109]

Morocco 98 [91, 99] 97 [96, 99] 95 [93, 96]

Paraguay 99 [95, 99] 95 [90, 96] 100 [99, 106]

El Salvador 100 [100, 105] 100 [97, 100] 105 [104, 113]

Algeria 101 [98, 104] 108 [105, 110] 90 [86, 94]

Bolivia, Plurinational St. 102 [96, 103] 106 [101, 109] 92 [90, 93]

Uganda 103 [101, 106] 99 [96, 100] 110 [108, 117]

Egypt 104 [99, 107] 109 [106, 113] 94 [86, 96]

Venezuela, Bolivarian Rep. 105 [101, 108] 112 [108, 118] 85 [76, 85]

Lesotho 106 [104, 113] 101 [101, 105] 112 [108, 119]

Tanzania 107 [105, 108] 102 [101, 105] 115 [110, 115]

Cambodia 108 [108, 110] 104 [103, 107] 117 [112, 117]

Pakistan 109 [101, 109] 115 [112, 116] 87 [79, 89]

Malawi 110 [107, 111] 107 [105, 109] 111 [104, 115]

Nicaragua 111 [109, 115] 103 [101, 104] 119 [118, 119]

Ethiopia 112 [109, 117] 110 [107, 113] 116 [108, 118]

Mali 113 [112, 116] 111 [108, 112] 114 [111, 116]

Bangladesh 114 [111, 116] 113 [112, 115] 107 [105, 115]

Zimbabwe 115 [111, 115] 116 [114, 117] 102 [101, 103]

Nepal 116 [114, 118] 117 [114, 118] 103 [103, 107]

Mozambique 117 [116, 118] 114 [111, 117] 118 [113, 118]

Madagascar 118 [115, 118] 118 [116, 118] 109 [103, 110]

Yemen 119 [119, 119] 119 [119, 119] 113 [101, 117]


[b]Sensitivity Analysis Results

- 19-

Complementary to the uncertainty analysis, sensitivity analysis has been used to identify which

of the modelling assumptions have the highest impact on certain country ranks. Figure 2 plots

the GTCI and both sub-index rankings versus one-at-a-time changes of either the EM imputation

method or the geometric aggregation formula (assuming equal weights for the six pillars as in

the GTCI).

The most influential methodological assumption turns out to be the choice of geometric

aggregation versus arithmetic aggregation (given that a lower rank correlation indicates greater

sensitivity). This choice has the largest impact on differences in ranking for the GTCI 2018 and

the Output sub-index, less so for the Input sub-index. For example, in the most extreme case, a

country falls by 13 positions in the Output ranking when geometric aggregation is applied, yet

the country increases by four positions if missing data are imputed. Note, however, that these

assumptions concern methodological choices only and might overall be less influential than

choices related to the background assumptions in the conceptual framework.8

Overall, given the fairly modest ranges of uncertainty on the final rankings, the JRC

recommendation is not to alter the GTCI methodology at this point, but to consider country

ranks in the GTCI 2018 and in the Input and Output sub-indices within the 90% confidence

intervals, as reported in Table 6, in order to better appreciate to what degree a country’s rank

depends on the modelling choices. It is reassuring that, for an overwhelming majority of the

countries included in the GTCI, their ranks in the overall GTCI 2018 and the Input and Output

sub-indices are the result of the underlying data and not of modelling choices.9

- 20-

Figure 2

Sensitivity analysis: Impact of modelling choices

1

11

21

31

41

51

61

71

81

91

101

111

1112131415161718191101111

GT

CI In

pu

t ra

nk

Rank based on imputation

GTCI Input Sub-Index 2018

Rs= 0.999

1

11

21

31

41

51

61

71

81

91

101

111

1 11 21 31 41 51 61 71 81 91 101 111

GT

CI In

pu

t ra

nk

Rank based on geometric average

GTCI Input Sub-Index 2018

Rs= 0.999

1

11

21

31

41

51

61

71

81

91

101

111

1112131415161718191101111

GT

CI O

utp

ut ra

nk


GTCI Output Sub-Index 2018

Rs= 0.997

1

11

21

31

41

51

61

71

81

91

101

111

1 11 21 31 41 51 61 71 81 91 101 111

GT

CI O

utp

ut ra

nk


GTCI Output Sub-Index 2018

Rs= 0.995

1

11

21

31

41

51

61

71

81

91

101

111

1112131415161718191101111

GT

CI ra

nk


Global Talent Competitiveness Index 2018

Rs= 0.999

1

11

21

31

41

51

61

71

81

91

101

111

1 11 21 31 41 51 61 71 81 91 101 111

GT

CI ra

nk


Global Talent Competitiveness Index 2018

Rs= 0.996

- 21-

Source: European Commission Joint Research Centre (2018). Note: Rs represents the Spearman rank correlation coefficient.

[A]Conclusions

The JRC analysis suggests that the conceptualised multi-level structure of the GTCI 2018 is

statistically coherent and balanced (i.e., not dominated by any pillar or sub-pillar; all variables

contribute to the variation of the respective Input/Output sub-indices and to the overall GTCI).

Furthermore, the analysis has offered statistical justification for the use of equal weights and

arithmetic averaging at the various levels of aggregation, showing that the GTCI is statistically

reliable in its current form as the simple average of the six pillars (as measured by a very high

Cronbach’s alpha value of 0.97, well above the recommended 0.7 threshold for a reliable

aggregate).

Points that call for possible refinements of the GTCI framework were also identified. These

refinements mainly concern five out of the 68 variables, namely 1.3.1 Ease of hiring, 2.2.5

Gender earnings gap, 3.1.3 Tertiary education expenditure, 3.2.2 Prevalence of training in firms,

and 6.2.3 New product entrepreneurial activity. Although present in the conceptual framework,

these variables do not appear to contribute significantly to the variation of the GTCI country

scores and, consequently, do not have an impact on the GTCI rankings.

On the whole, the analysis of the correlations at the sub-pillar level reveals that the statistical

structure of the GTCI is coherent with its conceptual framework, given that sub-pillars correlate

strongly with their respective pillars. Furthermore, all pillars correlate strongly and fairly evenly

with the GTCI itself, which indicates that the framework is well balanced.

The GTCI and both sub-index country ranks are relatively robust to methodological assumptions

related to the estimation of missing data, weighting, and aggregation formula. It is reassuring

that for a large majority of the countries included in the GTCI, the overall rank and those in the

Input and Output sub-indices are the result of the underlying data and not of the modelling

choices. Consequently, inferences can be drawn for most countries in the GTCI, although some

caution may be needed for a few countries. Note that perfect robustness would have been

undesirable because this would have implied that the GTCI components are perfectly correlated

and hence redundant, which is not the case for the GTCI 2018. In fact, one way in which the

- 22-

GTCI helps to highlight other components of human capital and talent competitiveness is by

pinpointing the differences in rankings that emerge from a comparison between the GTCI and

each of the six pillars: the GTCI ranking differs from any of the six pillar rankings by 10 positions

or more for at least one-third (up to two-thirds) of the countries. This outcome both evidences

the added value of the GTCI ranking and points to the importance of taking into account the

individual pillars, sub-pillars, and variables on their own merit. By doing so, country-specific

strengths and bottlenecks in human capital and talent competitiveness can be identified and

serve as an input for evidence-based policymaking.

The auditing conducted herein has shown the potential of the Global Talent Competitiveness

Index 2018, subject to some minor hints for future releases, in reliably identifying weaknesses

and best practices and ultimately monitoring national performance in human capital and

competitiveness issues around the world.

[A]References

Becker, W., Saisana, M., Paruolo, P., & Vandecasteele, I. (2017). Weights and importance in

composite indicators: Closing the gap. Ecological Indicators 80: 12–22.

Cornell University, INSEAD, & WIPO. (2017). The Global Innovation Index 2017: Innovation

feeding the world. Ithaca, Fontainbleau, and Geneva: Cornell University, INSEAD, &

WIPO. Available at https://www.globalinnovationindex.org/gii-2016-report

Groeneveld, R. A. & Meeden, G. (1984). Measuring skewness and kurtosis. Journal of the Royal

Statistical Society, Series D (The Statistician), 33, 391–399..

Little, R. J. A. & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). New York:

John Wiley & Sons.

Munda, G. (2008). Social multi-criteria evaluation for a sustainable economy. Berlin Heidelberg:

Springer-Verlag.

Nunnally, J. (1978). Psychometric theory. New York: McGraw-Hill.

OECD & EC JRC (Organisation for Economic Co-operation and Development and European

Commission Joint Research Centre). (2008). Handbook on constructing composite

- 23-

indicators: Methodology and user guide. Paris: OECD. Available at

http://www.oecd.org/std/42495745.pdf

Saisana, M., D’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university

rankings and policy implications. Research Policy, 40 (1): 165–177.

Saisana, M. & Saltelli, A. (2011). Rankings and ratings: Instructions for use. Hague Journal on the

Rule of Law, 3 (2), 247–268.

Saisana, M., Saltelli, A., & Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as

tools for the analysis and validation of composite indicators. Journal of the Royal

Statistical Society: Series A (Statistics in Society), 168 (2), 307–323.

Saltelli, A. & Funtowicz, S. (2014). When all models are wrong. Issues in Science and Technology,

Winter 2014, 79–85.

Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., &

Tarantola, S. (2008). Global sensitivity analysis: The primer. Chichester, England: John

Wiley & Sons.

World Economic Forum. (2016). The Global Competitiveness Report 2016–2017: Full data

edition. Geneva: World Economic Forum.

[a]Endnotes

1 The JRC analysis was based on the recommendations of the OECD & EC JRC (2008) Handbook on

Constructing Composite Indicators and on more recent research from the JRC. The JRC auditing studies of

composite indicators are available at http://composite-indicators.jrc.ec.europa.eu/ (all audits were

carried upon request of the index developers).

2 OECD & EC JRC (2008).

3 Groeneveld and Meeden (1984) set the criteria for absolute skewness above 1 and kurtosis above 3.5.

The skewness criterion was relaxed to account for the small sample (119 countries).

4 Only in three of the sub-pillars is there a second principal component with an eigenvalue slightly above

the 1.0 threshold: 1.3 Business and Labour Landscape (eigenvalue of 1.01), 2.2 Internal Openness

- 24-

(eigenvalue of 1.05), and 6.2 Talent Impact (eigenvalue of 1.03). This suggests that relevant information

might be lost when directly aggregating the variables into sub-pillars.

5 See Nunnally (1978).

6 Becker et al. (2017).

7 Saisana et al. (2005); Saisana & Saltelli (2011), Saisana et al. (2011); Saltelli et al. (2008).

8 Saltelli & Funtowicz (2014).

9 As already mentioned in the uncertainty analysis, about 95% of the simulated median ranks for the GTCI

and Input (sub-) indices are less than two positions away from the reported 2018 rank—this percentage

drops only to 81% in the Output sub-index.

testing the robustness of the global competitiveness index · the underlying concepts and framework...

Documents