Evaluating funding modestowards a concept of funding ecologies

Ulf Sandström

KTH Stockholm

Peter van den Besselaar

VU University AmsterdamOrg Science & Network Institute

van den Besselaar FTEval 2013 2

Proliferation of funding modes

• New policy aims -> new funding modes– Strengthening competition (councils) – Career grant schemes– Excellence programs– Strategic knowledge: large scale programs– Useful knowledge: thematic programs

– International funding in various ways

14/11/2013

van den Besselaar FTEval 2013 3

From evaluating funding institutions and programs …

• Evaluating of funding institutions and individual funding instruments– Does the evaluated funding

institution/instruments work well?

• But what is “well”? Frame of reference? – Comparison? But how and with what?– Different funders/instruments fulfill different and

complementary roles

14/11/2013

van den Besselaar FTEval 2013 4

… to evaluating the funding ecology

14/11/2013

5

Funding ecology

• Complex landscape of funders and funding instruments with partly different and partly overlapping agenda’s– Overlapping: this leads to competition between

funders, opening up the system for variety– Different: specializing, niche seeking, searching for

the promising fields, application domains, procedures and instruments -> variety

14/11/2013 van den Besselaar FTEval 2013

van den Besselaar FTEval 2013 6

Competition & complementarity

– What is the optimal funding ecology?

– Avoiding duplication of instruments?

– Variety versus size?

14/11/2013

van den Besselaar FTEval 2013 7

Three tasks in analyzing the funding ecology

• Describe– Distinguish modes and instruments– Measure the output and impact by funding mode– At field level

• Compare the funding institutions/instruments• Understand the dynamics of funding ecologies– Optimal variety and size and under what

conditions

14/11/2013

van den Besselaar FTEval 2013 8

Ongoing research

• Impact of large thematic programs– Knowledge for Climate program

• Searching for excellence and the effects of funding– Bank of Sweden Tercentenary Foundation – With Ulf Sandstrom and Agnes Wold

• Ongoing work -> scattered results up to now

14/11/2013

van den Besselaar FTEval 2013 9

Description: comparison and complementarity of funders

• Do different funding modes differ in terms of:

– Organizational forms, procedures, criteria

– Topical focus

– Productivity, impact, innovativeness, societal relevance

14/11/2013

van den Besselaar FTEval 2013 10

Methods and data• WoS data; delineation of fields• Topical structure• Field normalized impact • Publ: 2009/2010; Cit. window 2010-12• Clean/disambiguate funders• Classify funders: funder type

14/11/2013

van den Besselaar FTEval 2013 11

Empirical examples

• Two dimensions– Impact by funding mode– Topical coverage by funding mode

• Two fields– climate change research– immunology

• Two countries – Sweden – the Netherlands

14/11/2013

van den Besselaar FTEval 2013 12

Funding modes / organizations• Universities• Public research Institutes– Fundamental– Applied

• Councils– Basic– Applied

• Organizations – Private foundations– NGO’s– Societies

• Thematic

– Large national programs– Thematic councils

• EC: – FW program – ERC, Marie Curie

• Foreign funders

• Companies

• No funders mentioned

14/11/2013

van den Besselaar FTEval 2013 13

Delineation

• Climate change research: search terms (TS)• Immunology: journals (WoS research area)

• More fields/countries in next phase

14/11/2013

van den Besselaar FTEval 2013 14

New funders: climate change research 1988-2012

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110

100

200

300

400

500

world nl sweden germany swiss

14/11/2013

van den Besselaar FTEval 2013 15

Impact by funder (NL-climate)

14/11/2013

van den Besselaar FTEval 2013 16

Impact by funder (NL-climate)

1 3 4 5 6 7 8 90%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

no funder

appl agencies

council

univ

foreign

big progs

EC

14/11/2013

van den Besselaar FTEval 2013

Topical diifferences (NL-climate)

14/11/2013 17

van den Besselaar FTEval 2013 18

Impact by funder (Sw-climate)

14/11/2013

van den Besselaar FTEval 2013 19

Int. co-auth. papers (Sw-climate)

14/11/2013

van den Besselaar FTEval 2013 20

Nat auth papers (Sw-climate)

14/11/2013

van den Besselaar FTEval 2013 21

Summary of findings up to now

• Impact:– International funding > national funding– Partly but not fully international collaboration effect – Funding conditions are important

• Topics: differences & overlap

14/11/2013

van den Besselaar FTEval 2013 22

Current work

• Extending the analysis to other fields and countries:

– Patterns (1): relation between output/impact variables and number of funders / funder types

– Patterns (2): relation between output/impact variables and size of funders / instruments

14/11/2013

van den Besselaar FTEval 2013 23

Extending the evidence

• Similar results for Immunology– International funders > national funders– Effect remains (but smaller) when controlling for

international co-authorship– Funding conditions matter

• Size of instrument matters:– Size/selection trade off

14/11/2013

Peter van den Besselaar / predictive validity 24

Predictive validity by competition level

9/05/13

van den Besselaar FTEval 2013 25

Conclusions and further work

• Comparing funding modes and instruments seems promising

• Technical improvements (classification funding modes, organizational differences, statistics)

• More fields, more countries• More performance indicators (e.g., societal impact)

• Empirical regularities? And then explanations.

• Lessons about individual funding modes / instruments• Lessons about funding ecologies14/11/2013