an international review on governments’ research procurement strategies · 2018-04-23 · 3...

REPORT BY THE COMPTROLLER AND AUDITOR GENERALHC 586-II Session 2002-2003: 16 April 2003

An international review on Governments’ research procurement strategies

A paper in support of Getting the evidence: Using research in policy making

The National Audit Officescrutinises public spending

on behalf of Parliament.

The Comptroller and Auditor General, Sir John Bourn, is an Officer of the

House of Commons. He is the head of theNational Audit Office, which employs some750 staff. He, and the National Audit Office,

are totally independent of Government.He certifies the accounts of all Government

departments and a wide range of other publicsector bodies; and he has statutory authority

to report to Parliament on the economy, efficiency and effectiveness

with which departments and other bodieshave used their resources.

Our work saves the taxpayer millions ofpounds every year. At least £8 for every

£1 spent running the Office.

An international review on Governments’ research procurement strategies

A paper in support of Getting the evidence: Using research in policy making

REPORT BY THE COMPTROLLER AND AUDITOR GENERALHC 586-II Session 2002-2003: 16 April 2003

Ordered by theHouse of Commons

to be printed on 10 April 2003

LONDON: The Stationery Office£14.00Two volumes not to be sold separately

This report has been prepared under Section 6 of theNational Audit Act 1983 for presentation to the Houseof Commons in accordance with Section 9 of the Act.

John Bourn National Audit OfficeComptroller and Auditor General 10 April 2003

This study was contracted out to RAND Europe. Theproject team for the international review was led by Erik Frinking with support from Silvia Anton, Abigail Lierens, Dr Sally Ann Law and Dr Jonathan Grant.

This report can be found on the National Audit Officeweb site at www.nao.gov.uk

For further information about the National Audit Officeplease contact:

National Audit OfficePress Office157-197 Buckingham Palace RoadVictoriaLondonSW1W 9SP

Tel: 020 7798 7400

Email: [email protected]

ContentsExecutive summary 1

Part 1

Introduction 9

Current investment in research and development 9

Developments in R&D investments over time 10

Part 2

Canada 15

Institutional context 15

Priority setting and coordinating processes 17

Selecting and commissioning research 17

Measuring outcomes 18

Part 3

Finland 19





Part 4

Germany 25





Part 5

The Netherlands 31





Part 6

United States 37





References 42

AN INTERNATIONAL REVIEW ON GOVERNMENTS’ RESEARCH PROCUREMENT STRATEGIES

Preface

AN INTERNATIONAL REVIEW ON GOVERNMENTS'

RESEARCH PROCUREMENT STRATEGIES

pref

ace

This report presents the results of an international review of how thegovernments of five countries in North America and Europe procure andmanage research to improve service delivery and policy development. Itcomplements the National Audit Office report "Getting the evidence: Usingresearch in policy making". The main objectives of this paper, are twofold. First,it aims to describe how research and development is commissioned, managedand used in a number of different countries. Second, it provides a basis forexamining the research and development activities of the UK within aninternational context and for learning how innovative elements from othercountries may be incorporated into or modified to suit the UK research anddevelopment model.

The executive summary of this report outlines the key findings from theinternational review. In summary these are:

! There is no uniform approach to determine research priorities and to set research strategies. However, there seems to be increased awarenessand activity to make these strategies and priorities a more integral part of policymaking.

! There is growing emphasis on evaluating the policy outcomes of researchand development expenditure.

! There is widespread acknowledgement that some research anddevelopment expenditure is high risk and will bring no short-term return,but that it is essential for long-term development.

! There are some innovative examples of how research users are incorporatedinto the research process, with the aim of increasing research utilisation.

The Executive Summary also reviews the significant similarities and differencesin research and development practices among the selected countries andbetween the countries and the UK. First, investment in research anddevelopment is summarised, then priority setting and coordinating processesare compared, followed by selecting and commissioning practices and, finally,evaluation and research transfer are examined. The remainder of the reportpresents detailed information about the context of research and developmentactivities in the five selected countries and describes the different andinnovative approaches to research selection, procurement, implementation,management, evaluation and transfer in each.

executivesummary



1

exec

utiv

e su

mm

ary

1 This report presents the results of an international review of how thegovernments of five countries in North America and Europe procure andmanage research to improve service delivery and policy development. Itcomplements the National Audit Office report, "Getting the evidence: Usingresearch in policy making", which provides an assessment of the researchactivities of UK government departments and examines how research is used toimprove service delivery and inform policy making in this country.

2 The main objectives of this paper are twofold. First, it aims to describe howresearch and development is commissioned, managed and used in a number ofdifferent countries. Second, it provides a basis for examining the research anddevelopment activities of the UK within an international context and forlearning if and how innovative elements from other countries may beincorporated into or modified to suit the UK research and development model.Unlike "Getting the evidence: Using research in policy making", theinternational review covers science based activities as well as researchcommissioned by Government departments for policy making. This is becausethese two elements of publicly funded research are not always as easilydistinguishable as they are in the UK.

3 Countries were selected according to several criteria. First, only countries withsizeable investments in research and development (at least exceeding 1.75% ofGDP) were considered. Second, in order to examine the effects of institutionalcontext on research and development activities and outcomes, countries wereselected with a range of government structures. Finally, in order to presentcomparisons relevant to the UK context, selected countries did not differfundamentally from the UK on any of the previous criteria.

4 Application of the selection criteria led to a comparative examination of thefollowing five countries: Canada, Finland, Germany, the Netherlands and theUnited States (US). Information gathering mainly comprised desk researchsupplemented by additional, targeted interviews. The organisation and analysisof the information was based on a conceptual framework and issues thatemerged during the course of the overall study.

5 The executive summary first presents the main findings from the internationalreview. It then briefly reviews the significant similarities and differences inresearch and development practices among the selected countries and betweenthe countries and the UK. First, research and development investment level issummarised, then priority setting and coordinating processes are compared,followed by selecting and commissioning practices and, finally, evaluationapproaches and research transfer are examined. More details on the practicesof each individual country are provided according to the same structure in thesubsequent chapters.

2

exec

utiv

e su

mm

ary

AN INTERNATIONAL REVIEW ON GOVERNMENTS' RESEARCH PROCUREMENT STRATEGIES

Main findings6 The main findings from the international review are as follows:

! Government departments in the selected countries struggle with thecomplexity of how best to determine research priorities and set appropriateresearch strategies. New organisations and structures emerge to cope withthese complexities, some moving towards centralisation and concentration,some towards decentralisation. Either way, these changes aim to stimulatenew ways for departments to think about research and development andpolicymaking, to prioritise research decisions and to set research strategies.

! The need for more and improved information systems to providecomprehensive overviews of diverse research and developmentcommissioning practices and options is apparent in the selected countries.Ideally, such information systems could serve several important objectivesby maintaining and sharing information for analysis, thus improvingcoordination activities and increasing transparency.

! Evaluation of the quality of the research process is well established.However, there is a strong and developing emphasis on evaluation toencompass research relevance and value for money, as the link betweenresearch results and policy formulation increasingly becomes the focus ofattention. As yet, obvious models or practices that support the link are notreadily available. Similar findings emerged from the UK-based study ofresearch and development transfer into practice.

! As in the UK, government departments and research organisations in theselected countries strive to provide value for money in terms of researchoutput. However, there is widespread understanding of the need for "blue-sky" research that brings no, or little, short-term return on researchinvestments, but is essential for long-term development. Balancing theseoften competing demands proves difficult.

! In Canada, the "Linkage and Exchange" model provides an interestingexample of research implementation in the health services policy arena. Itproposes that involving eventual end users at all stages of the researchprocess will result in an increased impact of research on policymaking.

3

exec

utiv

e su

mm

ary


International expenditure on research and development7 While there is a considerable range in levels of investment in research and

development among the selected countries and the UK, two distinct groups canbe distinguished. Group one, Finland, the US and Germany, all spend close toor more than 2.5% of Gross Domestic Product (GDP) on research anddevelopment, and thus invest more than the Organisation for EconomicCooperation and Development (OECD) average of 2.24%. Group two, theNetherlands, UK and Canada invest less than the OECD average, ranging frombetween 1.84% (Canada) to 2.02% (the Netherlands). The gross expenditureson research and development (GERD) are shown in Figure 1. GERD is thestandard expenditure measure which covers all research and developmentcarried out on national territory.

8 Figure 2 shows the amount of government budget appropriations for researchand development (GBAORD), as a percentage of GDP. GBAORD presentsinformation about research and development financed by government basedon budget data and is more up-to-date than actual expenditures. Defencespending in the UK and US accounts for more than 50% of total GBAORD.However, when considering civil government spending on research anddevelopment (civil GBAORD), it becomes clear that the UK and the US are thelowest investors in civil research and development, dropping even below thecivil research and development average expenditure for OECD membercountries. In contrast, the importance of civilian research and developmentspending to the Netherlands, which spends relatively little on defence,becomes clearer and the leading position of Finland, with its marginal defencespending, is even more accentuated.

Similarities and differences in practice amongselected countries9 On priority setting and coordinating processes. Two main issues relating to

priority setting and research coordination emerged from the review ofgovernment departments and research bodies in the selected countries: first,how best to translate policy needs into research priorities, and second, how tocoordinate research priorities across, and, to a lesser extent, within ministries.

10 Aiming to address the first issue, a number of different practices can bedistinguished across the countries selected. The most important differencesrelate to the level of concentration or centralisation at which priority settingtakes place. In Canada and the US, the decisions are made within a highlydecentralised environment predominantly via external boards that advise therespective departments and agencies. In Canada, the process is formalisedthrough Science Advisory Boards (SABs). In the US, each agency tends to relyon its own iterative, and often complex, decision making process despite thefact that goals, priorities and budget allocation are all part of the research anddevelopment budgeting system.

4

exec

utiv

e su

mm

ary


% o

f GD

P

4.00

3.50

3.00

2.50

2.00

1.50

1.00

0.50

0.00

Canad

a

EU A

verag

e

United

King

dom

Netherl

ands

OECD A

verag

e

German

y

United

State

s

Finlan

d

Gross Domestic Expenditure on R&D (GERD) as % of GDP, 20001

Source: Main Science and Technology Indicators, OECD, Paris, 2002

1.2

1

0.8

0.6

0.4

0.2

0

Canad

a

EU A

verag

e

United

King

dom

Netherl

ands

OECD A

verag

e

German

y

United

State

s

Finlan

d

Difference between total GBAORD and GBAORD on civil R&D, 20002


GBAORD % of GDP Civil GBAORD % of GDP

5

exec

utiv

e su

mm

ary


11 In Germany and the Netherlands, traditional science policy advisory boardsprovide high level expertise and input to the government as a whole. Their roleis strictly advisory. The Netherlands also has a wide net of sector councils tosupport specific policy areas. In principle, the sector councils are not advisorybodies, but are intended to inform policymaking processes, often throughforesight studies. Finally, in Finland the ultimate authority for determining basicscience policy and the allocation of government research grants resides withParliament and the Cabinet of Ministers (CoM). The CoM relies heavily on thescientific expertise provided by the Science and Technology Policy Council(STPC) headed by the Prime Minister. The STPC takes a prominent role indetermining research strategies for the Finnish government.

12 With respect to the issue of coordinating research priorities across, and to alesser extent within, ministries, this takes place mainly at the policy levelrather than through external advisory boards. Individual ministries or policyimplementation agencies are usually responsible for coordination efforts.Finland is the exception, as the STPC has a visible role in the coordinationof innovation policy activities at a national level in addition to its prioritysetting powers.

13 On selecting, commissioning and monitoring research. Selecting the bestresearch to inform policymaking is the major focus of research procurementand monitoring activities among the countries reviewed. The link betweenpolicymakers' needs and research decisions is also strong in the UK (see"Getting the evidence: Using research in policy making" for detailed discussionof this point). Research providers all understand the importance of externaladvice as a basis for objective, unbiased research and realise that procurementapproaches ought to be determined by the strategic aims specific to eachorganisation. Examples of approaches to optimise the link between researchand policymaking that show the importance accorded to such efforts include:the establishment of independent, intermediary organisations to manage theselection and implementation of research based on expertise and dedicatedcapacity; the formation of research programmes to bring together researchproviders and create networks or centres of excellence; and the distribution ofguidelines and/or handbooks to operationalise procurement principles.

14 In Canada, a number of advisory reports highlighted the enormous range ofapproaches used to access and formulate the need for science to informdecision making. The review found that in the majority of cases, the preferredway of seeking advice was through in-house analyses and working groupsrather than by seeking independent reviews. A report by the Council of Scienceand Technology Advisors (CSTA) resulted in the Canadian governmentpublishing a "Framework for Science and Technology Advice: Principles andguidelines for the effective use of Science and Technology advice ingovernment decision making"1 to make preferred practice guidelines specific.

15 The US General Accounting Office (GAO) Report "Federal Research: PeerReview at Federal Science Agencies Vary" (March 1999) looked at how federalagencies conducted peer reviews of research products and concluded therewas no uniform federal peer review policy. There is general agreement that peerreview practices should not be dictated uniformly for every agency or for alltypes of federally funded research. Rather, the practices should be tailored toagency missions and type of research.

1 This report drew heavily on OST guidance in the UK.

6

exec

utiv

e su

mm

ary


16 In Finland, cluster, technology and research programmes are increasingly usedas strategic mechanisms for funding research and pursuing science policyobjectives. They are multidisciplinary, usually exist for a fixed period andideally involve consortia that combine several research projects. Programmeshave proved to be an effective form for selecting and involving variousresearch-related organisations and stimulating cooperation and networkingopportunities between private companies and the research sector.

17 The acquisition, planning, implementation, administration and evaluation ofindividual projects in Germany are not the responsibility of ministries, butrather of appointed research management organisations (Projektträger) outsideof government. Often these organisations are research institutes themselves.Their project management responsibilities are of both a scientific/technical andadministrative nature. The need for intermediary organisations is a result of thegrowth of sponsoring activities by the federal ministries beyond their capacityto manage. The agencies are typically sponsored by federal money. TheProjektträger often also functions as an international point of contact.

18 In the Netherlands, intermediary organisations, such as Senter and Novem,coordinate and commission the research activities of several ministries. Forprogrammes that have been set up by various ministries and that are ofsignificant size, some independent or temporary programme offices havebeen established that are responsible for implementing strategies andcommissioning research.

19 On evaluation and research transfer: The international review uncovered alarge range of long existing research evaluation practices. Evaluationsincreasingly take place throughout the research base leading to structuralchanges within the national research systems and the resulting research bodies.Evaluations are also conducted throughout the various stages of the researchprojects. The practice of ex-ante evaluation to examine the connection betweenproposed research and government policy needs is also increasing, as is themonitoring of ongoing research and re-evaluating its links to ongoing orupcoming policy.

20 Examples of well developed evaluation practices are found in Finland andGermany. For a long time, evaluation has played a steady role in theformulation of policy in Finland. The effectiveness of government action isassessed at different levels. External and international teams evaluate all majororganisations and the major policy players regularly have their programmesevaluated externally.

21 Evaluations in Germany have lead to many improvements in the researchsystem. First of all, funding for under-performing institutes was completelystopped. Second, a concentration of certain research institutes took place toeliminate the fragmentation of the research base. Finally, evaluations haveencouraged international cooperation in Germany.

22 Generally speaking, evaluation tools and approaches have become morediverse and sophisticated. Where peer review used to be the default process,standardised performance measures and impact analyses are now preferredand have become more common. In Canada, the research and developmentImpact Network and the Programme of Energy Research and Development(PERD) have implemented results-based performance measurement. They aretwo examples of federal science-based department and agency efforts to useimpact analysis to assess the outcomes and results of federal research anddevelopment and to ensure relevance and value for money.

7

exec

utiv

e su

mm

ary


23 The increased importance of evaluation has led to the need for more reliable,comprehensive and timely data sets about government funded research anddevelopment and improved information systems to support policymaking arebeing developed. The Netherlands Observatory of Science and Technology(NOWT) collects and analyses data about the Dutch research system in a broadsense. RaDiUS, which stands for "Research and Development in the UnitedStates", is the first information system that systematically connects highlyaggregated budget data on federal research and development with thedisaggregated information on individual research and development tasks andawards to provide a complete picture of all federal research and developmentactivities in the US.

24 With the establishment of the 1993 Government Performance and Results Act(GPRA) in the US, federal agencies, including those that fund research, wereformally required to set strategic goals and to use performance measures formanagement and budgeting. The objective of the GPRA is to encouragegreater efficiency, effectiveness and accountability in federal programmesand spending. A report by the US Committee on Science, Engineering andPublic Policy (COSEPUP) considered the most effective ways to assess theresults of research, in light of the GPRA. COSEPUP drew a number ofconclusions, including:

! Both basic research and applied research programmes can be meaningfullyevaluated on a regular basis;

! Agencies must evaluate research programmes by using measurements thatmatch the character of the research;

! The most effective means of evaluating federally funded researchprogrammes is expert review. Expert review - which includes peer review(judging the quality), relevance review (judging whether an agency'sresearch activities are relevant to its mission), and benchmarking (judgingthe relative standing in an international perspective) - should be used toassess both basic research and applied research programmes; and

! The development of effective methods for evaluating and reportingperformance requires the participation of the scientific and engineering community.

25 In addition to the increasing emphasis on research evaluation in theinternational arena, more and more attention is being focused on how topromote its transfer into policy. In Canada, the "Linkage and Exchange" modelprovides an interesting example of research implementation in the healthservices policy arena. It proposes that specific issues and bottlenecks arise incommunication between researchers and policymakers that often preventeffective transfer of research findings into policy decisions. It proposes thatinvolving eventual end users at all stages of the research process will result inan increased impact of research on policymaking.

26 In parallel, several efforts of the CSTA have focused on establishing principlesand guidelines to incorporate science advice in government decision making.These principles and guidelines address how science advice should be soughtand applied to enhance the ability of government decision makers to makeinformed decisions.

Part 1

AN INTERNATIONAL REVIEW ON GOVERNMENTS’


Introduction

9

part

one

1.1 This report presents the results of an international reviewof how the governments of five countries in NorthAmerica and Europe procure and manage research toimprove service delivery and policy development. Itcomplements the NAO report, "Getting the evidence:Using research in policy making", which provides avalue for money assessment of the research activities ofUK government departments and examines howresearch is used to improve service delivery and informpolicy making in this country.

1.2 The main objectives of this paper are twofold. First, it aims to describe how R&D is commissioned,managed and used in a number of different countries.Second, it provides a basis for examining the R&Dactivities of the UK within an international context andfor learning if and how innovative elements from othercountries may be incorporated into or modified to suitthe UK R&D model.

1.3 Countries were selected according to several criteria.First, only countries with sizeable investments in R&D(at least exceeding 1.75% of GDP) were considered.Second, in order to examine the effects of institutionalcontext on R&D activities and outcomes, countries wereselected with a range of government structures. Finally,in order to present comparisons relevant to the UKcontext, selected countries did not differ fundamentallyfrom the UK on any of the previous criteria.

1.4 Application of the selection criteria led to acomparative examination of the following fivecountries: Canada, Finland, Germany, the Netherlandsand the United States (US). Information gatheringmainly comprised desk research supplemented byadditional, targeted interviews. The organisation andanalysis of the information was based on a conceptualframework and issues that emerged during the courseof the overall study.

1.5 The next part of this chapter provides an overview of theR&D activities in each of the selected countries and therelative importance of the R&D sector across thesecountries. The overview is based on a selection of mostcommonly used R&D indicators which are relevant notonly to government departments commissioningresearch, but also to R&D stakeholders within a specificcountry. Although governments account for only afraction of total R&D spending, these indicators suggestthe relative value that these countries place on researchand development. More detailed information ofspending within the various government departments isprovided in the country chapters, where available.

Current investment in research and development1.6 The gross expenditures on research and development

(GERD) as a percentage of Gross Domestic Product(GDP) in 2000 for each of the five countries, ispresented in Figure 1 (Executive Summary)2. There is aconsiderable range in R&D investment in the countriesinvestigated. However, a clear distinction emergesbetween two groups. Finland, the US and Germany allspend close to or more than 2.5% of GDP on R&D andare above the Organisation for Economic Co-operationand Development (OECD) average. Finland is one of theworld's highest investors in R&D, spending almost 3.4%of its GDP on research. Of the OECD countries, onlySweden at 3.8% of GDP spends more. The other groupincludes the Netherlands, the UK and Canada, all ofwhose R&D expenditures fall towards the lower end ofthe range for western OECD countries.

2 Only exception is the Netherlands for which the numbers are from 1999.

10

part

one


1.7 Figure 3 provides an overview of the relative importanceof industry and government contributions to total GERD.Private expenditures are expressed in BusinessExpenditures on R&D (BERD, presented on the x-axis)and public expenditures as Government BudgetAppropriations for R&D (GBAORD on the y-axis)3. In allof the selected countries, BERD expenditures exceedGBAORD. In Germany, the US and Finland, GERD is atleast twice as much as GBAORD. Observed differencesbetween the countries reflect the structural mix ofindustries and their reliance on R&D. However, Finland'sstatistics demonstrate that when a government placespriority on the development of R&D intensive industries,it can have a large impact. Within the six countries, therange in GBAORD is much narrower than the range inBERD, although GBAORD in the UK and Canada isnoticeably less than in the other four countries.

1.8 As illustrated in Figure 2 (Executive Summary), both theUK and the US spend high levels of GBAORD ondefence-related R&D. Defence spending in thesecountries accounts for more than 50% of totalGBAORD. When the proportion of defence spending asa total of government spending on R&D is considered, adifferent picture emerges as to the importance thatgovernments place on research. After subtractingdefence spending from GBAORD, it becomes clear thatthe UK and the US are the lowest investors in civil R&D,

dropping below the civil R&D average expenditure forOECD member countries. In contrast, the importance ofcivilian R&D spending to the Netherlands, whichspends relatively little on defence, becomes clearer andthe leading position of Finland, with its marginaldefence spending, is even more accentuated.

1.9 The level of R&D investment reflects the proportion ofthe labour force employed in the R&D sector (Figure 4). Canada and Finland employ relatively moreresearchers than the Netherlands and Germany. Inrecognition of this situation, the Netherlands hasemphasised efforts to attract, develop and stimulatehighly talented researchers.

Developments in R&D investmentsover time1.10 In this section, longitudinal statistical data on GERD,

BERD and GBAORD indicate the direction ofdevelopment in particular areas in the six countriesthrough the 1980s and 1990s. The most noticeabledevelopment as shown in Figure 5 is the enormousincrease in R&D investment made by Finland. In 1981,Finland was one of the OECD's lowest spenders onR&D. By 2000, it was one of the highest. Thisturnaround can be attributed to measures that

Levels of GBAORD and BERD as % of GDP, 20003


GBA

OR

D %

of G

DP

BERD % of GDP

0.50

1.00

0.90

0.80

0.70

0.60

0.50

0.40

0.70 0.90 1.10 1.30 1.50 1.70 1.90 2.10 2.30

Canada

Finland

Netherlands

United Kingdom Germany

United States

3 For further explanations of the above terms, see the standard OECD methodology for the collection ofR&D statistics entitled The Measurement of Scientific and Technological Activities: Proposed Standard Practice for Surveys of Research and Experimental Development -- Frascati Manual 2002.

11

part

one


per

1000

labo

r fo

rce

% o

f GD

P

12.0

10.0

Researcher per 1000 labour force Total GERD as % of GDP

8.0

6.0

4.0

2.0

0.0

4.00

3.50

3.00

2.50

2.00

1.50

1.00

0.50

0.00

Canad

a

United

King

dom

Netherl

ands

OECD A

verag

e

German

y

United

State

s

Finlan

d

Research labour force compared to GERD, 20004


Gross Domestic Expenditure on R & D (GERD), 1981-20005

% o

f GD

P

1981

3.50

3.00

2.50

2.00

1.50

1.00

1989 1991 1995 1997 1999 2000

Canada

United Kingdom

Netherlands

Germany

United States

Finland

12

part

one


successive Finnish governments have implemented todevelop Finland's R&D sector. During the 1990s, grossexpenditure on R&D in most countries remained flat.Germany and the US have seen increases in R&D levelssince 1995 after years of considerable reductions. R&Dspending in the UK has fallen since 1981, droppingbelow the EU average for the first time in 2000.

1.11 Figures 6 and 7 show that Finland's growth in R&Dspending is primarily a result of an increase in businessrather than government investment, as indicated byGBAORD. The increase in industry spending accountsfor more than 75% of the total increase in R&Dinvestment. Increases in business-supported R&Dexhibit a similar but more modest effect on spending inthe US and Germany since 1995. Business spending onR&D in the Netherlands and the UK has remained stableand even decreased slightly in Canada over the lastseveral years. Government expenditure on R&D hasdecreased over the past two decades in all the selectedcountries. The most marked decline is in the UK, wheregovernment spending on R&D in 1999 was less than50% of its 1981 level, although following recentSpending Review settlements there has been a recoverywith the UK government spending, in 2000/01, 0.7% ofGDP on science, engineering and technology. The USgovernment has fallen from being the largest spender onR&D in the mid 1990s, to being the third largest now.

1.12 The remainder of the report presents detailedinformation about the institutional context of R&Dactivities in the five selected countries and describes thedifferent and innovative approaches to researchselection, procurement, implementation, management,evaluation and transfer. Each chapter follows the samestructure, namely:

! The institutional context with respect to R&D ispresented. The key players and theirinterrelationships are shown.

! The processes involving research strategising,priority determination and coordination are described.

! The selection and commissioning of research isexplained.

! Research outcome measurement and transfer isdescribed, particularly focusing on how researchquality is evaluated.

Business R&D expenditures, 1981-19996


% o

f GD

P

Year1981

2.50

2.00

1.50

1.00

0.50

0.00

1989 1991 1995 1997 1999

Canada

United Kingdom

Netherlands

Germany

United States

Finland

13

part

one


Government budget appropriations on R&D, 1981-19997


% o

f GD

P

Year1981

1.30

1.10

1.20

1.00

0.90

0.70

0.60

0.80

0.50

0.40

1989 1991 1995 1997 1999

Canada

United Kingdom

Netherlands

Germany

United States

Finland

Part 2

15

part

two

Institutional context2.1 Developed in 1996, the document "Science and

Technology for the New Century: A Federal Strategy"sets out the vision and organisation of science andtechnology research in Canada. Resulting from thisinitiative, new government structures have been put inplace and more explicit and structured approaches todetermining research strategies and research fundinghave been developed. Figure 8 shows the currentinstitutional context of science and technology researchin Canada.

Policy/funding level

2.2 At the highest level, reporting to a Secretary of State,the Cabinet Committee on the Economic Union(CCEU) annually reviews the government's progressand priorities in science and technology (S&T). Belowthat, the Assistant Deputy Ministers (ADM) Committee

on Science and Technology works to coordinategovernment-wide approaches to managing S&T and to ensure that departmental initiatives and priorities are shared across the federal science andtechnology community.

2.3 A number of roles can be distinguished among thescience-based departments and agencies (SBDAs) at thefederal level:

! A number of organisations, e.g., the CanadaFoundation for Innovation (CFI) and the grantingcouncils, allocate federal funds to support academic research.

! Industry research is funded mainly by IndustryCanada and the National Research Council (NRC).

! Specific topic-related departments, e.g., Health andEnvironment, mostly fund research conducted bytheir own institutes.

Part 2 Canada



Organisation of research responsibilities in Canada8

Parliament

SAB

Cabinet (CCEU)

SBDAs

ACST

CSTAADM Com on S&T

Universities Government Labs Research Institutes Private sector

16

part

two


2.4 The new federal strategy stresses ministerialaccountability for science and technology across SBDAs.It also established Industry Canada as the leadinggovernment department to coordinate S&T related issues.The Canada Foundation for Innovation was formed in1997 and has become the single largest funder at thefederal level, consolidating the budgets of various otherSBDAs and spearheading increased attention oninnovation overall. It mainly awards funds to universities,research hospitals and private non-profit organisations tosupport their research infrastructure. The funding budgetof three granting councils was also increased and togetherthey have become the largest source of R&D funding inCanada at the federal level. The established budgets andassociated programmes have had a significant impact onacademic research in Canada, but much less influence onintramural governmental research.4

2.5 Despite the increasing concentration of funds foracademic research, the overall R&D budget remainsfairly stable and fragmented across a large number ofdepartments and agencies. However, in 2001-2002, thefederal government planned to spend $4.6 billion onR&D, a 10% increase over 2000-2001, reflectingCanada's resolve to move from fifteenth to fifth placeamong the most competitive and innovative nations.

2.6 Since the publication of the federal Science andTechnology (S&T) strategy, all federal SBDAs have takensteps to increase the effectiveness of federally sponsoredresearch. These steps include establishing new planningand reporting mechanisms and instituting expert reviews,client surveys, impact studies, partnership building andbenchmarking exercises. As an example, Health Canadarecently appointed a Chief Scientist from academia. Oneof his primary responsibilities is to ensure governmentalresearch is subjected to the same degree of peer scrutinyand evaluation as academic research.

Advisory roles

2.7 In "Science and Technology for the New Century: AFederal Strategy", the government made thecommitment that "each science-based department andagency will set clear S&T targets and objectives,establish performance measurement indicators based onoutputs, develop evaluation frameworks, and maintainmechanisms for external advice and review."Accordingly, most of the science-based departmentsand agencies have established external advisory bodies.These Science Advisory Bodies (SABs) have replaced theindustry-related advisory boards and are consulted onissues related to policy formulation. The SABs bringrelevant knowledge and expertise on S&T issues andadvise on broader policy issues from an S&Tperspective. The roles of the SABs vary, and, while theirvalue is undisputed, there are concerns that their

potential contribution to policy making is not beingmaximised. A government report, "Reinforcing ExternalAdvice to Government" (READ), recently examined theroles and contributions of the SABs to ensure theirresource potential is being exploited.

2.8 Established in April 1998, the Council of Science andTechnology Advisors (CSTA), provides the CabinetCommittee for the Economic Union (CCEU) withexternal expert advice on internal federal S&T issues. Itcomprises primarily representatives from the SABs, whoreport to ministers or other senior officials of SBDAs. TheCSTA draws these advisors into a single body to improvefederal S&T management by examining issues commonto a number of departments, and by highlightingopportunities for synergy and joint action.

2.9 The Advisory Council on Science and Technology(ACST) provides the Canadian Prime Minister withexpert, non-partisan advice on S&T goals and policiesand their application to the Canadian economy. Thisbody reviews Canada's performance in S&T, identifiesemerging issues and advises on a forward-lookingagenda. It comprises leading industrialists andacademics with S&T experience and knowledge. Its roleand composition are comparable to many advisorycouncils in other countries.

Research providers

2.10 Compared to other countries, relatively large amounts ofthe Canadian federal R&D budget of $4.6 billionsupport intramural research activities. More than 40% of the total budget funds 120 federal researchlaboratories to conduct research for governmentdepartments, particularly agriculture, energy, defenceand health and environment. The National ResearchCouncil (NRC) is Canada's foremost federal scientificresearch organisation and is responsible for

Distribution of R&D funding among Federal SBDAs9

CFI (innovation)

Environment

Industry

Development Aid

Granting Councils

Statistics

Other

6%

7%

7%

7%

49%

10%

14%

4 Examples of these are the Canada Research Chair Programme (annual investment of Can $300 million), Genome Canada (funded with a Can $300 millionendowment) and the indirect cost payment programmes for universities.

17

part

two


approximately 20 research institutes. Furthermore, thereare several federal science-based departments, such asNatural Resources Canada (NRCa), who also conductresearch in their own labs. Universities receiveapproximately one third of the total federal funds,mainly from the Canada Foundation for Innovation (CFI)and the granting councils. Universities are also fundedby provincial governments who receive a largelyundirected allocation from the federal government foreducation research.

Priority setting and coordinating processes

Identifying emerging issues

2.11 Since the introduction of the S&T strategy in 1996, R&Dpriority setting has taken on new levels of importancewithin the federal government. Newly establishedand/or restructured external S&T advisory boards andtechnical review panels are now consulted bydepartments and agencies reviewing the relevance oftheir S&T programmes and activities. SABs helpdepartments meet their science mandate by focusingattention on relevant lines of scientific inquiry. FederalSBDAs help steer internal R&D planning towardsemerging areas of high priority. The READ report madeclear that no single model of external S&T advice willmeet the needs of all government departments. Thesignificance of future planning has been recognised bydepartments aiming to identify relevant issues forscientific enquiry. At the same time, however, manydepartments feel that they lack the capacity to conductplanning and horizon scanning activities.

Coordinating S&T activities

2.12 The 1996 federal S&T strategy provides a framework forenhanced cooperation and collaboration among SBDAs,and between federal and non-federal partners on issuesof shared concern. Since the release of the strategy,federal departments and agencies have significantlyimproved how they interact with other players in theCanadian and international research environment. Whileeach SBDA is responsible for setting its own S&T policiesand research activities, the need for cooperation acrossSBDAs remains paramount and should go beyond simplecoordination to include collective action. The AuditorGeneral noted this in 1998. Several facilitativemechanisms have since been put in place.

2.13 As mentioned above, the Cabinet Committee on theEconomic Union (CCEU) conducts an annual review ofthe government's progress and priorities in S&T researchactivities. Beyond that, the ADM Committee on S&Tcoordinates government-wide approaches to managing

S&T and ensures that departmental initiatives andpriorities are shared across the federal science andtechnology community. In addition to the Cabinet'sexplicit S&T mandate, the Advisory Council on Scienceand Technology (ACST), with eminent representativesfrom industry and academia, was created in 1996 toreview the nation's performance in S&T, by identifyingemerging issues and advising on a forward-lookingagenda. While ACST and also CSTA are positioned tocoordinate research policies and strategies, as yet, theirmandate and level of activity have been too restricted tomake a significant impact on the formulation of anintegrated policy on research and development.

2.14 The Assistant Deputy Ministers (ADM) Committee on S&Thas taken an overall coordinating role of broad, sharedinterests, e.g., science capacity and technology transfer.5

The committee adopted the following new mandate:

! to implement appropriate cross-government S&Tplanning, e.g., the sensible use of federalinvestments in S&T and the sharing of best practices;

! to develop proposals and advise government on keyshared S&T policy issues; and

! to provide a forum for interdepartmental consultationon S&T policy and programme directions, sharing ofinformation, and coordination of efforts andinitiatives across the federal S&T community.

2.15 The committee comprises assistant deputy ministers orequivalent-level representatives from departments andagencies with S&T activities and/or interests. With thestrategy in place, the ADM Committee has helpeddevelop a stronger sense of community among federalS&T bodies, by encouraging information-exchange andraising the profile of S&T issues within the government.These activities have facilitated increased cooperationand collaboration between departments and agencies.The committee has provided the Canadian Cabinet witha clearer and more comprehensive picture of the federalS&T effort, and is currently working to create newmechanisms for addressing national S&T needs in waysthat integrate capabilities across the federal governmentand throughout the S&T community.

Selecting and commissioning research2.16 Various CSTA reports have shown that most government

agencies base their decisions about the need for variousS&T initiatives on in-house advice and working groupsrather than by seeking independent reviews. As a resultof one of the CSTA reports, the Canadian governmentdeveloped a "Framework for Science and TechnologyAdvice: Principles and guidelines for the effective use of

5 Smith and Halliwell (1999).

18

part

two


Science and Technology advice in government decisionmaking".6 One of the guidelines stresses that "whileadvice from external and international sources needs tobe sought regularly, it is especially important to seeksuch advice when:

! the problem raises scientific questions that exceedthe expertise of the in-house staff;

! the issue is 'horizontal' or cuts across lines ofjurisdiction within or among departments;

! there is significant scientific uncertainty;

! there is a range of scientific opinion;

! there are potentially significant implications forsensitive areas of public policy; and

! where independent scientific analyses canstrengthen public confidence."

A subsequent CSTA report "Science and TechnologyExcellence in the Public Service" (STEPS) reviewed waysto ensure that the best possible advice was indeedsought by focusing on employing external, expertreview processes to support project selection.

Measuring outcomes2.17 The concept of "Linkage and Exchange" developed by

the Canadian Health Services Research Foundation isdiscussed in detail in "Getting the evidence: Usingresearch in policy making". The model proposes thatbottlenecks arise in the communication betweenresearchers and policymakers that can prevent theapplication of research into effective policies. Toovercome these barriers, the model suggests involvingpolicymakers in priority setting, programme fundingdecisions, assessing research applications, conductingresearch and communicating findings. The Foundationhas adopted the model aiming to increase the impact oftheir research on policymaking. The effects andtransferability of the model are yet to be fully tested,however, other organisations active in funding healthresearch, e.g., the Canadian Institutes for HealthResearch, have stated similar aims and approaches.

2.18 The CSTA have established principles and guidelines forbetter incorporating scientific research in governmentdecision making. They state:

! The government needs to anticipate, as early aspossible, those issues for which science advice willbe required, in order to facilitate timely andinformed decision making.

! Advice should be drawn from a variety of scientificsources and from experts in relevant disciplines inorder to capture the full diversity of scientific schoolsof thought and opinion.

! The government should employ measures to ensurethe quality, integrity and objectivity of the scienceand science advice it uses, and ensure that scienceadvice is considered in decision making.

! Science in public policy always contains uncertaintythat must be assessed, communicated and managed.Government should develop a risk managementframework that includes guidance on how and whenprecautionary approaches should be applied.

! The government is expected to employ decisionmaking processes that are open, as well astransparent, to stakeholders and the public.

! Subsequent review of science-based decisions isrequired to determine whether recent advances inscientific knowledge have an impact on the scienceadvice used to reach the decision.

The CSTA also recommends appointing Science AdviceChampions to guide and encourage the implementationof these principles and guidelines.

2.19 In an effort to ensure increased value for money from S&Tinitiatives, the federal SBDAs conduct R&D impactanalyses to assess the outcomes and results of S&Tactivities. The R&D Impact Network and the Programmeof Energy Research and Development (PERD) haveimplemented results-based performance measurement,two examples of how government is aiming to ensurerelevance and value for money. The R&D Impact Networkwas established by National Resource Canada (NRCan),the Treasury Board and other research partners to advanceR&D impact assessment and provide researchorganisations with simple, credible and broadly acceptedperformance measurement tools for results-basedmanagement and decision making. The network hasrefined and adapted tools for measuring the social andeconomic impacts of R&D, developed strategies tocommunicate impact information and promoted theexchange of best practices. Many departments andagencies now routinely make use of R&D impact analysisin addition to traditional audit and evaluation practices.

2.20 PERD is a competitive process administered by NRCanthat provides funding for non-nuclear federal energyR&D. After reviewing PERD programming in 1999,NRCan negotiated a revised memorandum ofunderstanding with the 12 participating federaldepartments and agencies, incorporating newaccountability provisions and performance measures.Implementation of results-based performancemanagement is leading to closer monitoring of theresults of work conducted with PERD funds and to betterresource allocation decisions in the future.

6 This effort and the subsequently published framework drew heavily on guidance by the OST in the UK.

Part 2

19

part

thre

e

Institutional context3.1 In Finland, the ultimate authority about science policy

and decisions involving granting and allocatinggovernment research funds resides with Parliament andthe Cabinet of Ministers (CoM). The CoM relies on thescientific expertise provided by the Science andTechnology Policy Council (STPC) headed by the PrimeMinister. The STPC is responsible for directing andintegrating science and technology policy, developingscientific research and education and for organisingFinland's participation in the international science andtechnology community7

3.2 Figure 10 shows the institutional context in whichpolicymakers, research funders and research providersoperate in Finland and the relationships among them. At

the highest level, the CoM, aided by the STPC, isresponsible for developing science policy. The variousministries then translate this policy into specific projectsthrough their various research institutes. The researchinstitutes finance these individual projects. Both publicand private bodies carry out research.

3.3 The Ministry of Education is the general scienceministry and bears primary responsibility forpromoting basic research. The universities and certainresearch establishments are subordinate to thisministry. The Academy of Finland, whose mandateincludes advancing scientific research, developinginternational research cooperation and serving as anexpert body on questions relating to science policy,also reports to the Ministry of Education.

Part 3 Finland



Institutional context of science and technology in Finland10

Cabinet of MinistersScience and Technology

Policy Council

Ministry of Education

Academy ofFinland

Universities Ownorganisations

Ministry of Trade & Industry

Research Institutes

Industry

Industry

OtherMinistries

TechnologyDevelopmentCenter (TEKES)

7 Euromecum, Euroresearch: An overview of Research Policy in Europe, 1996.

20

part

thre

e


3.4 The Ministry of Trade and Industry is responsible for thepromotion of technology, as is its reporting body, theNational Technology Agency (Tekes). Tekes seeks topromote the well-being and stable development ofsociety by enhancing, either directly or indirectly,industry's capacity for technological innovation throughcompetitive products, production methods and services.

3.5 The Finnish National Fund for Research andDevelopment (SITRA) is an independent public fundthat provides venture capital for high technologycompanies and has its own research programme. Otherimportant public financing sources for start-upcompanies are the Regional Development Fund (KERA)and Finnish Industrial Fund (Suomen TeollisuussijoitusOy) which invests mainly in a variety of regional funds.

3.6 The Technical Research Centre (VTT) is one of the largestresearch institutes in Scandinavia. VTT reports directly tothe Ministry of Trade and Industry, although most fundingis channelled through Tekes. The Centre conducts researchfor a wide range of clients including other ministries,foreign clients and private industry. A third, more minor,element of the Finnish public research system comprisessectoral ministries such as Health and Agriculture andForestry with their own research institutes. The mostsignificant of these are the National Research andDevelopment Centre for Welfare and Health (Stakes) andthe Finnish Forest Research Institute (Kurki, 1996).

3.7 Despite the dramatic recession that hit Finland duringthe early 1990's, the Finnish government has investedheavily in R&D during the last decade. R&D spendinghas grown faster than overall government spending.There is a wide consensus on the overridingimportance of R&D policy and the need to viewdifferent policies as parts of a whole.

3.8 Figure 11 shows the development of the gross domesticexpenditure on R&D (GERD) from 1991 to 2001. Itindicates that growth in GERD is driven mainly byprivate business, while funding from public anduniversity sectors has remained fairly stable. Thegovernment investment in R&D has grown to €1.4 billion in 2002, increased from the previous yearby about €50 million. Government research fundingamounted to 4.5% of total government expenditure,excluding state debt management cost, and remained atthe same level as in 2001.8

3.9 The share of total public R&D expenditure (shown inFigure 12 as both public sector and universityfunding) of the gross domestic product was 1.05% in1999, while the EU mean was only 0.7%. As a resultof rapid growth in GDP the share of public R&Dfunding has decreased to 1.0% in 2001.9 In 2000, thegross domestic expenditure on R&D (GERD) was3.4% of GDP. Government financed 26% of GERDwhile 70% was financed by industry.

8 Source: www.research.fi9 Source: www.research.fi

20012000199919981997199519931991

Gross domestic expenditures on R&D (GERD)11

Perf

orm

ance

of G

DF

4

3.5

3

2.5

2

1.5

1

0.5

0

Year

GERD Business enterprises

University sector Public sector

Source: Satistics Finland

21

part

thre

e


3.10 Figure 12, shows the distribution of research fundingacross the national funding agencies. Tekes (Ministry ofTrade and Industry), the Ministry of Agriculture andForestry and the Ministry of Social Affairs and Healthfund most of the R&D activities. Although the R&Dexpenditure of the Ministry of Education is quite small,its reporting body, the Academy of Finland, isresponsible for developing basic research policy.

Priority setting and coordinating processes3.11 Coordination and priority setting in Finland is

approached in two distinct ways:

! Horizontal coordination through STPC and agencylevel cooperation aims to avoid duplication of effort.

! Vertical coordination/integration seeks to translatepolicy needs into research strategies, with Tekes as akey player.

3.12 In Finland, the STPC takes a prominent role indetermining research strategies for the entiregovernment. Unlike advisory councils or sub-councilsof cabinets at this level, it has real decision makingpower, as it is headed by the Prime Minister andincludes members from both government and keystakeholder organisations. The members must includerepresentatives of the Academy of Finland, Tekes,universities and industry as well as employer andemployee organisations.

3.13 The STPC has a visible role coordinating innovationpolicy activities at a national level. It is a strategybuilding body, employing a consensus approach todecision making that minimises conflict andduplication across different ministries. While Finlandis clearly a small country, it is not just size thatfacilitates the consultative process. The Finnishpolicymaking culture of foundation laying via informaldiscussion promotes consensus building, oftenobviating the need for more formal procedures. TheSTPC has the following responsibilities:

! To direct S&T policy and ensure its nationalcompatibility and to prepare relevant plans andproposals for the Council of State.

! To develop scientific research and education,prepare relevant plans and reviews for the Councilof State and to review the need for research invarious fields.

! To follow up and assess measures taken to developand apply technology and to prevent or solveeventual problems associated with this.

! To handle important issues relating to Finland'sparticipation in international scientific andtechnological cooperation.

! To issue statements on the allocation of publicscience and technology funds to the variousministries, and their allocation to the various fields.

! To handle the most important legislative matterspertaining to the organisation and prerequisites ofresearch and the promotion and implementationof technology.

! To take initiative and make proposals in mattersunder its competence for the Council of State andits ministries.

3.14 While STPC is the example of horizontal coordination ata policy level, agency level cooperation between Tekes,Academy of Finland, Sitra, Finnvera, TE-centres,Finnvera, Finpro and TESI (Industry Investments Ltd) alsoexists. Among research providers, the main platform forhorizontal coordination is at the programme level. Thisis especially true of the Tekes technology programmes.Tekes has the main coordination and implementationresponsibility for directing public R&D funds to industry.In 1998, Tekes distributed around 90% of the publicR&D money directed to industry; 20% was given tocompanies as loans and the rest as subsidies for productdevelopment and research.10 Tekes operates by:

! coordinating and implementing technologyprogrammes, as well as providing funds forindividual R&D projects;

10 Tekes Annual Report 1998.

Distribution of R&D funding across main government departments

12

Source: Statistics Finland, http://www.stat.fi/tk/yr/tttiede_rd11_en.html (updated 07.02.02)

Other11%

Defence7%

Social Affairs and Health

17%

Agriculture and Forestry

20%

Trade and Industry

45%

22

part

thre

e


! offering strategic advice and technology expertise tocompanies, focusing on small to mid-sizedenterprises (SMEs) and new technology-basedbusinesses; and

! developing the innovation environment, byencouraging networking, disseminating informationand participating in technology policy discussions.

3.15 The Academy of Finland supports research activities inuniversities. It also has in-house research programmesthat provide additional research funds for universitiesand focus on specific themes. During the 1990s, the Academy of Finland and Tekes increased theirlevel of cooperation.

Selecting and commissioning research3.16 In Finland, cluster, technology and research

programmes are increasingly viewed as strategic meansfor funding research. Historically, these programmeshave played an important role in pursuing scientificpolicy objectives. They are multidisciplinary, usuallyexist for a fixed period of time (around three years), andoften involve consortia combining several researchprojects. Such programmes have provided an effectivemeans for selecting and involving various research-related organisations and for stimulating cooperationand networking opportunities among private companiesand the research community.

3.17 In Finland, research projects are generallycommissioned as a result of companies and/or researchinstitutes responding to a public call for proposals.Proposals are usually evaluated by committees based ona loosely defined set of criteria. Specification of thework required is often left quite open, and proposalsand the ensuing projects can therefore vary widely incontent. Programmes are under no obligation to spendall of their budget if the quality of proposals is low.

3.18 Technology programmes have been introduced recentlyto promote development in specific sectors oftechnology or industry, and to transfer research results tobusiness in an efficient way. Private companies, researchinstitutes and Tekes jointly plan the technologyprogrammes in work-groups and open preparatoryseminars. The decision whether or not to launch aparticular programme is made by the Tekes board. Eachtechnology programme has a steering group, a co-ordinator and a responsible person at Tekes. Programmeduration ranges from three to five years and fundinglevels range from €6 million up to hundreds of millionsof euros. Tekes usually finances about half of the cost of

each programme. The remaining funding comes fromparticipating companies. Foreign evaluators assessmany of the completed programmes. The main benefitslie in the close cooperation between research institutesand industry, the widespread involvement of small andmedium sized companies and the high level ofinternational cooperation.

3.19 The Academy of Finland is the central financing andplanning body for basic research. The majority ofAcademy funding supports basic research conducted inuniversities. The Academy of Finland uses researchprogrammes to achieve its goals relating to sciencepolicy. Various annually launched research programmesare gaining more and more importance. Theseprogrammes consist of several connected projectswithin the same field of research. Programmes focus ontopical research problems, important research fields andnew fields and disciplines generated by researchfindings. The aims of the programmes are to raise thestandard of research, to promote interdisciplinary,multidisciplinary and international research, to advanceresearcher careers and researcher networking, tointensify researcher training and to contribute to creativeresearch environments. They are set up for a fixed term,usually three years, and are often co-financed by otherpartners. In 2001 the Academy supported23 programmes, of which four were new and six wereabout to be completed.

3.20 The STPC guidelines also set out funding principles, inparticular the targeted allocation of financing and theallocation of funding on the basis of competition. Withregard to university training, graduate placement isconsidered one performance indicator and is used todirect resource allocation. In research, the selection ofso-called 'Centres of Excellence' by the Academy hasstrengthened the competitive element among researchgroups.11 The Academy funds the best researchers,research teams and the most promising youngresearchers. Funding decisions are based on carefulscientific assessment of the applications and cover afixed period of time.

Monitoring and managing Research

3.21 Research programmes and individual projects areusually overseen by steering groups. Generally, duringthe course of a project the burden of reporting is kept toa minimum. Private contractors are often only requiredto provide proof that they have conducted the work;other public documentation, such as the researchresults, is not demanded. Projects conducted byresearch institutes are generally required to report moreformally and produce publicly available results. Projectwork generally follows the submitted proposal and,

11 Centres of Excellence are selected on the basis of international evaluations for a period of six years. They are subject to interim evaluations after three years.If the evaluation is negative, the unit is given one year in which to correct its shortcomings; if it fails to do so, it loses its Centre of Excellence status.

23

part

thre

e


because Finland is a small country, failure to meetproject goals becomes well-known, jeopardising futurefunding opportunities. Recently, more attention hasbeen paid to project dissemination activities andnetworking of information providers and users.Programmes usually include seminar activities wherethe results from different projects are presented at aprogramme level.

Measuring outcomes3.22 Evaluation has long played an important role in policy

formulation in Finland. National policies, institutionsand research programmes have been subjected toextensive evaluation, especially since the early 1990s.The effectiveness of government action is assessed atdifferent levels. All the major bodies (e.g., the variousministries, Tekes, the Academy of Finland and researchinstitutes) have been evaluated by external teams.Experts from abroad are commonly involved inevaluation panels. In this way, new evaluation examplesare taken into account and the objectivity of theevaluation is increased. Furthermore, the opportunitiesfor sharing and disseminating knowledge, practice andpolicies across countries are increased. Evaluations feedinto recommendations for future programme, projectand instrument development.

3.23 A primary responsibility of researchers at Tekes is toassess the impact of technology on economicdevelopment. The findings are used to steer technologyfunding and the development of the technologyprogrammes. Impact analysis has been integrated intoTekes operations, which are steered according to impacttargets. Impact is monitored and evaluated at the projectlevel. Tekes also commissions external evaluations of allnational technology programmes.

3.24 Towards the end of the 1980s, all Tekes technologyprogrammes were evaluated. Since that time, expertshave evaluated more than 60 programmes. In recentyears, evaluations have become more structured andstandardised and are the responsibility of a specificallycreated Impact Analysis Directorate, consisting of anEvaluation Unit and a Quality Unit. As far as possible,external evaluations and international experts areemployed. The Impact Analysis Directorate usesexternal experts to evaluate technology programmesand to compile varied and independent effectivenessdata. The evaluations highlight research anddevelopment practice and the factors contributing to itssuccess or failure. One evaluation can cover several

programmes if they belong to the same field or cluster,if they have similar goals, or have some relevant featurein common. Impact assessments also aim to provideanswers to ongoing queries and make it easier for thoseconcerned to respond to changes.

3.25 Tekes technology programmes are always evaluated uponcompletion and sometimes midway through. Theevaluation provides feedback on whether and to whatextent the programme aims have been realised and on therelevancy of the programme. It also produces informationto support the strategic development of future programmeactivities and the activities of Tekes in general.

3.26 Evaluation is a primary function of the Academy ofFinland. Expert opinions, impact analysis, disciplinaryevaluations and research system reviews all serve asevaluation mechanisms. Between 1983 and 2000 theAcademy conducted a total of 21 disciplinaryevaluations. Reviews of the whole research system werepublished in 1997 and 2000. A few discipline-basedevaluations are carried out annually and greatlyinfluence the future development of the field. At theMinistry of Education, a Higher Education EvaluationCouncil was set up to assist higher education institutionsconduct evaluations.

3.27 Over the past years, the structure of research instituteshas been revised after feedback from peer review. VTTwas reorganised throughout 1993 and 1994, primarilyto become more customer-orientated, improve internaland external interaction and to support functional andeconomic rationalisation. Direct budget appropriationsallocated to state research institutes decreased in realterms between 1992 and 1996 by as much as one-fifth.Research organisations compete for researchassignments and financing and also form partnerships.The trend towards sectoral research is eliminating strongdemarcation lines between different fields and betweenand among ministerial research institutes and otherresearch organisations. Research institutes now alsocompete with university research groups for Tekes andAcademy funding. Academic research groups have beenespecially successful in acquiring external funding fromthese sources. The government has acknowledged thesuccess of universities in the research provider marketand has allowed universities more freedom to definetheir own research priorities, while simultaneouslytightening financial accountability.

Part 2

25

part

four

Institutional context4.1 In Germany, the context in which science policy is set and

research produced is complex, due both to the diverse andfragmented research provider base, and becauseresponsibility for science and technology policy is splitbetween the federal government and the Länder (states).The resulting institutional context reflects the size of thecountry, the federal structure and significant historicaldevelopments. In Germany there is no single, central bodydetermining research and funding policies, in fact there isa fundamental philosophy of separation between thefunders and providers of research, reflected by the numberof institutions playing an intermediary role. Figure 13provides an overview of the flow of institutional andproject-oriented R&D funding in Germany, depicted bysolid and shaded lines, respectively.

4.2 The responsibility for education, including highereducation and academic research, lies solely with thevarious states (Länder). They provide the basic fundingand institutional support for universities as well as anumber of independent research institutes. At the statelevel, there is very little programmatic coordination, letalone control, of research. An indirect influence isexercised by those bodies that provide additional funds (Drittmittel) to supplement the budgets of thevarious research providers. Where the states have anycontrol at all over setting research strategy, it tends to take the form of medium-term plans developed by the joint Federal/Länder Commission for Education Planning and Research Promotion (BLK), andthematic guidance to universities through the ScienceCouncil. The Federal government supports almost allresearch providers by giving institutional support to

Part 4 Germany



System of Research Funding in Germany13

Government Laboratories

National Research Centres, Max Planck Society, Fraunhofer Society,

Blue List

Higher Education

Länder Institutes, Academies

Corporate Laboratories

AiF Institutes

Industry-related R&D

Federal Government

Science Council

Länder

BLK

Foundations

Private sector

DFG

Source: Cave et al (1999)

26

part

four


government research establishments and the Deutsche Forschungsgemeinschaft (DFG) and project-orientedsupport to other providers.

Research funding agencies

4.3 At the federal level, the main research funders are the Federal Ministry of Education and Research (BMBF),the Ministry of Defence, the Ministry of Economy and Technology (BMWi) and the DeutscheForschungsgemeinschaft (DFG). The DFG is equivalentto Research Councils in the UK and receives moneyfrom the BMBF and the various state governments. Itsupports academic research that is aligned with its ownresearch strategy. As shown in Figure 14 below, BMBF isthe largest research funder, disbursing more than 60% ofthe total federal R&D budget.

4.4 The states also provide an important source of funding,primarily offering institutional support to highereducation institutes and research institutes. Some states,for instance Bavaria through the Bavarian ResearchFoundation, have specific agencies to oversee researchthat supports their mission. State R&D expenditurealmost equals that of the federal government and theirfunding efforts are coordinated to some extent by theGerman Science Council (Wissenschaftsrat) and the BLK.

4.5 The BMBF focuses on applied research and tends toemploy a top-down approach to its allocation of funds,broadly divided between institutional and appliedresearch, the latter being complementary to the DFG.Institutional funding (approximately 60% of BMBFbudget) is mostly disbursed under a federal frameworkagreement. German research institutions such as theFraunhofer Society (FHG) which mainly conductsapplication-driven research, or the HelmholtzAssociation that assembles 16 national laboratories withremits covering a broad range of research fields fromparticle physics to cancer research, or the Max-Planck-

Gesellschaft (MPG), are the typical recipients of thisfunding. Such research institutes receive approximately90% of their funds from the federal government andonly 10% from the states.

4.6 The various departmental research institutes are thosemost closely linked to the federal government. Theirresearch relates directly to the mission of a particularministry and informs ministerial policy making. BMBFfunds specific projects at some government researchinstitutes, for instance at the Robert-Koch institute.Within the framework of its government researchprogrammes, BMBF primarily funds applied researchprojects that adhere to particular key themes, such aslaser research or nanotechnology. Project selectionresults from a strategically driven top-down steer, whichsometimes has a detrimental effect on the funding ofnew projects. Quality control is carried out throughinternational reviews.

4.7 Research funding by DFG is governed by a bottom-upapproach, with "Normalverfahren", or normalprocedures, at its core. In the various DFG programmes,both individuals and their institutions can compete forfunding which is awarded on the basis of scientificexcellence. This approach aims to encourage innovativethinking at an individual level. However innovativeproposals can still be difficult to support as the fundingdecision relies on a high degree of consensus at a supra-regional level.

Advisory Councils

4.8 The BLK provides a permanent forum for discussingquestions concerning education and research ofcommon interest to the federal and state governmentsand submits recommendations to policy makers at bothlevels. The Science Council (Wissenschaftsrat) isGermany's independent science policy advisory body.Its members include representatives of the federal andstate governments and eminent scientists. Its remitcovers higher education and research policy issues,including the evaluation of research organisations andindividual institutes, proposals for universityinfrastructure investment and the structure of theresearch system.

Distribution of research and development fundingover Federal ministries

14

Source: BMBF BERICHT Faktenbericht Forschung (2002)

Other11%

Defence13%

Education67%

Economy9%

27

part

four


Research Providers

4.9 The research provider base has traditionally beendivided among:

! Independent academic centres at over 100universities and similar institutions with 140,000full-time research staff.

! Government research12 at federal and state researchinstitutes (Bundes- and Ländesforschungsanstalten)that conduct research directly related to thesponsors' missions (Ressort-Forschung).

! Other research institutes, for instance the MPG, the FHG, the 16 national laboratoriesGroßforschungseinrichtungen (GFEs) nowassembled under the umbrella of the Helmholtzsociety, the institutes of the Leibnitz-Gemeinschaftand others. Together these institutes employ about70,000 full-time research staff.

! Product-oriented industrial research laboratories, forinstance the Confederation of Industrial ResearchAssociations (AiF) which conducts research onapplied topics particularly for small to mid-sizedcompanies (SMEs) and is funded by a mix of publicand private sources.

Priority setting and coordinating processes4.10 Despite the complexity of Germany's decentralised

research system, there are distinct functional roles thatfederal and state institutions have assumed to providecoherence and coordination of overall research strategy,research structure and programme focus of civilresearch. Towards these aims, two organisations havebeen assigned specific advisory responsibilities: theScience Council and the BLK. In addition, the federalgovernment and state ministries have devolved fundingresponsibilities to organisations with specific researchcommissioning functions. The central independentorganisation for funding and coordinating academicresearch is the DFG. It supports the entire spectrum ofresearch from the physical sciences to the humanities.Research priorities at government level are to someextent determined by the BLK, which is responsible forcoordinating federal and state research and educationpolicies, and developing medium- and long-termresearch programmes. Its three main areas ofresponsibility are:

! to coordinate the federal and state governments'research policy planning and develop their medium-term plan;

! to prioritise and make recommendations concerningthe mutual exchange of information between thefederal and state governments in matters of researchpromotion; and

! to propose to the heads of the federal and stategovernments annual grants for research institutions,research funding organisations and jointly financedresearch projects.

4.11 The recommendations of the Science Council are notbinding. They advise on issues broadly relating toscience and evaluate the structure and institutions of theGerman research system. The Science Council iscurrently examining the potential role for a NationalAcademy of Science and is reviewing the existingsystem of scientific advice to government.

4.12 The DFG is the principal intermediary organisation thatcoordinates activities between research funders andproviders. Its corporate membership includesrepresentatives from universities, research institutes andacademies and it performs both intermediary andresearch provider roles. Project funding decisions,particularly those affecting universities, are taken bybodies comprising more scientists than governmentrepresentatives. To promote innovation and as the mainsource of support for basic research, DFG stresses theimportance of adopting a bottom-up approach toproposal generation. Approximately 40% of DFG fundsare disbursed to investigator-initiated peer reviewedproposals, judged on the scientific quality of eachproject rather than on considerations such as theirpotential social or economic impact. DFG does notattempt to steer research in particular areas.

4.13 In summary, government-level research is coordinatedat 3 levels:

! at the ministerial level within research institutions,where strategic themes are discussed and the DFGpresident is present.

! within the Science Council which provides broadscientific advice and evaluates the structure andinstitutions of the German research system. Thecouncil represents both state and federalgovernment but its advice is not binding.

! among the advisory bodies and peer reviewcommittees of the BMBF at a programme level.Rounds of experts review and evaluate researchproposals and ensure that research objectives areachieved. The DFG would be included in thisevaluation process, but its primary objective is basicresearch whereas BMBF has a programme-specificfocus. Currently, approximately 20 suchprogrammes have been defined.

12 Typically guided by considerations of security, standardisation and other public concerns.

28

part

four


Selecting and commissioning research4.14 The degree of autonomy granted to research

institutions to determine their own research prioritiesand strategies is unique to Germany. In theory,research content is entirely independent of stateinfluence. However, there is a long-standing, mutuallyadvantageous relationship of cooperation between thegovernment and research institutions.

4.15 Specific research projects are selected andcommissioned according to a framework ofprogrammes defined on the basis of either particularthemes or disciplinary fields. However, all steps in theresearch process (e.g., purchase, execution,administration, evaluation) are the responsibility ofappointed research management organisations(Projektträger) outside the government. Often theseorganisations are research institutes themselves, such asthe German Air and Space Institute, the projectmanagement organisation Julich, (housed within theJulich Research Centre), or Fraunhofer projectmanagement. As authorised funding agencies for thegovernment, their responsibilities combine bothscientific-technical and administrative roles. They alsoorganise project seminars and workshops and assist withresults dissemination on behalf of their contractors.

4.16 Intermediary management agencies became establishedwhen the increase in research sponsored by federalministries became too great to manage in-house. Theagencies are funded typically by federal money andprovide a higher level of relevant R&D support than theministry itself. They also function as an internationalpoint of contact for other research or governmentorganisations in their respective areas of expertise.

4.17 Research support for BMBF and BMWi activities hasbeen accompanied by various quality assurancemeasures, relating to both institutional andprogramme/project funding. An assessment of thepotential results takes place even before a project isfunded, and monitoring continues throughout thelifecycle of the project. Developing and checking theperformance plan has become mandatory, enablinggovernment departments to monitor how researchfunding is being used.

4.18 DFG supplies grants to academics on the basis of reviewand competition. Most reviewers are selected by theprogramme directors and managers in the DFG office,but a core of some 600 are elected each year. Projectselection and review are maintained as separateactivities. However, this independence has recentlycome under threat as increasing workloads have madereviewers reluctant to serve on committees.Furthermore, as demand for proposals increasinglyoutstrips supply, criteria other than scientific merit arebeginning to play a part in funding decisions. Inresponse to falling approval rates, the DFG hasdeveloped adaptive mechanisms, such as a three-yeargrant to supplement existing 1 and 2-year grants toreduce the overall volume of applications.

4.19 In contrast, priority programmes in specific areas makeuse of invited calls for proposal. Funding for researchunits, clinical research and centres of excellence arecomparable to the UK programme of centre funding.Funds are available for a limited 6 to 12-year period,compared to 10 years in the UK. These funds areallocated competitively, against individual grantapplications. On average, between 3 and 10 individualgrants are awarded, spread over a 6-year period.

4.20 The Helmholtz Association, one of Germany's four non-university research organisations, is in the process ofmajor structural reform following a recent review by theScience Council. Helmholtz has an annual budget ofover €2 billion and employs more than 25,000 staff. Infuture, its research activities will be carried out in-houseand funded on the basis of strategic researchprogrammes rather than as a part of institutionalfunding. The reform efforts aim to increase competition,promote collaboration between individual centres andthus improve the exploitation of synergies. Developingpriority programmes will continue to be an ongoingprocess, taking into account the federal government'sresearch priorities as well as input from the sciencecommunity and industry.

4.21 When the programme-based funding strategy is in place,about 80% of all government funding will be allocatedon a competitive basis under specific programmes.Approximately 90% of the funds will come from thefederal government and 10% from the states. About 20% of the public sector funding will remain unallocatedto be used on a flexible basis to strengthen corecompetencies of individual centres and to develop newresearch priorities. During the transition period, aboutone-third of the Helmholtz Association's institutionalfunding will be allocated on a competitive basis.

29

part

four


13 The 'Blue List' are research organisations founded in 1999, and focus on long-term, interdisciplinary research that is independent of university research, butsupports both Laender and federal research objectives.

Measuring outcomes4.22 Germany has a rich tradition of evaluation of its

research institutions and activities. Evaluation takesplace at the system and institutional level, as well as atthe programme and project level. It also occurs duringseveral stages of the research process, mostly in the form of internal or external peer reviews by academicsand industrialists.

4.23 As discussed previously, the Science Council has theprimary responsibility for evaluating researchorganisations and institutes and the structure of theresearch system itself. An evaluation of the entirestructure has taken place during the past decade. As aresult of German reunification, the Science Council firstreviewed all major research institutes in the former East Germany. Subsequently, the responsible federalministries and the BLK decided to review the remainingnon-university research institutions across all ofGermany. The 'Blue List'13 and Helmholtz organisationswere evaluated by the Science Council and variousinternational evaluation committees reviewed DFG,Fraunhofer and the Max Planck Society.

4.24 The evaluations have resulted in a number of changes tothe research system. First, funding for six institutes onthe Blue List was completely halted. Second, withinFraunhofer, a number of research institutes wereconsolidated to reduce fragmentation of the researchbase. Finally, the internationalisation of research inGermany has been strongly encouraged.

4.25 In addition to imposed external evaluations, self-evaluation has long been an integral part of scientific lifein German research institutes, albeit to differing degrees.Internal reviews are usually conducted by the scientificadvisory boards of the research centres and aregenerally highly significant to the self-assurance andperspective of the individual institutes. Over that lastfew years all non-university, government-fundedresearch institutes are expected to show evidence ofregular, systematic evaluation. Furthermore, they havebeen compelled to undergo external evaluations, someof which had grave repercussions for their future.Evaluations also take place at the programme andproject level, mostly initiated within the individualresearch organisations. There is no common frameworkfor evaluation at these levels, nor is there agreement onthe impact evaluations ought to have on futureoperations and research work.

4.26 Research project evaluation takes place at various stagesof the project life-cycle. Ex-ante evaluations are oftenconducted by means of workshops aimed at assessingthe appropriateness of certain research priorities andprogrammes. Since 2000, all programmes of the BMWi,the main sponsor of research, use programmemonitoring systems to assess how the objectives of theresearch programmes are translated into action and theeffects of the preliminary results.

Part 2

31

part

five

Institutional context5.1 The system of government funding of research and

development in the Netherlands, and the organisationsthat play a role within it, have not changed significantlyover the past decade. However, recent changes havebeen implemented to simplify regulatory environment.As shown in Figure 15, at the funding and/or policylevel, two organisations play a dominant role, theMinistry of Education, Culture, and Science (OCenW)and the Ministry of Economic Affairs (EZ). The formerfocuses primarily on education and research while thelatter is concerned with technology development andinnovative aspects of research. Together, these twoministries provide approximately 80% of all researchfunding at the national level in the Netherlands.

5.2 At the intermediate level, the Netherlands Organisationfor Scientific Research (NWO) finances universityresearch through various initiatives, including projectfinancing14, research programmes and individualsupport. Other intermediary organisations, e.g., Novemand Senter, implement policy, apply research subsidiesor provide information services.

5.3 Government research institutes are financed by theirrespective ministries. The Ministry of Agriculture andFisheries (LNV) is in a special position because itfinances research in its own governmental researchinstitute (DLO), as well as in the AgriculturalUniversity in Wageningen. The Ministry of EconomicAffairs (EZ) supports the research institutes, includingthe Large Technological Institutes (DLO), that conductresearch which is of particular significance to theDutch economy.

Part 5 The Netherlands



TNOPeripheral

Research InstitutesGoverment

Research Institutes

System of research funders and providers in the Netherlands15

Other Ministries

NWO KNAW Senter, Novem

Ministry of Education, Science

and Culture

AWT, otheradvisory councils

Other Private Funds (esp. health) Industrial Firms

Universities Large TechnologicalInstitutes (DLO) Industry R&D

Ministry of Economic Affairs

Source: Frinking et al

14 In the case of the NWO, Foundation of Technical Sciences (STW) funding also requires the participation of firms with a potential interest in the results.

32

part

five


Funding/policy level

5.4 The Ministry of OCenW provides the major overallsource of R&D financing in the Netherlands, fundingnearly 66% of government expenditure on R&D in2002. Almost 70% of these funds is allocated touniversities as so-called 'first flow funding' for research.The Ministry also indirectly supports university researchby its contribution to the research council budget.Universities receive direct ministry funding through tworoutes; partly via a block grant allocation based onstudent numbers, and partly as conditional financingexplicitly directed to support research based on five-year plans. Recently this latter route has changed.Conditional financing now occurs through researchschools within the university which must have gainedthe approval of a Royal Academy of Sciences (KNAW)committee. The Netherlands Organisation for ScientificResearch (NWO) finances university research, and bothNWO and KNAW fund research in institutes peripheralto the universities. The Ministry's policy role regardingR&D is to promote an environment that encouragesoptimal use of resources to support the transfer of highstandard research into policies to improve the welfareof society.

5.5 The Ministry of EZ is responsible for more than 15%of the national research expenditure. Its R&D relatedmission is to stimulate new fields of scientific andtechnological research deemed significant to theDutch economy. The Ministry promotes technologicalinnovation by introducing tax incentives andencouraging research cooperation among companiesand between companies and educational andresearch institutes.

5.6 As Figure 16 shows, with the exclusion of OCenW andEZ, the other ministries together fund less than 20% ofR&D at a national level, with the ministries ofagriculture and transport being the largest contributorsamong them. Agriculture allocates two-thirds of itsfunds to its own agricultural research institutes whileTransport funds a wide variety of research activities,only 25% of which directly through government relatedresearch institutes.

Advisory Councils

5.7 Advisory councils support the government and theresearch system as a whole. The Advisory Council forScience and Technology Policy (AWT) provides generalscience and technology policy support. Sector councils,with the Sector Councils Consultative Committee (COS)as their consultation platform, support policymaking ina number of social science areas. In principle, the sectorcouncils are not advisory bodies, but are intended toinform the policymaking processes.

5.8 The Netherlands Scientific Council for GovernmentPolicy (WRR) was established originally on a provisionalbasis in 1972. It was granted formal legal status underthe Act of Establishment of 30 June 1976. The Councilfocuses especially on developments that affect society inthe longer term. Policy decisions taken in areas likehealth-care, transport, communication and educationcan determine the shape of society for decades. TheCouncil seeks to explore new problems, analyseexisting problems from a coordinated standpoint,outline new perspectives and offer possible directionsfor government policy. By addressing the uncertaintiesinherent in particular policy areas and the potentialconsequences of various options, decision makingquality ultimately can be improved.

Intermediary agencies

5.9 Intermediary organisations and boards implementpolicies and perform brokerage activities at a middlelevel between the ministries and the specific researchprovider institutes. Their role has been stronglydeveloped and emphasised in the Netherlands,originating from the principle of separating policydecision making and policy implementation. Both theMinistry of OCenW and the Ministry of EZ have anumber of implementing agencies that manage theirresearch programmes and other policies that aim tostimulate research and innovation.

Distribution of spending across main national ministries16

Agriculture

Defence

EZ

Foreign affairs

Health

OCenW

Transport

Other

2.4%5.5%

2.1%

2.6%2.1%

15.3%4.1%

65.9%

Source: Science Budget 2002, Ministry of OCenW

33

part

five


5.10 NWO and the Royal Academy of Sciences (KNAW) arethe two dominant intermediate national institutions thatfall within the domain of the Ministry of OCenW. NWOis the central Dutch organisation in the field of basic andstrategic scientific research. NWO acts as the nationalresearch council of the Netherlands. NWO finances andcommissions research throughout the country, usually atuniversities, on issues relating to innovation, quality,society and international profiling. KNAW is the majoradvisory council for the Dutch government on scientificmatters. Its function is to ensure scientific foresightstudies are carried out. It also plays important roles withrespect to quality assurance, the maintenance of widthand balance in the Dutch science portfolio and byproviding scientific advice to the Cabinet. NWO andKNAW also finance their own research institutions, withoverall budgets of €120 and €60 million, respectively.

5.11 Other important institutions at this intermediary middlelevel are the government boards of international andnational research institutions, such as TNO or the GTIs(see below), and universities. The main intermediaryagencies that report to the Ministry of EZ are Senter andNovem. Both agencies are responsible for large subsidyschemes relating to research and technologydevelopment in the fields of energy, environment andtechnological innovation.

Research providers

5.12 In the Netherlands, scientific research is conducted atthe level of the research institutions, the majority withinthe university system. Other important researchinstitutions are:

! the Netherlands Organisation for Applied ScientificResearch (TNO), a for-profit research contractor.TNO forms a bridge between basic research andpractical application.

! the five 'Large Technological Institutes' (GTIs). Amore competitive programme of governmentfunding is being established for the GTIs, wherebya certain fraction of government funding has to bematched by industry, to ensure that the institutes'research takes a direction that will be of interestto companies.

! the research bodies of some of the ministries. Theministries for health, transport, environment andjustice all maintain in-house research facilities.

5.13 Universities receive research funding from varioussources. A distinction is made between the so-calledfirst, second and third money streams flowing into the universities.

! The first money stream largely follows historicallyestablished mechanisms. It takes the form of a lumpsum budget calculated on the basis of number ofstudents and research criteria. The calculatedresearch component in the lump sum budget isabout €1.4 billion annually and is divided among14 universities. In 1999, the total of the first moneystream including the education componentamounted to €2.2 billion. First money streambudget allocations are not dependent on evidence ofscientific research quality15 although the Ministryintends to change this situation.

! The second money stream, amounting to about€250 million, is allocated by the National ScienceFoundation (NWO) on the basis of quality, throughwhich it aims to reward excellent research. It alsopublishes priority research themes that provideguidance to research organisations who submitwinning proposals. NWO has an €25 millionannual capital budget to cover the purchase oftechnologically advanced equipment. The demandfor these monies outstrips the availability by 2 to 5times. KNAW's budget is about one-fifth the size ofNWO's, and is largely used by its own researchinstitutions. It has no specific capital expenditurebudget for outfitting research.

! The third money stream, totalling about€300 million, finances contract research.Independent medical foundations fund more thanhalf of the third money stream. Only about 20% ofcontract research is paid for by private industry.

15 Of course, universities and institutes have their own independent internal and external quality assurance procedures.

34

part

five


5.14 The organisations that play a significant role in the DutchR&D system are fairly constant. However, changes withinthem and the context resulting from different science andtechnology policies of successive Dutch governmentshave resulted in overall changes to the system. Among themost significant changes are the following:

! The Ministry of OcenW's direct funding of universityresearch changing from block grants based onstudent numbers to explicit research financing, firstwithin the 'conditional financing' scheme and, morerecently, with the establishment of research schools.

! NWO evolving from an organisation dedicatedmainly to basic research to an organisation withactivities in all research fields. The organisation alsochanged its project-oriented research funding focusto more programme- and individual-orientedschemes. Moreover, the organisation now alsoattracts funds from other ministries, including theMinistry of OcenW.

! TNO evolving from an organisation predominantlyfinanced via public administration block grants toone financed by programme contracts withministries and private enterprises.

Priority setting and coordinating processes5.15 Foresight helps play an important role in the process of

setting research strategies and priorities in theNetherlands. Foresight exercises are specificallydesigned, interactive processes. As described above, theNetherlands has a dense intermediary level ofinstitutions, councils and independent bodies whichcontribute to agenda setting, mediate the link betweenresource allocation and research production andoversee parts of the research processes. Each of theseorganisations conduct foresight activities, however, inmost cases from a distinct perspective, e.g.:

! The Advisory Council for Science and TechnologyPolicy (AWT) looks broadly at the strategic directionof science and policymaking.

! The Royal Academy of Sciences (KNAW) examines developments that affect the variousscientific disciplines.

! The Sector Councils initiate foresight studies thatexplore the social, economic and environmentalchallenges to specific policy domains.

5.16 These information-gathering exercises ensure that allrelevant stakeholders, such as academics andindustrialists, as well as interest groups and researchusers are all included in the process and decisions thataffect them.

5.17 The AWT and the sector councils assess the demand forspecific research. It is the AWT's task to advise theGovernment and the Parliament on the science andtechnology policy to be pursued nationally andinternationally, and on information policy in the scienceand technology fields. It is also the Council's task, at therequest of the Minister of OCenW, to carry out foresightstudies in the field of science and technology or else tocommission them. The core of its advisory function isfocused on the knowledge and innovation process andits development. The recommendations made by theCouncil may also relate to matters that affect, or are theresult of, research and science practice and technologydevelopment. The Council comprises representativesfrom academia, industry and government who areelected on personal merit and do not represent anyvested interest.

35

part

five


5.18 The most significant structural example of horizontalcoordination is the Consultative Committee of SectorCouncils (COS). It is the platform for cooperation amongsector councils and their members. The functions of theCOS are as follows:16

! to consult on issues of common interest, exchangingideas and discussing research themes and information;

! to promote a coordinated approach by addressingmultidisciplinary issues, organising projects andjoint studies by COS members, funded by the COSCo-ordination Fund;

! to organise studies, workshops and conferences onmethod development;

! to coordinate joint input during administrativeconsultations (e.g., between ministries, NWO, theAssociations of universities in the Netherlands);

! to reflect joint interests; and

! to conduct PR at a system level.

5.19 Currently, there are four sector councils (The InnovationNetwork Rural Areas and Agricultural Systems, TheNetherlands Development Assistance Research Council,The Council on Health Research, The Council forResearch on Spatial Planning, Nature and Environment)with another four councils are being added (PublicAdministration, Justice and Security, Education, Traffic,Transport and Infrastructure, Employment, Health andSocial Security).

5.20 The councils do not endorse a particular policyapproach. Their priorities are determined on the basis ofsocietal support, interest and/or scientific need; in otherwords, they are primarily 'curiosity driven'. However,since this approach tends to provide the primaryresearch focus it can be considered consultative,involving many potential research providers.

5.21 In a sector council, researchers, members of society,(including trade and industry), and governmentrepresentatives (as advisory members) together exploremedium- and long-term scientific and social trends andpresent an independent view of the priorities forstrategic medium- and long-term research in their sector.On the basis of these explorations or foresight activities,sector councils also look at trends in science andtechnology and their consequences for society. Sectorcouncils are independent. They are not responsible forpolicy setting, nor do they allocate funds or manageresearch. They derive their authority from the weight oftheir evidence.

5.22 The integrated explorations and analyses of the sectorcouncils can identify knowledge gaps and lead to basicand strategic research programmes, which in turn maylead to the establishment of a research programme by,for example, NWO. NWO's strategic plan would bebased partly on the input of the sector councils and alsoon the mandate of the ministry involved. Sector councilsmay also make proposals to the ministers concernedabout the coordination of research, development andknowledge infrastructure. Furthermore, although theyare not advisory bodies, the sector councils may,through their explorations, contribute to policymakingin general. The ministries involved may take the resultsfrom sector councils' efforts into account when makingpolicy decisions or when formulating requests for adviceto advisory bodies.

5.23 The Dutch Cabinet published five foresight reports inSeptember 2001 covering main issues and policyoptions for five different ministries, including theministries of OCenW and EZ. In the wake of thesereports, policy options for a more quality-basedallocation of government funds to universities arebeing considered.

Selecting and commissioning research5.24 In the Netherlands, organisations such as Senter and

Novem coordinate and commission some of theresearch activities relevant to the ministries of EZ,Transport and Environment. Various other independentor temporary programme offices are responsible forimplementing research strategies and commissioningresearch for programmes significant to the otherministries, e.g., the Economy, Ecology, Technology (EET)programme, and Flyland.

5.25 Flyland is a programme of research examining thefeasibility and implications of an airport in the NorthSea. It is a Dutch cabinet level initiative, with funding of€22.7 million, shared among the ministries of Transport,Public Works and Water Management; Economic Affairsand Housing, Spatial Planning and the Environment andby the air transport sector. Parliament has granted theFlyland programme office independent agency status. Itstarted operations in spring 2001.

16 Source: www.minocw.nl/cos

36

part

five


5.26 The Minister of Transport, Public Works and WaterManagement holds primary political responsibility forthe research programme. The Minister acts on behalf ofall involved as the principal of the programme officeand reports to Parliament. The programme officecommissions specific studies. It is Flyland's aim that,upon completion of the programme in 2006, there willbe agreement that the relevant questions have beeninvestigated in the appropriate manner. Heavy emphasisis placed, therefore, on quality assurance and the public'anchoring' of the research programme by acting onresults from authoritative experts. The Minister receivesadvice from one of the sector councils at the programmelevel and from the programme office at the topic level.

5.27 The aim of the Dutch Economy, Ecology, Technology(EET) programme is to stimulate and support long-termprojects aimed at technological breakthroughs that willgenerate substantial ecological and economic profit.During the past five years, private companies andresearch institutes have collaborated on 150 EETprojects with ambitious goals in the field ofsustainability. EET is a joint programme of three DutchMinistries: the Ministry of Economic Affairs, the Ministryof OCenW, and the Ministry of Housing, SpatialPlanning and the Environment. The practical executionof the programme is the responsibility of the EETprogramme office, a partnership between Novem andSenter. Within EET, projects are awarded by an externaladvisory committee on the basis of quality and expectedcontribution to the policy aims of the programme.

Measuring outcomes

Communication of results

5.28 The Netherlands Observatory of Science andTechnology (NOWT) plays a significant role in thecommunication of information to policy makers.NOWT is a formal cooperation between the Centre forScience and Technology Studies linked to the Universityof Leiden and the Maastricht Economic ResearchInstitute on Innovation and Technology of MaastrichtUniversity. It is funded by the Ministry of OCenW.

5.29 NOWT's mission is to collect and analyse data aboutthe Dutch research system in a broad sense, includingits interface with public information services relating toscience, higher education and technological innovation.NOWT is monitored by a committee comprisingrepresentatives from all large public organisationsinvolved in the preparation or realisation of nationalresearch policy.

Evaluation

5.30 Resulting from pressure by the General Audit Office ofthe Netherlands, the Dutch government introduced anew concept relating to the presentation of thegovernment's budget. Rather than presenting traditionalitemisation of expenditure, the budget must now show amore transparent relationship among policy, effect andfunding. The government bill "From Policy Budget toPolicy Accountability" formalised this concept. Thepresentation of the 2002 budget was based on this newapproach and subsequent funding reports will require asimilar approach.

5.31 The ministry of EZ has recently established a separateevaluation unit, which will be responsible formonitoring the effectiveness of its policies, including theinvestments to research. The unit has developedevaluation guidelines and conducted a review ofevaluation techniques. These guidelines emphasise theneed for improved data collection, ex-ante evaluationsand the use of econometric methods in evaluation.

5.32 Evaluation of scientific research is conductedsystematically in the Netherlands. Academic researchunits are required to conduct internal evaluations oftheir objectives and programmes every three years,supplemented by external evaluations every six years. In2000, the association of universities, NWO and KNAWpresented a report on the quality assurance systems inthe Netherlands. It outlined a new approach toconducting quality evaluations. Research units nowprovide frequent information to supervising bodies oninput and output indicators (e.g., personnel, budget,publications), conduct internal evaluationsapproximately twice every six years and undergoexternal, ex-post evaluations every six years based onpeer review. This approach replaces the more frequentpractice of continuous external evaluation which placesconsiderable stress on the research organisations.

Part 6



United States

37

part

six

Institutional context 6.1 The US federal government supports R&D in pursuit of

diverse national goals and objectives. Federal spendingfor R&D is heavily focused on defence, health, spaceand energy, as indicated in Figure 17.

6.2 Federally supported R&D is performed in a number ofdiverse institutions, including government laboratories,industry, academic institutions and independent R&Dorganisations. Thousands of institutions in the US conductR&D, funded by government, industry, state and localgovernments, private foundations, funds from colleges anduniversities and other sources. Industrial research iscarried out by thousands of firms, large and small,although some 100 large firms account for more than 50%of all industrial R&D spending. The largest providers ofindustrial R&D are the aircraft, communicationsequipment, chemical and computer and office equipmentindustries. Nearly every academic institution conductssome research. However, about 100 universities accountfor more than 80% of all academic R&D spending. It isestimated that there are more than 700 federal laboratories

including federally funded research and developmentcentres (FFRDCs). However, a much smaller number ofthese are of substantial size, with a few dozen conductingmost of the R&D done in such facilities.

Priority setting processes and coordinating processes6.3 The US federal research funding process has

traditionally been highly decentralised. During the lastdecade, various reports have been publishedemphasising that the lack of coherence in this fundingprocess results in the suboptimal transfer of researchresults to support national interests. One of theconclusions of a National Academy of Sciences (NAS)report (NAS, 1999) stated that "mechanisms forcoordinating research programmes in multiple agencieswhose field or subject matters overlap are insufficient."

6.4 Goals, priorities and budget allocation are all a part ofthe R&D budgeting system, yet there is no formallydefined process within the federal government to setgoals and priorities or make allocation decisions forscience. The process is iterative and complex. Manyplayers with different interests interact to influence theoutcomes. As the NAS has noted, as these playersinteract, the interests of science come up against othernational goals and priorities, and funding for specificareas of science compete with one another forallocation of funds (NAS, 1995, p. 4). The outcome isroughly balanced across many areas of science andtargets a number of specific goals and priorities.

6.5 The Executive Branch has initiated various debates andinitiatives to support coordination of research acrossagencies.17 However, it has not yet generated a widelyaccepted process for the federal government.

Source: AAAS Report XXVI: Research and Development FY2002, Intersociety Working Group, 2000.

Distribution of R&D funds among the agencies, FY 200217

Defence

Health and Human Services

NASA

Energy

NSF

Other

7%

48%

23%

10%

8%

3%

17 National Academy of Sciences (1995); National Science Board (1997)l National Science Board (2001).

38

part

six


6.6 The Office of Science and Technology Policy (OSTP),under the direction of the Science Advisor to thePresident, provides a resource for scientific andtechnical information within the government. It alsoplays a role coordinating R&D activities throughout theagencies and ensuring comprehensiveness in the budgetfor science activities. Its main responsibility is thecontinuing development and implementation of anational strategy to determine and achieve theappropriate scope, direction and extent of scientific andtechnological efforts. The science advisor and his staffserve as advisors to the President and so act toimplement the President's policy. (OTA, 1991, p. 74)With each new administration, the goals and prioritiesfor science and technology funding change. The OSTPhelps to coordinate federal science activities to helpmeet the President's goals.

6.7 Within the domain of OSTP's responsibilities are twoadvisory councils, the National Science and TechnologyCouncil (NSTC) and the President's Council of Advisorson Science and Technology (PCAST).18

6.8 NSTC was established to integrate the President's S&Tpolicy agenda across the federal government and ensurethat S&T is considered alongside the development andimplementation of federal policies and programs. It is apolicy and budgetary coordination body through whichall-executive departments and agencies coordinate S&Tactivities that require significant levels of interagency

coordination. It operates with assistance from the OSTPwhich helps to set the NSTC agenda by suggestingtopics around which the NSTC forms committees. Thesecommittees then review government spending inspecific areas of research and recommend to OSTPwhere priority or allocation shifts might be needed. Themembers of PCAST are drawn from industry, educationand research institutions. PCAST was established tosolicit advice from the private sector and the academiccommunity on technology, scientific research prioritiesand mathematics and science education.

6.9 The OSTP uses the information from both councils toadvise executive agencies where shifts in fundingpriorities might be considered. OSTP also solicitsinput from the larger scientific community aboutwhere priorities and resource allocation should focus.This does not translate directly into budgetary action,however. The Science Advisor and his staff at OSTPare advisors to the President, and therefore do nothave direct line responsibilities for budget allocation.The OSTP director and staff act primarily tocoordinate activities and persuade other players of theimportance of science.

6.10 The Office of Management and Budget (OMB)compiles the President's Budget seeking, withinbudget constraints, to support the President's policies,programmes and commitments. The formal processstarts each summer in the Executive Branch when the

Goal setting and allocation process in the US federal goverment18

President &OMB

CongressionalAppropriations

Source: Popper (2001)

A

B

C

A

B

C

Scie

nce

AgencyAllocations

AgencyDirectorateDecisions

Priorities

Goals

NST

C

Sc

ient

ists

Stak

ehol

ders

18 In Figure 18, OSTP, PCAST, and NSTC all fall under the direct responsibility of the White House, i.e., the US President.

39

part

six


agencies begin preparing their budgets for the fiscalyear beginning in October of the following year. OMBreviews and coordinates the budget, resolves anydisagreements and submits the budget to Congress inlate January or early February. There is no one R&Dbudget as such: there are many R&D budgetsembedded in the budgets of 21 federal departmentsand agencies that conduct or support R&D. The R&Drequests are handled in the budget process as parts ofthe agency's total budget. OMB seeks to implementthe President's policy and so does not set specificpriorities for science. The OMB also has aresponsibility to examine agency proposals forredundancy and opportunities for coordination.

6.11 OSTP and OMB issue a budget memorandum to allresearch agencies each year on research anddevelopment priorities. Guidance is sent to agencieslisting presidential priorities, including trade-offs andreallocations across agencies that reflect these priorities,as well as crises, opportunities or evaluations. Thepriorities for R&D are based on a set of goals named bythe Administration.

6.12 Agencies differ in setting priorities for science, based onwhether they have a scientific or mission orientation.Most agencies now use some combination of anoutreach or advisory approach to priority setting bygathering views from various stakeholders, combinedwith a strategic planning approach, setting strategicgoals and seeking to implement them.

6.13 Three main models for priority setting can bedistinguished at the agency level:

! Agency Outreach Model, where input from leadingscientists and technologists from government,industry and academia is sought. Research themesare defined and then prioritised.

! Advisory Model, where an agency asks anindependent organisation (often the NAS) to provideinput and recommend priorities.

! Science Advisory Board Model, where agencies areadvised by committees that are established toperform that specific task.

6.14 The budget that the US president submits to the USCongress might benefit from coordination efforts.However, as there is no single R&D budget presented toCongress, several committees consider various budgetappropriations to R&D. There is no congressionalcoordinating authority for R&D to overview andcoordinate changes made to the budget. The HouseCommittee on Science, which has oversight authorityover all federal non-defence R&D, comes closest.However, it does not have legislative authority over thebudgets of various departments, including the twobiggest funding agencies The Department of Defence(DoD) and The National Institutes of Health (NIH).

Selecting and commissioning research6.15 US federal government research is conducted in a

number of different organisational settings. Each placesdifferent contractual obligations and levels of freedomupon the research providers. The four most commonlyused approaches are:

! Government-owned, government-operated laboratory,or GOGO-a laboratory owned, operated and fundedby the federal government and staffed by federalemployees. Examples include National Institute ofStandards and Technology (NIST) laboratories, NIHintramural laboratories, the National Institute ofOccupational Safety and Health and the U.S.D.A.Peoria Regional Laboratory.

! Government-owned, contractor-operated laboratory,or GOCO - a laboratory owned and funded by thefederal government and operated and staffed by aprivate contractor. The contractor may be a profit-making firm, a non-profit organisation or one ormore academic institutions. Examples include all ofthe Department of the Energy (DOE) nationallaboratories mentioned below.

! National Laboratories - e.g., a large, multipurposelaboratory of the Department of Energy, including themajor weapons laboratories - Los Alamos, Sandia,and Livermore - as well as Argonne, Brookhaven, OakRidge, Lawrence Berkeley, and others. (NationalLaboratories are one type of FFRDC).

! Federally funded research and development centres,or FFRDC - a particular form of long-term governmentcontract with a non-governmental organisation to staffand operate a laboratory or other research centre thatis funded in whole or in substantial part by the federalgovernment. Some FFRDCs are agreements to operate GOCOs, while others are contracts thatsupport contractor-owned and contractor-staffedorganisations (NAS, 1995, p.60).

40

part

six


6.16 Within the individual programmes of agencies anddirectorates, research funds are allocated on the basis ofpeer review, an agency's board review, managers'discretion and combinations thereof. Grants are oftenprovided on the basis of peer review while contracts areoften established on the basis of a board review or byresearch managers. Through annual budget guidance tofederal agencies, the OSTP and the OMB encouragefunding of research projects that are peer reviewed overthose that are not reviewed.

6.17 The US General Accounting Office (GAO) Report"Federal Research: Peer Review at Federal ScienceAgencies Vary" (March 1999) looked at how federalagencies conducted peer reviews of research productsto chart the various expert review procedures. Itconcluded that there was no uniform federal policy forconducting peer reviews. Each of the 12 agenciesexamined by the GAO conducted their peer reviewsaccording to a variety of policies, orders or otherinternal guidance. There is general agreement that peerreview practices should not be dictated uniformly forevery agency or for all types of federally fundedresearch. Rather, the practices should be tailored toagency missions and research type.

6.18 The US Environmental Protection Agency (EPA) hasmade the most intensive effort to expand theapplication of peer review to the use of science in itsown decision making. In the mid 90s, US EPA issued aformal peer review policy that outlined the need forpeer review of all scientific and technical products. Inthe late 90s, to aid the implementation of the peerreview policy, the EPA developed a Peer ReviewHandbook to ensure that proper peer reviewprocedures are followed for major work products. EPAstrives to ensure that the scientific and technicalunderpinnings of its decisions meet two importantcriteria: they should be based upon the best currentknowledge from science, engineering and otherdomains of technical expertise; and they should bejudged credible by those who deal with the Agency.For those work products that are intended to supportimportant decisions or that have special importance intheir own right, external peer review is the procedureof choice. Peer review is not restricted to thepenultimate version of work products; in fact, EPA hasfound that peer review at the planning stage can oftenbe extremely beneficial.

Measuring outcomes

Tracking research investments, activities and outcomes

6.19 Decision makers both inside and outside the US federalgovernment have lacked a tool for understanding whereand how much the federal government is spending oneach area of science and technology. RaDiUS, whichstands for "Research and Development in the UnitedStates", is the first information system that systematicallyconnects the highly aggregated budget data on federalR&D with the disaggregated information on individualR&D tasks and awards to provide a complete picture ofall federal R&D activities. RaDiUS collects data onfederal R&D activities from information alreadygathered by the federal government, even if they havenot traditionally been viewed as relevant to the trackingof federal R&D activities. For example, RaDiUS includesinformation from the Federal Assistance Awards DataSystem (FAADS) and the Federal Procurement DataSystem (FPDS), neither of which has ever before beenused to track R&D. The data have then been broughttogether and augmented with information from agency-specific R&D databases and information gathering usingcommon data fields and codes to form a comprehensivepicture of federal R&D. RaDiUS has been used bynumerous federal agencies and contractors to supportR&D planning and coordination efforts, leverage R&Dinvestments and transfer technology. RaDiUS allowsusers to see the total R&D investment in specific areasof science and technology across all federal agencies orexamine the details of research investments within aspecific agency. It thus offers the potential formonitoring redundancies in federally supportedresearch and development.

Utility of research outcome

6.20 With the establishment of the 1993 GovernmentPerformance and Results Act (GPRA), federal agencies,including those that fund research, were formallyrequired to set strategic goals and to use performancemeasures for management and budgeting. The objectiveof the GPRA is to encourage greater efficiency,effectiveness and accountability in federal programmesand spending. The GPRA has been an important driverfor increased pressure on research funding agencies todemonstrate effects and results of their research anddevelopment investments19.

19 At the same time, some scholars (e.g., C. Hill, P. Shapira) argue that accountability systems have already been in place for a long time to ensure that missionoriented R&D programmes accomplish their objectives and that fundamental research programmes are well managed and directed.

41

part

six


6.21 The US GAO published a report on measuring R&Dperformance. While it had, in the first instance, focusedon two specific technology programs, the report drewsome general conclusions with respect to measuring theimpacts of federal research programs: "there is no singleindicator or evaluation method that adequately capturesthe results of R&D. However, indicators do providehelpful information for making decisions about R&D."Establishing relevant quantitative and qualitative outputindicators are seen as supporting policy decisions buthave clear limitations in their performancedemonstration. The report stresses that "output measuresare highly specific to the management and mission ofeach federal agency and that no single indicator existsto measure the results of research" (GAO, 1997, p. 6-7).

6.22 Because of concerns that implementing the act would beparticularly difficult for research activities, theCommittee on Science, Engineering and Public Policy(COSEPUP) of the National Academy of Sciences, theNational Academy of Engineering, and the Institute ofMedicine considered the most effective ways to assessthe results of research. The results of this study aredescribed in two reports: Evaluating federal researchprograms: Research and the GPRA and Implementing theGovernment Performance and Results Act for research.COSEPUP made a number of conclusions, including:

! Both basic and applied research programmes can beevaluated meaningfully on a regular basis. For theapplied research programmes of the missionagencies, specific practical outcomes can bedocumented and progress toward their achievementcan be measured annually. Basic research yieldsannual results that can be evaluated meaningfully,but these evaluations might not give sufficientinformation about the ultimate practical outcomes.

! Agencies must evaluate research programmes byusing measurements that match the character of theresearch. Differences in the character of theresearch will lead to differences in the appropriatetimescale for measurement, in what is measurableand what is not, and in the expertise needed bythose who contribute to the measurement process.Evaluating basic research requires substantialscientific or engineering knowledge; measures ofthe practical outcomes of basic research usuallymust be retrospective.

! The most effective means of evaluating federallyfunded research programmes is expert review.Expert review, including peer review (judging thequality), relevance review (judging whether anagency's research activities are relevant to itsmission), and benchmarking (judging the relativestanding in an international perspective), should beused to assess both basic research and appliedresearch programmes.

! The development of effective methods for evaluatingand reporting performance requires the participationof the scientific and engineering community.

BMBF BERICHT Faktenbericht Forschung (2002). Bundesministerium für Bildung und Forschung (BMBF), January 2002

Cave J, Frinking E, Malone K, van Rossum W, te Velde R (1999). Modalities of R&D Funding: A Comparison of EU MemberStates. Leiden, RE-99.018.1.

Cook-Deegan RM (1995). Previous Analysis of the U.S. R&D Allocation Process, working paper, Washington, D.C.:National Academy of Sciences.

Council of Science and Technology Advisors (Canada) (1999). Science Advice for Government Effectiveness (SAGE) AReport of the Council of Science and Technology Advisors. Ottawa, CSTA.

Council of Science and Technology Advisors (Canada) (2001). Science and Technology Excellence in the Public Service(STEPS): A Framework for excellence in Federally Performed Science and Technology, Ottawa, CSTA.

Council of Science and Technology Advisors (Canada) (2001). Reinforcing external advice to Departments (READ) a reportof the Council of Science and Technology Advisors (CSTA), Ottawa, CSTA.

Cunningham, P (2001). European Study Report: Germany, in: Measuring and Ensuring Excellence in Government Scienceand Technology: International Practices (Report prepared for the S&T Strategy Directorate, Industry Canada; In support ofthe work of the Council of Science and Technology Advisers).

Environmental Protection Agency (EPA) (1997). Peer Review and Peer Involvement at the U.S. Environmental ProtectionAgency. Washington, DC: EPA Available at http://www.epa.gov/ordntrnt/ord/spc/perevmem.htm.

Environmental Protection Agency (EPA) (1998). Science Policy Council Peer Review Handbook. Office of Science Policy.EPA 100-B-98-00. Washington, DC: EPA.

Frinking E, Hjelt M et.al (2002). Benchmarking Innovation Systems: Government Funding for R&D, Tekes TechnologyReview 122/2002, Helsinki, 2002

Frinking E, Kahan JP (2000). Stimulating Industrial Innovation for Sustainability: Country Reports, Leiden, RE-2000.16,November 2000.

General Accounting Office (1997). Measuring Performance: Challenges in Evaluating Research and Development GAO/T-RCED-97-130.

General Accounting Office (1999). Federal Research: Peer Review Practices at Federal Science Agencies Vary,GAO/RCED-99-99.

Government of Canada (1996). Science and Technology for the New Century, a Federal Strategy. 1996. Government ofCanada, Minister of Supply and Services Canada.

Hannele, K (1996). The Finnish Research System in the 1990s, in: EUROMECUM: An Overview of Research Policy inEurope, Raabe, Stuttgart, 1996.

Hickling Arthurs Low (1999). The Roles of the Federal Government in Performing Science and Technology: An InternationalPerspective (Report Prepared for Canadian Council of Science and Technology Advisors Secretariat Industry Canada)

Hill, CT (1995). On the Allocation of Federal R&D Funds, working paper, Fairfax, Va.: The Institute of Public Policy, GeorgeMason University.42

refe

renc

es


References

Industry Canada (2000). Framework for Science and Technology Advice: Principles and Guidelines for the Effective Use ofScience and Technology Advice in Government Decision Making. 2000. Ottawa.

Industry Canada (2000). Forging ahead: a report on federal science and technology: 1999. 2000. Ottawa.

Industry Canada (2001). Investing in Excellence, 1996-2001: A Report on Federal Science and Technology - 2001 Ottawa.

Kai H, Karjalainen S & Parkkari T (2000). The State and Quality of Scientific Research in Finland: A Review of ScientificResearch and Its Environment in the late 1990s. Helsinki: Academy of Finland.

Lomas J (2000). Jonathan, Using 'Linkage And Exchange' To Move Research Into Policy At A Canadian Foundation. HealthAffairs, 19 (3), p.236-240.

Ministry of Education (2001). Research in Finland, Helsinki: Ministry of Education.

Ministry of Education, Culture and Science (1999). Nothing ventured, nothing gained, Science Budget 2000. Zoetermeer:Ministry of Education, Culture, and Science.

Ministry of Education, Culture, and Science, Progress Report Science Policy (2002). Zoetermeer: Ministry of Education,Culture and Science.

National Academy of Sciences (1999). Evaluating Federal Research Programs: Research on the Government Performanceand Results Act. Committee on Science, Engineering, and Public Policy.

National Academy of Sciences, Committee on Criteria for Federal Support of Research and Development. (1995). AllocatingFederal Funds for Science and Technology, Washington, D.C.: National Academy Press.

National Science Board (1997). Government Funding of Scientific Research, working paper, Washington, D.C.: U.S.Government Printing Office, NSB-97-186.

National Science Board (2001). Federal Research Resources: A Process For Setting Priorities Washington, D.C.: U.S.Government Printing Office, NSB-01-156.

Popper SW, Wagner CS, Fossum DL, Stiles WS (2000). Setting Priorities and Coordinating Federal R&D Across Fields ofScience: A Literature Review (DRU-2286-NSF). Washington DC: RAND Science and Technology Policy Institute.

Prihti A et.al (2000). Assessment of the additional appropriation for research, Sitra Reports Series 2. Helsinki: Sitra

Science & Technology Policy Council (1993). Towards an Innovative Society: A Development Strategy for Finland, Helsinki.

Shapira P. and Kuhlmann S (2001). Proceedings from the 2000 U.S. - European Workshop on Learning from Science andTechnology Policy Evaluation, Bad Herrenalb, Germany: School of Public Policy, Georgia Institute of Technology, Atlanta,U.S.A and the Fraunhofer Institute for Systems and Innovations Research, Karlsruhe, Germany

Smith W and Halliwell J (1999). Principles and practices for using scientific advice in government decision making:international best practices : report to the S&T Strategy Directorate, Industry Canada in support of the work of the Councilof Science and Technology Advisers Ottawa: Industry Canada

Statistics Finland, Science and technology statistics (see http://tilastokeskus.fi/tk/yr/tttiede_en.html)

Svein K (1997). Funding University Research in the Nordic Countries. Science and Public Policy, 24 (4), August 1997(pp.233-244).

Tekes Annual Review 2001. Helsinki: Tekes National Technology Agency.

Tuomaala, E (2001). Research and technology programme activities in Finland, Helsinki: Tekes Technology Review106/2002.


43

refe

renc

es

an international review on governments’ research procurement strategies · 2018-04-23 · 3...

Documents